Temporal-Spatial Anti-Aliasing for Shaky Digicam Videos

[um3k]

Some of you may know about scharfis_brain's long-ago attempt at temporal-spatial super resolution. http://forum.doom9.org/showthread.php?t=81868&highlight=super+resolution+depan

It's certainly the most successful attempt I've seen at avisynth-based super resolution. In a quest to improve some low-quality digicam videos, I've crudely modified his script to preserve differential motion.

AVISource("Picture0013.AVI")
ConvertToYUY2()

# for comparision
i=lanczos4resize(640,480).converttoyv12()

# for rendering
#~ j=pointresize(640,480).converttoyv12()
j=nnedi2_rpow2(2).converttoyv12()

data=i.depanestimate(range=2,pixaspect=1.0,zoommax=1,improve=false,trust=0)
r=4
j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=15,inputlog="0013.log")

m=mt_makediff(last,j.ChangeFPS(last.Framerate)).Grayscale()
m=Overlay(m, m.Invert(), x=0, y=0, opacity=1.0, mode="lighten", greymask=true, ignore_conditional=false, pc_range=True).Levels(145, 1.000, 160, 0, 255, coring=false)

mt_merge(last, j.ChangeFPS(last.Framerate), m, luma=true)
temporalsoften(r,255,255)
selectevery(2*r+1,r)

#~ Unsharp(varY=1.0,strength=2)

stackhorizontal(last.subtitle("temporal super resolution"),i.subtitle("original (320x240)"))


As you can probably see, my Avisynth scripting ability is pretty basic. But the results I get from this are definitely a step in the right direction. A couple things to note:
1. I've replaced the pointresize with NNEDI2. The pointresize probably would be a little sharper, but it doesn't get along very well with my crude masking.
2. I've replaced the DepanEstimate results with a Deshaker log, which I found to result in more accurate correction.

Here is a still example of its efficacy. Notice that there is a huge reduction in JPEG artifacts, as well as aliasing, particularly on the house in the background. (Click for full size)
http://lh6.ggpht.com/_MCBF9vFZLks/S22wmOlE6bI/AAAAAAAAEWU/Pw0XLF115oA/s800/Example.png (http://picasaweb.google.com/lh/photo/A9s23zYa1Q9Dxl-7Cwu1YQ?feat=embedwebsite)
For comparison, here is the output of scharfis_brain's original script:
http://lh5.ggpht.com/_MCBF9vFZLks/S23KfqsRn1I/AAAAAAAAEXQ/LhyijhIkhVQ/s800/ExampleOriginal.png (http://picasaweb.google.com/lh/photo/FfL18znKkTKAB5osQjVYUQ?feat=embedwebsite)

Unfortunately, there are still motion artifacts, primarily ghosting on differential motion, as well as some chroma artifacts. I'll continue working on it, but am not sure how far I'll be able to go. And, of course, there is the additional limitation of differential motion not receiving any enhancement.

If you want to take a stab at it, or just see the results for yourself, here is the original video file as well as the log file:
Picture0013.AVI (https://docs.google.com/leaf?id=0BxfsQhrtbYrqN2EwMTgzZTUtZmZhYi00ZjEzLWEwZmItZThjMTg4YzlhNjNl&hl=en)
0013.log (https://docs.google.com/leaf?id=0BxfsQhrtbYrqNTJiMGEzYTQtYmQ2Zi00MjU2LWI3YTEtYjg5NDdiMWRhZjAy&hl=en)

So, take a look, and toss in your two cents.

[wonkey_monkey]

j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=15,inputlog="0013.log")

Others will need to remove the inputlog parameter, since we don't have your log file...

Edit: Doh, it's right there in the post :stupid:

David

[um3k]

I included the log file in my post. Look at the links near the end. ;)

BTW, I've been working on this some more, and have made some improvements. I'll post an update when I feel it is ready.

[2Bdecided]

I've used this in the past...
nnedi(field=0,dh=true).turnleft().nnedi(field=0,dh=true).turnright()

vf=last.mvanalyse(idx=1,pel=1,isb=false,blksize=16, truemotion=true)
vb=last.mvanalyse(idx=1,pel=1,isb=true,blksize=16, truemotion=true)

xf=last.mvflow(idx=1,vf)
xb=last.mvflow(idx=1,vb)

interleave(xf,last,xb)
temporalsoften(1,20,20,mode=2)
selectevery(3,1)

There's also a cripplingly slow script here...
http://forum.doom9.org/showthread.php?p=812569#post812569
...which looks like a newer / better version of the one you started with.

I've tried various attempts to mask badly predicted and unpredictable movement and not merge those parts. Nothing worth sharing - and glacially slow!

TBH, it's nothing like real super resolution (I think we all know this) - I have clips with fine details that flicker horribly - real super resolution would fix these nicely.

Cheers,
David.

[um3k]

I just gave those both a try. The script in your post didn't create motion artifacts, but it also didn't really improve the image much. The script in the link caused horrific motion artifacts, and still didn't provide as much improvement as the current version of the script I posted. So I'll keep trying to improve what I've got here. And true, it isn't really super resolution--but it does make it easier to watch.

[poisondeathray]

2Bdecided - what products/software are you referring to with "real super resolution" ? Do you mean like video enhancer , topaz enhance , instanthd , etc... or something else?

[Didée]

"real super resolution" - the myth that some enterprising people claim it would exist.

It has been discussed so often. Search for "(temporal) superresolution" and {my name}, or by MfA as well, who IIRC in the past has elaborated on that topic too.

Short version: "superresolution" is basically possible when the input is aliased due to decimated resolution. Motion compensated (Bob-) Deinterlacers are one special case where TSR is actually working.

But for the typical, average, non-aliased input with non-decimated resolution, there is no super-resolution to achieve.


Also you script has very little to do with super-resolution. The biggest worker is NNEDI2, which softens edge-aliasing. Then you hit after it with some rather weak, rather cautious temporal averaging filter that mostly works in areas with very weak signal.
NNEDI2 on its own is not a true superresolution filter - it does not bring any new information into the frame, it "only" interpolates edge oriented. Smoother edges, but no new information. The next step - temporal averaging only of areas that are highly similar anyway - is no superresolution either. Superresolution would require to appropriately use adjacent information that is quite different to the current information.
(Don't judge just by the housewall texture - result appears nicer, yes, but to some extend that's just lucky coincidence. E.g.: on the trees in the background there is apparent *loss* of detail. On the house's chimney there is apparent *loss* of detail. How comes, since both parts are without 'incremental' motion, in the same way as the house wall is?)

What your script does is upsizing with smooth edges, plus some denoising. There's nothing wrong with that. If the result seems pleasing, then it's fine.

But better don't call that "super resolution". Because it isn't.

[um3k]

Ok, I changed the name of the thread. It now better reflects the function of the script, to improve temporal stability by selectively averaging frames to reduce aliasing, moire, and compression artifacts, for the specific case of low-resolution, shaky video produced by a handheld digicam.

[Terka]

search for "Filip Sroubek" superresolution

[2Bdecided]

2Bdecided - what products/software are you referring to with "real super resolution" ? Do you mean like video enhancer , topaz enhance , instanthd , etc... or something else?I've tried the first two, and been disappointed.

No, I meant the mathematical algorithms proposed in a few papers (see earlier super resolution threads here on doom9) which work in the specific cases Didée mentioned - and certainly we have such a case here. Problem is the maths assumes that the underlying scene does not change - only the camera moves. That's fair enough for background details, but useless and someone walks across the frame swinging their arms!

Cheers,
David.

[2Bdecided]

I just gave those both a try. The script in your post didn't create motion artifacts, but it also didn't really improve the image much. The script in the link caused horrific motion artifacts, and still didn't provide as much improvement as the current version of the script I posted.Oh well - I thought maybe you could adapt it, or re-use bits of it - not use it as-is.

e.g. normally block-based motion compensation is going to work better than global motion compensation. Depends on the content, but I think I'd try to use that part, at least.

FWIW I think it counts as super resolution - the source is clearly aliased, and NNEDI + motion compensated temporal smearing is a kind of poor man's super resolution - it's maybe 25%-40% of the real thing - but not 0%.

Cheers,
David.

[Didée]

@ Terka

Do you have a working copy of the famous Apply_PDF_Claims() filter? No? A pity! :)

For what Filip Sroubek can or can not do, the same restrictions as noted in the past do apply. His method relies on subtile subpixel-differences - the problem is that in real-world sources, those tiny differences are strongly distorted by compression. (You can safely assume that the distortions are many-times as big as the original usable subpixel-diff has been. Hence, probably unusable.) His method may work for showcases. For general practical appliance, chances are rather poor.
Also, his examples of "webcam video" are ... well, not particularly impressing. That we can do with NNEDI2+sharpening, too.

Modern nuclear science has shown that it is by all means possible to turn lead into gold. The old alchemist's dream has become true!!
- But wait, why isn't there any industrial appliance of that method? Because the result does not justify the effort.


@ 2Bdecided

Referring to the posted sample "Picture0013.avi", there is not much aliasing going on. (A little bit aliasing is needed for such small resolution to make it appear sharp. 320x240 without any aliasing would be not a picture, but a blurry mess.)
A bit of aliasing, yes, but not very much. In particular, no aliasing as in "decimated resolution".

***

Just remembered some sport that was done one year ago ...

source - http://img692.imageshack.us/img692/6448/sourcey.th.png (http://img692.imageshack.us/i/sourcey.png/) --- result - http://img62.imageshack.us/img62/1306/resulta.th.png (http://img62.imageshack.us/i/resulta.png/)

Not really earthshattering, it's just spatial interpolation (NNEDI2) combined with dedicated multistage sharpening. The good thing is that this is not a myth. It is real, and practically doable.

[aegisofrime]

source - http://img692.imageshack.us/img692/6448/sourcey.th.png (http://img692.imageshack.us/i/sourcey.png/) --- result - http://img62.imageshack.us/img62/1306/resulta.th.png (http://img62.imageshack.us/i/resulta.png/)

Not really earthshattering, it's just spatial interpolation (NNEDI2) combined with dedicated multistage sharpening. The good thing is that this is not a myth. It is real, and practically doable.

OMG! That's really amazing! :eek:

[2Bdecided]

@ 2Bdecided

Referring to the posted sample "Picture0013.avi", there is not much aliasing going on. (A little bit aliasing is needed for such small resolution to make it appear sharp. 320x240 without any aliasing would be not a picture, but a blurry mess.)
A bit of aliasing, yes, but not very much. In particular, no aliasing as in "decimated resolution".You think? I thought the rather random patterns appearing on the house in the background were exactly that (assuming it actually consists of horizontal white boards). I suspect the house is marginally more in focus than the trashed car.

FWIW the way these cameras work when capturing video - grabbing only every n-th pixel from a large sensor - is the very definition of decimated resolution IMO, assuming the image on the sensor is in-focus.

I've shot hours of video at 320x480 on 4MP, 6MP and 8MP digital still cameras over the years - plenty of severe aliasing!

Cheers,
David.

[Didée]

Those random patterns are just interference. Interference is not necessarily caused by dropping pixels. It can also appear with regular area-based sampling methods.
On the other hand, all straight edges appear more or less anti-aliased. And that's what I (mostly) based my opinion on.

Well, whatever. Looking at that source closely, my guts tell me that this source is hardly suited for temporal superresolution. (I've made best experiences by trusting my guts.)

[um3k]

Well, regardless, I went ahead and transformed it into a rather slow function. Let me know if you want an explanation of the arguments, I'm not going to bother typing one out if no one is interested. Keep in mind this is the first avisynth function I've created, and is mostly a result of shooting in the dark. But, I like the results it gives with certain sources.

# NoMoreMoire filter by um3k/Justin Phillips
# Based on a "Super Resolution" script by scharfis_brain
# Reduces moire, aliasing, noise, and compression artifacts
# in shaky, low-quality video from digital still cameras
# Alpha 1

function NoMoreMoire_alpha1(clip c, int "scale", int "r", int "freq", int "con", int "mmin", int "mmax", int "sthr", int "srad", string "inputlog", int "mode", bool "chroma")
{
scale = Default(scale, 2)
r = Default(r, 4)
freq = Default(freq, 3)
sym = 50
con = Default(con, 72)
mmin = Default(mmin, 63)
mmax = Default(mmax, 88)
sthr = Default(sthr, 0)
srad = Default(srad, 0)
mode = Default(mode, 0)
chroma = Default(chroma, true)

# for comparision
i = c.lanczos4resize(c.width*scale,c.height*scale).converttoyv12()

# for rendering
j = c.nnedi2_rpow2(scale, cshift="spline36resize").converttoyv12()

# Motion Comp
comp = (Defined(inputlog)==true) ? Eval("""
data = i
j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=15,inputlog=inputlog)
""") : Eval("""
data = i.depanestimate(range=2,pixaspect=1.0,zoommax=1,improve=false,trust=0)
j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=15)
""")

# Mask Creation
m = (chroma==true) ? Eval("""
m = mt_makediff(F2Quiver(comp, "gcl", freq, sym, degree=4, rescale=true, morph=false) ,F2Quiver(j, "gcl", freq, sym, degree=4, rescale=true, morph=false)\
.ChangeFPS(comp.Framerate), chroma="process").SmoothLevels(0+con, 1.000, 255-con, 0, 255, chroma=100, limiter=1, Lmode=1)
mc = Overlay(AbsDif(m.UToY()), AbsDif(m.VtoY()), opacity=1.0, mode="lighten", pc_range=false).BicubicResize(m.width,m.height)
m = AbsDif(m).Greyscale()
m = Overlay(m, mc, opacity=1.0, mode="lighten", pc_range=True)
m = mt_hysteresis(m.MT_deflate().MT_deflate().MT_deflate().MT_deflate().Medianblur(3).mt_binarize(mmax), m.mt_binarize(mmin)).mt_expand().Grayscale()
m
""") : Eval("""
m = mt_makediff(F2Quiver(comp, "gcl", freq, sym, degree=4, rescale=true, morph=false) ,F2Quiver(j, "gcl", freq, sym, degree=4, rescale=true, morph=false)\
.ChangeFPS(comp.Framerate), chroma="1").SmoothLevels(0+con, 1.000, 255-con, 0, 255, chroma=0, limiter=1, Lmode=1)
m = AbsDif(m).Greyscale()
m = mt_hysteresis(m.MT_deflate().MT_deflate().MT_deflate().MT_deflate().Medianblur(3).mt_binarize(mmax), m.mt_binarize(mmin)).mt_expand().Grayscale()
m
""")

# Shadow Mask
m = (sthr==0) ? Eval("""
m
""") : Eval("""
Overlay(m, comp.mt_expand().YLevels(sthr-srad, 1.000, sthr+srad, 0, 255).invert().mt_expand().mt_deflate().ChangeFPS(comp.Framerate), opacity=1.0,\
mode="lighten", greymask=true, ignore_conditional=false, pc_range=true).Grayscale()
""")

# Mask Visualization
mv = m.temporalsoften(r,255,255).selectevery(2*r+1,r)

# Merge Compensated with Original to Protect Motion
mix = mt_merge(comp, j.ChangeFPS(comp.Framerate), m, luma=true)

# The Important Part
temp = mix.temporalsoften(r,255,255)

# Back to normal framerate
dec = temp.selectevery(2*r+1,r)

# Decide output
out = (mode==2) ? Eval("""
Overlay(dec.Levels(0, 1.000, 255, 0, 127, coring=false), mv.Levels(0, 1.500, 255, 0, 255, coring=false), opacity=1.0, mode="add", greymask=true,\
ignore_conditional=false, pc_range=true)
""") : \
(mode==1) ? Eval("""
i
""") : Eval("""
dec
""")

return out
}

function AbsDif(clip c)
{
Overlay(c.YLevels(127, 1.000, 255, 0, 255), c.Invert().YLevels(127, 1.000, 255, 0, 255), opacity=1.0, mode="lighten", pc_range=True)
}

The scripting isn't pretty by any definition of the word, and I'm sure there is plenty room for optimization. But it does something, and for that, I'm happy.

If you omit the "inputlog" argument, it will default to depanestimate. However, I recommend creating a log in Deshaker with the slowest settings for the best results. "mode=1" will output lanczos-scaled footage, so you can feed that into VirtualDub, or you can do the scaling yourself.

Also, for best results, you might want to follow up by sharpening and adding some grain--the results tend to be somewhat soft and smooth, but they don't seem to have lost many details that weren't already destroyed by the compression.

[um3k]

Oh well - I thought maybe you could adapt it, or re-use bits of it - not use it as-is.


I'll take another look at it at some point--right now I'm seeing how far I can go with what I started with.

e.g. normally block-based motion compensation is going to work better than global motion compensation. Depends on the content, but I think I'd try to use that part, at least.

I haven't done extensive testing, but from what I've read and done, it seems MVTools doesn't have the accuracy I need--nor does DepanEstimate, hence my usage of a Deshaker log. However, I'm looking into using MV for the moving bits in order to give them some degree of enhancement, as opposed to the none they have currently. Well, nothing beyond NNEDI2, anyways.

[WorBry]

Just remembered some sport that was done one year ago ...

source - http://img692.imageshack.us/img692/6448/sourcey.th.png (http://img692.imageshack.us/i/sourcey.png/) --- result - http://img62.imageshack.us/img62/1306/resulta.th.png (http://img62.imageshack.us/i/resulta.png/)

Not really earthshattering, it's just spatial interpolation (NNEDI2) combined with dedicated multistage sharpening. The good thing is that this is not a myth. It is real, and practically doable.

Any chance of the/a script for that? :)

I'd like to try it on my aliasing-prone 'pseudo-progressive' (frame mode) DV material

[Didée]

Any chance of the/a script for that? :)
Here you go. (http://forum.gleitz.info/showthread.php?t=33835&p=383875) Just yesterday it was exactly one year ago. ;)

[WorBry]

Great, thanks. Havent had chance to play with it yet.

[aegisofrime]

um3k, I'm interesting in trying your script, however I'm getting the error that there is no Function for Ylevels, despite me having the Masktools dlls in my plugin folders, and manually loading them. :/

[um3k]

Do you have this? http://forum.doom9.org/showthread.php?t=79898

[aegisofrime]

OHHHH.

*Facepalms*

Right. I thought that Masktools was all that's needed. Thanks!

[um3k]

Ok, I've worked on it some more. There is now a brief description of the variables. They've changed, so if you've experimented with the parameters of the old version, they are probably not going to produce the expected result in the new version.

# NoMoreMoire filter by um3k/Justin Phillips
# Based on a "Super Resolution" script by scharfis_brain
# Reduces moire, aliasing, noise, and compression artifacts
# in shaky, low-quality video from digital still cameras
# Alpha 2

function NoMoreMoire_alpha2(clip c, int "mode", int "scale", int "r", int "lmshr", int "lmmed", int "lmmin", int "lmmax", int "cmshr", int "cmmed", int "cmthr", int "smthr", int "smrad", string "inputlog", bool "chroma")
{
# 0 = lanczos-scaled video (for DeShaker log creation), 1 = filtered video, 2 = scaled with mask overlay, 3 = processed with mask overlay
mode = Default(mode, 1)

# Determines whether the chroma is included in the mask calculations
chroma = Default(chroma, true)

# Scaling factor of the video. Not much advantage to going above 2.
scale = Default(scale, 2)

# Temporal radius of the blending
r = Default(r, 4)

# Luma mask adjustments.
lmshr = Default(lmshr, 5)
lmmed = Default(lmmed, 3)
lmmin = Default(lmmin, 25)
lmmax = Default(lmmax, 70)

# Chroma mask adjustments
cmshr = Default(cmshr, 12)
cmmed = Default(cmmed, 8)
cmthr = Default(cmthr, 20)

# Shadow mask adjusments. Smthr controls the cutoff point, smrad controls the feathering.
smthr = Default(smthr, 27)
smrad = Default(smrad, 3)

####################################################################################
# Now stuff happens.

# for comparision
i = c.lanczos4resize(c.width*scale,c.height*scale).converttoyv12()

# for rendering
j = c.nnedi2_rpow2(scale, cshift="spline36resize").converttoyv12()

# Motion Comp
comp = (Defined(inputlog)==true) ? Eval("""
data = i
j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=0,inputlog=inputlog)
""") : Eval("""
data = i.depanestimate(range=2,pixaspect=1.0,zoommax=1,improve=false,trust=0)
j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=0)
""")

# Mask Creation
diff = (chroma==true) ? mt_makediff(comp, j.ChangeFPS(comp.Framerate), chroma="process").mt_lut("x 128 - abs 2 *", chroma="process")
\ : mt_makediff(comp, j.ChangeFPS(comp.Framerate), chroma="1").mt_lut("x 128 - abs 2 *", chroma="1")

# Luma Mask / Processing
lumask = diff
lms = lumask.DeflateX(count=lmshr).Medianblur(lmmed).mt_binarize(lmmax)
lmb = lumask.mt_binarize(lmmin)
lmproc = mt_hysteresis(lms, lmb).mt_expand()

# Chroma Mask / Processing
chromask = Overlay(diff.UToY(), diff.VtoY(), opacity=1.0, mode="lighten", pc_range=true).BicubicResize(lumask.width,lumask.height)
cmproc = chromask.DeflateX(count=cmshr).Medianblur(cmmed).mt_binarize(cmthr)#.mt_expand()

# Motion Mask
momask = (chroma==true) ? Overlay(lmproc, cmproc, opacity=1.0, mode="lighten", pc_range=True)
\ : lmproc

# Shadow Mask
finmask = (smthr==0) ? momask
\ : Overlay(momask, comp.mt_expand().YLevels(smthr-smrad, 1.000, smthr+smrad, 0, 255).invert().mt_expand().mt_deflate().ChangeFPS(comp.Framerate), opacity=1.0, mode="lighten", pc_range=true).Grayscale()

# Mask Visualization
maskvis = finmask.temporalsoften(r,255,255).selectevery(2*r+1,r).Greyscale()

# Merge Compensated with Original to Protect Motion
mix = mt_merge(comp, j.ChangeFPS(comp.Framerate), finmask, luma=true)

# The Important Part
temp = mix.temporalsoften(r,255,255)

# Back to normal framerate
dec = temp.selectevery(2*r+1,r)

# Decide output
out = (mode==3) ? Overlay(dec.YLevels(16, 1.000, 235, 16, 127), maskvis.YLevels(0, 1.000, 255, 16, 235), opacity=1.0, mode="add", pc_range=false)
\ : (mode==2) ? Overlay(i.YLevels(16, 1.000, 235, 16, 127), maskvis.YLevels(0, 1.000, 255, 16, 235), opacity=1.0, mode="add", pc_range=false)
\ : (mode==1) ? dec
\ : i

#~ return out
return out
}

function DeflateX(clip c, int "count", bool "strict") {
strict = Default(strict, true)
Assert((strict ? count >= 0 : true), "DeflateX: 'count' cannot be negative")
return count > 0 ? c.MT_deflate().DeflateX(count=count - 1) : c
}
I've also cleaned up the scripting a fair bit, making it somewhat easier to follow. Performance seems to be similar to the previous version, i.e. slow. Please let me know if you spot any mistakes in the script, or see something that can be optimized. I'm learning as I go along.

EDIT: As an experiment, I went ahead and created a MT version of this script, which can be downloaded here (https://docs.google.com/leaf?id=0BxfsQhrtbYrqNGRiMzRhZGMtYTc1MC00OTlmLWI2MmItNWMzMWU3ODZiMjFj&hl=en). It gives me a 30% speed increase on my hyperthreaded Pentium 4.
EDIT2: I replaced the MT link with a non-broken script.

[um3k]

Update: I've been working on implementing block-based motion compensation to the next version, but have hit a snag. MVTools tends to apply compensation to the moire artifacts, all but nullifying the main goal of the filter. I've decided to work on implementing other filters while I ponder a solution to this problem, the first one being posted here: http://forum.doom9.org/showthread.php?p=1374487#post1374487

[Undead Sega]

http://img692.imageshack.us/img692/6448/sourcey.th.png[/IMG] (http://img692.imageshack.us/i/sourcey.png/) --- result - http://img62.imageshack.us/img62/1306/resulta.th.png (http://img62.imageshack.us/i/resulta.png/)

Not really earthshattering, it's just spatial interpolation (NNEDI2) combined with dedicated multistage sharpening. The good thing is that this is not a myth. It is real, and practically doable.

Is this for real?

Has anyone here actually tried this but with different sources? Please let me know, i am interested to know how this works exactly :D

[um3k]

Here's the third alpha. I'm not really sure when to start calling it beta, or finished. It should give reasonable results on the default settings. Let me know how well (or if) it works for your footage. Keep in mind, it has the greatest effect on small (320x240), aliased, overcompressed footage, of the sort output by compact digital still cameras in video mode.

Link (https://docs.google.com/leaf?id=0BxfsQhrtbYrqYTE0NzIxYTMtMGU5ZC00M2MyLTliNjMtZThkZWQ1NjE2YTcz&hl=en) (because the darn thing is too big to fit in this post)

Please let me know if the link doesn't work for you.

EDIT: Note, I've added block-based motion compensation, which is now the default mode.

[AnnaFan777]

Don't know if you can extract "extra" details from "shaky" videos :)

But the SR examples look good..

http://neuron2.net/library/icip99.pdf

http://www.thedeemon.com/VideoEnhancer/

[2Bdecided]

Didée,

Do you remember exactly how you used this script...
Here you go. (http://forum.gleitz.info/showthread.php?t=33835&p=383875) Just yesterday it was exactly one year ago. ;)

...to produce this example...

Just remembered some sport that was done one year ago ...

source - http://img692.imageshack.us/img692/6448/sourcey.th.png (http://img692.imageshack.us/i/sourcey.png/) --- result - http://img62.imageshack.us/img62/1306/resulta.th.png (http://img62.imageshack.us/i/resulta.png/)

Not really earthshattering, it's just spatial interpolation (NNEDI2) combined with dedicated multistage sharpening. The good thing is that this is not a myth. It is real, and practically doable.

...because I can't figure out what order you did that in.

Using NNEDI2 (twice; both dimensions!) then that script gives a large image - if that's what you did, then I can't figure out how you downscaled it to 800x600 while maintaining such sharpness.

I've tried scaling before or after the sharpening - I've even tried sharpening, scaling, then sharpening again. I can't make it match your result at all - I either get a softer result, or more halos.

I know this stuff comes easy to you, but some of us mortals struggle to keep up - what did you do?!

Cheers,
David.

[Boulotaur2024]

I was wondering that too :p
(sorry for the "thread hijacking" um3k)

[2Bdecided]

The closest I can get is by using Didée's script to process the double res (NNEDI'd) image, then using sssresize to drop it to 800x600.

But the result still isn't as good. There are features in Didée's result which are simply blurred in mine.

In fact, with sssresize on the end, it's better if you don't use Didée's script in the middle - NNEDI then sssresize is pretty good (though still not as good as Didée's result!).


Now I'm wondering if this old example from Didée was an April Fool's joke on the wrong date!

It's either that, or there's another trick that I'm missing.

Cheers,
David.

P.S. sssresize is this...

function sssresize(clip clp, int "dx", int "dy"){
cp2=clp.spline36resize(dx*2,dy*2).converttoyv12()# automttap3 works best when resizing by factors of 2, though it can resize by any factor aslong as at no point the image is less then 144*144 pizels
cp3=cp2.limitedsharpenfaster(smode=2, strength=400, ss_x=1, ss_y=1)#modified to use a different sharpening kernel
cp4=cp3.sssharp(denoise=0, iter=2)#in the avisynth development forum, absurdly slow sharpener
cp5=cp4.automttap3(dx, dy)#a sub function of an interpolation script, uses backprojection to adaptively choose the number of taps to use on a per source pixel pasis, avoidsmost ringing while otherwise performing similarly to high tap lanczos
return(cp5)}which came from *.mp4 guy here: http://85.230.118.163/showthread.php?p=1211873#post1211873 (see that post for dependencies) but this is insanely slow - it makes tgmc seem like bob() speed in comparison!

[Undead Sega]

I think the sharpening filter that's used was Super Slow Sharpen (SSSharpen), that apparently being the multistage sharpening filter. As for resize, I would either use Spline36resize or Blackmanresize for trying to maintain as much sharpness and detail. Try it out :D

[Didée]

I think the sharpening filter that's used was Super Slow Sharpen (SSSharpen)
Sure as anything, that is not the case. Simple reasoning: I never have used SSSharp (except for testing and finding it's too slow).
It was Archimedes who used SSSharpen to come up with a result challenging mine ... a tiny bit better, but 10-fold slower.

I can't manage to trace back what exactly was happening on that one evening one year ago - the scripts have mutated later on (without backup), and the migration to a new PC isn't exactly helpful on the matter, either.

But if you guys take on your glasses and your common sense, then it's pretty obvious that I must've made a mistake! - See:

Examine e.g. this region:
http://img573.imageshack.us/img573/7576/roiz.jpg (http://img573.imageshack.us/i/roiz.jpg/)

640x480 picture:
http://img52.imageshack.us/img52/4231/640x480y.png (http://img52.imageshack.us/i/640x480y.png/)

800x600 picture:
http://img411.imageshack.us/img411/1475/800x600.png (http://img411.imageshack.us/i/800x600.png/)


Look at the vertical window bars to the left, and at the horizontal wall texture to the right.

It is impossible to enhance the 640 picture's detail level to that what the 800 picture is showing.


Conclusion: what happened was a plain sharpening task of an 800x600 picture. No upscaling has been done in that example.

[um3k]

Now that Didée has debunked himself, I'll just leave this here.

Latest (though not particularly recent) version of NoMoreMoire (http://www.stupidideasstudios.com/NoMoreMoire_a4.avsi)

[2Bdecided]

But if you guys take on your glasses and your common sense, then it's pretty obvious that I must've made a mistake!Oh, somebody already spotted that...
http://forum.videohelp.com/threads/328291-re-A-little-balance-please?p=2035431&viewfull=1#post2035431

You've got to give that person credit - they found another different picture of the same building to prove that those were real details in the larger image.

Cheers,
David.

[2Bdecided]

Now that Didée has debunked himself, I'll just leave this here.

Latest (though not particularly recent) version of NoMoreMoire (http://www.stupidideasstudios.com/NoMoreMoire_a4.avsi)I've got to try that. Am I counting this correctly - 13 frames averaged?!

You could add it to this thread, which seems like a kind of SR repository...
http://forum.doom9.org/showthread.php?t=154141
...though the sample video in that thread isn't useful for demonstrating SR IMO.

Cheers,
David.

[um3k]

I've got to try that. Am I counting this correctly - 13 frames averaged?!
It's adjustable. I think the default is 9.

I'm no Didée, my solution isn't exactly elegant. It more or less beats the artifacts into submission. I've only ever tested it on footage from my Canon Powershot G3, it seems to work pretty decently. There are indeed places where it succeeds in recovering detail from moire. And it certainly does not fail at stabilizing detail, much of the time.

Looking at my scripts, here are the arguments I normally use for testing:

NoMoreMoire_alpha4(j4i=true, mcmode="hybrid")

Keep in mind that this is intended for a very specific type of video--shaky, low-resolution, heavily aliased footage from a digicam. It attempts to remove aliasing and moire while preserving real details--which are generally lacking to begin with. I advise following up with a bit of grain for a more natural looking image. I made a script once to motion-compensate and slow down the grain to make it look like details on the image, but I can't locate it.

Anyways, that's enough rambling from me. Let me know what you think of the (glacial) output.

[Dogway]

I tested with a small clip and result was good, but there's a lot of frameblending, also I had to change grayscale, with greyscale.

[Gavino]

I had to change grayscale, with greyscale.
I don't see why.
GrayScale and GreyScale are both accepted by Avisynth, with the same meaning (although the docs only mention the latter).

[Dogway]

Not mine, only greyscale works. Im using MT Avisynth 2.5.7 : / maybe the version?

[Gavino]

Yes, you're right.
GrayScale was added as an alternative to GreyScale in v2.58.

[um3k]

Hm, frameblending, you say? Is there any chance you could upload an unprocessed sample clip (and post the parameters used) so I can try and get to the root of the problem? I haven't noticed any framblending in the assortment of test footage I've tried, but it doesn't exactly span the spectrum.

[Dogway]

It creates frameblending on this and another small crappy footages. I just call the function without flags.

http://www.sendspace.com/file/itj13s

[um3k]

Ah...I've never used it before on animated footage, that's not really its intended use. I'm trying to take a look at it, but my computer is being difficult. I'll let you know if I get it working.

[Dogway]

Well, yes, it's not shaky, I was just looking for a Temporal Super Resolution solution for small sources, after finding this thread (http://forum.doom9.org/showthread.php?p=808013#post808013) but the code is old.

[xbox360]

Here you go. (http://forum.gleitz.info/showthread.php?t=33835&p=383875) Just yesterday it was exactly one year ago. ;)

src=last
base=src.HPblur().HPblur()#.HPblur()
head=base.mt_adddiff(mt_makediff(base,base.removegrain(4,-1)))
shrp=base.mt_adddiff(mt_makediff(base,base.removegrain(11,-1)))
lim=shrp.repair(head,1)
shrpL=src.mt_adddiff(mt_makediff(lim,base),U=2,V=2)
shrpLL=shrpL.mt_adddiff(mt_makediff(lim,base),U=2,V=2)
shrpLLD=mt_makediff(src,shrpLL)
shrpLLx=src.mt_lutxy(shrpLLD,"x y 128 - abs 15 / 1 2 / ^ 15 * y 128 - 2 ^ y 128 - 2 ^ 2.3 + / * y 128 - y 128 - abs 0.0001 + / * -",U=2,V=2)
shrpLLxD=mt_makediff(shrpLLx,src)
backglow=shrpLLx.mt_makediff(shrpLLxD.removegrain(11,-1).removegrain(11,-1),U=2,V=2).mt_makediff(shrpLLxD.removegrain(11,-1),U=2,V=2) .mergeluma(shrpLLx,0.618)
return shrpLLx # or backglow

function HPblur(clip o) {
rg11=o.removegrain(11)
rg11D=mt_makediff(o,rg11)
xD=mt_lutxy(rg11D,rg11D.removegrain(11,-1),"x 128 - y 128 - * 0 < x y + 128 - x 128 - abs y 128 - abs < x y ? ?")
o.mt_makediff(xD,U=2,V=2) }

Hi can you tell me how do I use the script ? is this an avsi script ?, where do I put the output resolution I want ?

Also I have seen this thing being used in the Men In Black BluRay movie.

http://www.blu-ray.com/movies/Men-in-Black-Blu-ray/748/

[Didée]

Hi can you tell me how do I use the script ? is this an avsi script ?
Either place it directly in your script, after loading the source. Or make a function out of it:
function MickeyMouse(clip src)
{
base=src.HPblur().HPblur()#.HPblur()
head=base.mt_adddiff(mt_makediff(base,base.removegrain(4,-1)))
shrp=base.mt_adddiff(mt_makediff(base,base.removegrain(11,-1)))
lim=shrp.repair(head,1)
shrpL=src.mt_adddiff(mt_makediff(lim,base),U=2,V=2)
shrpLL=shrpL.mt_adddiff(mt_makediff(lim,base),U=2,V=2)
shrpLLD=mt_makediff(src,shrpLL)
shrpLLx=src.mt_lutxy(shrpLLD,"x y 128 - abs 15 / 1 2 / ^ 15 * y 128 - 2 ^ y 128 - 2 ^ 2.3 + / * y 128 - y 128 - abs 0.0001 + / * -",U=2,V=2)
shrpLLxD=mt_makediff(shrpLLx,src)
backglow=shrpLLx.mt_makediff(shrpLLxD.removegrain(11,-1).removegrain(11,-1),U=2,V=2).mt_makediff(shrpLLxD.removegrain(11,-1),U=2,V=2) .mergeluma(shrpLLx,0.618)
return shrpLLx # or backglow
}

function HPblur(clip o) {
rg11=o.removegrain(11)
rg11D=mt_makediff(o,rg11)
xD=mt_lutxy(rg11D,rg11D.removegrain(11,-1),"x 128 - y 128 - * 0 < x y + 128 - x 128 - abs y 128 - abs < x y ? ?")
o.mt_makediff(xD,U=2,V=2) }


where do I put the output resolution I want ?
Nowhere. It's just a sharpening script. Input resolution == output resolution. (When you use "sharpen(1)", where do you set the output resolution?)

Also I have seen this thing being used in the Men In Black BluRay movie.
What?

[2Bdecided]

Now that Didée has debunked himself, I'll just leave this here.

Latest (though not particularly recent) version of NoMoreMoire (http://www.stupidideasstudios.com/NoMoreMoire_a4.avsi)Just wanted to let you know that I've been using this. It kept crashing with the dreaded " CAVIStreamSynth: System exception - Access Violation" message, and no amount of memory tweaking etc would fix it. Once I tried your suggested parameters, it crashed VirtualDub with an error in mt_masktools.dll instead. Traced it to an old version of mt_masktools.dll in my plugins folder. Removed it. Now it works, but is flaky. Suspect there's still a plug-in or memory problem.

Still trying to figure out what balance of removing junk and keeping details pleases me best. Stills are one thing, but when watching moving video, I find more details (+ more junk) is slightly preferable to the really smooth image that I prefer when looking at a still.

Cheers,
David.

[Bloax]

Just wanted to let you know that I've been using this. It kept crashing with the dreaded " CAVIStreamSynth: System exception - Access Violation" message, and no amount of memory tweaking etc would fix it.
Sounds like the issues I've been having with MFlow for some reason.
Do try and make a dummy script to see if that's what's causing it.

[2Bdecided]

Do try and make a dummy script to see if that's what's causing it.Thanks Bloax. What do you mean by a dummy script?

FWIW I have other scripts using MVtools (1 and 2 - different scripts!) which work just fine. I even tried one yesterday to check nothing had changed.

Seems a bit random, but I guess it can't be really.

Cheers,
David.