Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
6th February 2010, 19:22 | #1 | Link |
Registered User
Join Date: May 2007
Posts: 220
|
Temporal-Spatial Anti-Aliasing for Shaky Digicam Videos
Some of you may know about scharfis_brain's long-ago attempt at temporal-spatial super resolution. http://forum.doom9.org/showthread.ph...solution+depan
It's certainly the most successful attempt I've seen at avisynth-based super resolution. In a quest to improve some low-quality digicam videos, I've crudely modified his script to preserve differential motion. Code:
AVISource("Picture0013.AVI") ConvertToYUY2() # for comparision i=lanczos4resize(640,480).converttoyv12() # for rendering #~ j=pointresize(640,480).converttoyv12() j=nnedi2_rpow2(2).converttoyv12() data=i.depanestimate(range=2,pixaspect=1.0,zoommax=1,improve=false,trust=0) r=4 j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=15,inputlog="0013.log") m=mt_makediff(last,j.ChangeFPS(last.Framerate)).Grayscale() m=Overlay(m, m.Invert(), x=0, y=0, opacity=1.0, mode="lighten", greymask=true, ignore_conditional=false, pc_range=True).Levels(145, 1.000, 160, 0, 255, coring=false) mt_merge(last, j.ChangeFPS(last.Framerate), m, luma=true) temporalsoften(r,255,255) selectevery(2*r+1,r) #~ Unsharp(varY=1.0,strength=2) stackhorizontal(last.subtitle("temporal super resolution"),i.subtitle("original (320x240)")) 1. I've replaced the pointresize with NNEDI2. The pointresize probably would be a little sharper, but it doesn't get along very well with my crude masking. 2. I've replaced the DepanEstimate results with a Deshaker log, which I found to result in more accurate correction. Here is a still example of its efficacy. Notice that there is a huge reduction in JPEG artifacts, as well as aliasing, particularly on the house in the background. (Click for full size) For comparison, here is the output of scharfis_brain's original script: Unfortunately, there are still motion artifacts, primarily ghosting on differential motion, as well as some chroma artifacts. I'll continue working on it, but am not sure how far I'll be able to go. And, of course, there is the additional limitation of differential motion not receiving any enhancement. If you want to take a stab at it, or just see the results for yourself, here is the original video file as well as the log file: Picture0013.AVI 0013.log So, take a look, and toss in your two cents. Last edited by um3k; 8th February 2010 at 20:24. Reason: Changed Title |
8th February 2010, 17:48 | #2 | Link |
Formerly davidh*****
Join Date: Jan 2004
Posts: 2,496
|
Code:
j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=15,inputlog="0013.log") Edit: Doh, it's right there in the post David Last edited by wonkey_monkey; 8th February 2010 at 20:02. |
8th February 2010, 17:55 | #4 | Link |
Registered User
Join Date: Dec 2002
Location: UK
Posts: 1,673
|
I've used this in the past...
PHP Code:
http://forum.doom9.org/showthread.ph...569#post812569 ...which looks like a newer / better version of the one you started with. I've tried various attempts to mask badly predicted and unpredictable movement and not merge those parts. Nothing worth sharing - and glacially slow! TBH, it's nothing like real super resolution (I think we all know this) - I have clips with fine details that flicker horribly - real super resolution would fix these nicely. Cheers, David. |
8th February 2010, 18:05 | #5 | Link |
Registered User
Join Date: May 2007
Posts: 220
|
I just gave those both a try. The script in your post didn't create motion artifacts, but it also didn't really improve the image much. The script in the link caused horrific motion artifacts, and still didn't provide as much improvement as the current version of the script I posted. So I'll keep trying to improve what I've got here. And true, it isn't really super resolution--but it does make it easier to watch.
|
8th February 2010, 19:01 | #7 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
"real super resolution" - the myth that some enterprising people claim it would exist.
It has been discussed so often. Search for "(temporal) superresolution" and {my name}, or by MfA as well, who IIRC in the past has elaborated on that topic too. Short version: "superresolution" is basically possible when the input is aliased due to decimated resolution. Motion compensated (Bob-) Deinterlacers are one special case where TSR is actually working. But for the typical, average, non-aliased input with non-decimated resolution, there is no super-resolution to achieve. Also you script has very little to do with super-resolution. The biggest worker is NNEDI2, which softens edge-aliasing. Then you hit after it with some rather weak, rather cautious temporal averaging filter that mostly works in areas with very weak signal. NNEDI2 on its own is not a true superresolution filter - it does not bring any new information into the frame, it "only" interpolates edge oriented. Smoother edges, but no new information. The next step - temporal averaging only of areas that are highly similar anyway - is no superresolution either. Superresolution would require to appropriately use adjacent information that is quite different to the current information. (Don't judge just by the housewall texture - result appears nicer, yes, but to some extend that's just lucky coincidence. E.g.: on the trees in the background there is apparent *loss* of detail. On the house's chimney there is apparent *loss* of detail. How comes, since both parts are without 'incremental' motion, in the same way as the house wall is?) What your script does is upsizing with smooth edges, plus some denoising. There's nothing wrong with that. If the result seems pleasing, then it's fine. But better don't call that "super resolution". Because it isn't.
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
8th February 2010, 20:49 | #8 | Link |
Registered User
Join Date: May 2007
Posts: 220
|
Ok, I changed the name of the thread. It now better reflects the function of the script, to improve temporal stability by selectively averaging frames to reduce aliasing, moire, and compression artifacts, for the specific case of low-resolution, shaky video produced by a handheld digicam.
Last edited by um3k; 8th February 2010 at 20:55. |
10th February 2010, 10:42 | #10 | Link | |
Registered User
Join Date: Dec 2002
Location: UK
Posts: 1,673
|
Quote:
No, I meant the mathematical algorithms proposed in a few papers (see earlier super resolution threads here on doom9) which work in the specific cases Didée mentioned - and certainly we have such a case here. Problem is the maths assumes that the underlying scene does not change - only the camera moves. That's fair enough for background details, but useless and someone walks across the frame swinging their arms! Cheers, David. |
|
10th February 2010, 10:48 | #11 | Link | |
Registered User
Join Date: Dec 2002
Location: UK
Posts: 1,673
|
Quote:
e.g. normally block-based motion compensation is going to work better than global motion compensation. Depends on the content, but I think I'd try to use that part, at least. FWIW I think it counts as super resolution - the source is clearly aliased, and NNEDI + motion compensated temporal smearing is a kind of poor man's super resolution - it's maybe 25%-40% of the real thing - but not 0%. Cheers, David. |
|
10th February 2010, 13:02 | #12 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
@ Terka
Do you have a working copy of the famous Apply_PDF_Claims() filter? No? A pity! For what Filip Sroubek can or can not do, the same restrictions as noted in the past do apply. His method relies on subtile subpixel-differences - the problem is that in real-world sources, those tiny differences are strongly distorted by compression. (You can safely assume that the distortions are many-times as big as the original usable subpixel-diff has been. Hence, probably unusable.) His method may work for showcases. For general practical appliance, chances are rather poor. Also, his examples of "webcam video" are ... well, not particularly impressing. That we can do with NNEDI2+sharpening, too. Modern nuclear science has shown that it is by all means possible to turn lead into gold. The old alchemist's dream has become true!! - But wait, why isn't there any industrial appliance of that method? Because the result does not justify the effort. @ 2Bdecided Referring to the posted sample "Picture0013.avi", there is not much aliasing going on. (A little bit aliasing is needed for such small resolution to make it appear sharp. 320x240 without any aliasing would be not a picture, but a blurry mess.) A bit of aliasing, yes, but not very much. In particular, no aliasing as in "decimated resolution". *** Just remembered some sport that was done one year ago ... source - --- result - Not really earthshattering, it's just spatial interpolation (NNEDI2) combined with dedicated multistage sharpening. The good thing is that this is not a myth. It is real, and practically doable.
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
10th February 2010, 15:31 | #14 | Link | |
Registered User
Join Date: Dec 2002
Location: UK
Posts: 1,673
|
Quote:
FWIW the way these cameras work when capturing video - grabbing only every n-th pixel from a large sensor - is the very definition of decimated resolution IMO, assuming the image on the sensor is in-focus. I've shot hours of video at 320x480 on 4MP, 6MP and 8MP digital still cameras over the years - plenty of severe aliasing! Cheers, David. |
|
10th February 2010, 15:45 | #15 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
Those random patterns are just interference. Interference is not necessarily caused by dropping pixels. It can also appear with regular area-based sampling methods.
On the other hand, all straight edges appear more or less anti-aliased. And that's what I (mostly) based my opinion on. Well, whatever. Looking at that source closely, my guts tell me that this source is hardly suited for temporal superresolution. (I've made best experiences by trusting my guts.)
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
10th February 2010, 17:06 | #16 | Link |
Registered User
Join Date: May 2007
Posts: 220
|
Well, regardless, I went ahead and transformed it into a rather slow function. Let me know if you want an explanation of the arguments, I'm not going to bother typing one out if no one is interested. Keep in mind this is the first avisynth function I've created, and is mostly a result of shooting in the dark. But, I like the results it gives with certain sources.
Code:
# NoMoreMoire filter by um3k/Justin Phillips # Based on a "Super Resolution" script by scharfis_brain # Reduces moire, aliasing, noise, and compression artifacts # in shaky, low-quality video from digital still cameras # Alpha 1 function NoMoreMoire_alpha1(clip c, int "scale", int "r", int "freq", int "con", int "mmin", int "mmax", int "sthr", int "srad", string "inputlog", int "mode", bool "chroma") { scale = Default(scale, 2) r = Default(r, 4) freq = Default(freq, 3) sym = 50 con = Default(con, 72) mmin = Default(mmin, 63) mmax = Default(mmax, 88) sthr = Default(sthr, 0) srad = Default(srad, 0) mode = Default(mode, 0) chroma = Default(chroma, true) # for comparision i = c.lanczos4resize(c.width*scale,c.height*scale).converttoyv12() # for rendering j = c.nnedi2_rpow2(scale, cshift="spline36resize").converttoyv12() # Motion Comp comp = (Defined(inputlog)==true) ? Eval(""" data = i j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=15,inputlog=inputlog) """) : Eval(""" data = i.depanestimate(range=2,pixaspect=1.0,zoommax=1,improve=false,trust=0) j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=15) """) # Mask Creation m = (chroma==true) ? Eval(""" m = mt_makediff(F2Quiver(comp, "gcl", freq, sym, degree=4, rescale=true, morph=false) ,F2Quiver(j, "gcl", freq, sym, degree=4, rescale=true, morph=false)\ .ChangeFPS(comp.Framerate), chroma="process").SmoothLevels(0+con, 1.000, 255-con, 0, 255, chroma=100, limiter=1, Lmode=1) mc = Overlay(AbsDif(m.UToY()), AbsDif(m.VtoY()), opacity=1.0, mode="lighten", pc_range=false).BicubicResize(m.width,m.height) m = AbsDif(m).Greyscale() m = Overlay(m, mc, opacity=1.0, mode="lighten", pc_range=True) m = mt_hysteresis(m.MT_deflate().MT_deflate().MT_deflate().MT_deflate().Medianblur(3).mt_binarize(mmax), m.mt_binarize(mmin)).mt_expand().Grayscale() m """) : Eval(""" m = mt_makediff(F2Quiver(comp, "gcl", freq, sym, degree=4, rescale=true, morph=false) ,F2Quiver(j, "gcl", freq, sym, degree=4, rescale=true, morph=false)\ .ChangeFPS(comp.Framerate), chroma="1").SmoothLevels(0+con, 1.000, 255-con, 0, 255, chroma=0, limiter=1, Lmode=1) m = AbsDif(m).Greyscale() m = mt_hysteresis(m.MT_deflate().MT_deflate().MT_deflate().MT_deflate().Medianblur(3).mt_binarize(mmax), m.mt_binarize(mmin)).mt_expand().Grayscale() m """) # Shadow Mask m = (sthr==0) ? Eval(""" m """) : Eval(""" Overlay(m, comp.mt_expand().YLevels(sthr-srad, 1.000, sthr+srad, 0, 255).invert().mt_expand().mt_deflate().ChangeFPS(comp.Framerate), opacity=1.0,\ mode="lighten", greymask=true, ignore_conditional=false, pc_range=true).Grayscale() """) # Mask Visualization mv = m.temporalsoften(r,255,255).selectevery(2*r+1,r) # Merge Compensated with Original to Protect Motion mix = mt_merge(comp, j.ChangeFPS(comp.Framerate), m, luma=true) # The Important Part temp = mix.temporalsoften(r,255,255) # Back to normal framerate dec = temp.selectevery(2*r+1,r) # Decide output out = (mode==2) ? Eval(""" Overlay(dec.Levels(0, 1.000, 255, 0, 127, coring=false), mv.Levels(0, 1.500, 255, 0, 255, coring=false), opacity=1.0, mode="add", greymask=true,\ ignore_conditional=false, pc_range=true) """) : \ (mode==1) ? Eval(""" i """) : Eval(""" dec """) return out } function AbsDif(clip c) { Overlay(c.YLevels(127, 1.000, 255, 0, 255), c.Invert().YLevels(127, 1.000, 255, 0, 255), opacity=1.0, mode="lighten", pc_range=True) } If you omit the "inputlog" argument, it will default to depanestimate. However, I recommend creating a log in Deshaker with the slowest settings for the best results. "mode=1" will output lanczos-scaled footage, so you can feed that into VirtualDub, or you can do the scaling yourself. Also, for best results, you might want to follow up by sharpening and adding some grain--the results tend to be somewhat soft and smooth, but they don't seem to have lost many details that weren't already destroyed by the compression. Last edited by um3k; 10th February 2010 at 17:24. |
10th February 2010, 17:40 | #17 | Link | ||
Registered User
Join Date: May 2007
Posts: 220
|
Quote:
Quote:
|
||
10th February 2010, 17:44 | #18 | Link | |
Registered User
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,197
|
Quote:
I'd like to try it on my aliasing-prone 'pseudo-progressive' (frame mode) DV material
__________________
Nostalgia's not what it used to be Last edited by WorBry; 10th February 2010 at 17:46. |
|
10th February 2010, 19:26 | #19 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
Here you go. Just yesterday it was exactly one year ago.
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
Thread Tools | Search this Thread |
Display Modes | |
|
|