Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
|
6th February 2010, 19:22 | #1 | Link |
Registered User
Join Date: May 2007
Posts: 220
|
Temporal-Spatial Anti-Aliasing for Shaky Digicam Videos
Some of you may know about scharfis_brain's long-ago attempt at temporal-spatial super resolution. http://forum.doom9.org/showthread.ph...solution+depan
It's certainly the most successful attempt I've seen at avisynth-based super resolution. In a quest to improve some low-quality digicam videos, I've crudely modified his script to preserve differential motion. Code:
AVISource("Picture0013.AVI") ConvertToYUY2() # for comparision i=lanczos4resize(640,480).converttoyv12() # for rendering #~ j=pointresize(640,480).converttoyv12() j=nnedi2_rpow2(2).converttoyv12() data=i.depanestimate(range=2,pixaspect=1.0,zoommax=1,improve=false,trust=0) r=4 j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=15,inputlog="0013.log") m=mt_makediff(last,j.ChangeFPS(last.Framerate)).Grayscale() m=Overlay(m, m.Invert(), x=0, y=0, opacity=1.0, mode="lighten", greymask=true, ignore_conditional=false, pc_range=True).Levels(145, 1.000, 160, 0, 255, coring=false) mt_merge(last, j.ChangeFPS(last.Framerate), m, luma=true) temporalsoften(r,255,255) selectevery(2*r+1,r) #~ Unsharp(varY=1.0,strength=2) stackhorizontal(last.subtitle("temporal super resolution"),i.subtitle("original (320x240)")) 1. I've replaced the pointresize with NNEDI2. The pointresize probably would be a little sharper, but it doesn't get along very well with my crude masking. 2. I've replaced the DepanEstimate results with a Deshaker log, which I found to result in more accurate correction. Here is a still example of its efficacy. Notice that there is a huge reduction in JPEG artifacts, as well as aliasing, particularly on the house in the background. (Click for full size) For comparison, here is the output of scharfis_brain's original script: Unfortunately, there are still motion artifacts, primarily ghosting on differential motion, as well as some chroma artifacts. I'll continue working on it, but am not sure how far I'll be able to go. And, of course, there is the additional limitation of differential motion not receiving any enhancement. If you want to take a stab at it, or just see the results for yourself, here is the original video file as well as the log file: Picture0013.AVI 0013.log So, take a look, and toss in your two cents. Last edited by um3k; 8th February 2010 at 20:24. Reason: Changed Title |
8th February 2010, 17:48 | #2 | Link |
Formerly davidh*****
Join Date: Jan 2004
Posts: 2,493
|
Code:
j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=15,inputlog="0013.log") Edit: Doh, it's right there in the post David Last edited by wonkey_monkey; 8th February 2010 at 20:02. |
8th February 2010, 17:55 | #4 | Link |
Registered User
Join Date: Dec 2002
Location: UK
Posts: 1,673
|
I've used this in the past...
PHP Code:
http://forum.doom9.org/showthread.ph...569#post812569 ...which looks like a newer / better version of the one you started with. I've tried various attempts to mask badly predicted and unpredictable movement and not merge those parts. Nothing worth sharing - and glacially slow! TBH, it's nothing like real super resolution (I think we all know this) - I have clips with fine details that flicker horribly - real super resolution would fix these nicely. Cheers, David. |
8th February 2010, 18:05 | #5 | Link |
Registered User
Join Date: May 2007
Posts: 220
|
I just gave those both a try. The script in your post didn't create motion artifacts, but it also didn't really improve the image much. The script in the link caused horrific motion artifacts, and still didn't provide as much improvement as the current version of the script I posted. So I'll keep trying to improve what I've got here. And true, it isn't really super resolution--but it does make it easier to watch.
|
10th February 2010, 10:48 | #6 | Link | |
Registered User
Join Date: Dec 2002
Location: UK
Posts: 1,673
|
Quote:
e.g. normally block-based motion compensation is going to work better than global motion compensation. Depends on the content, but I think I'd try to use that part, at least. FWIW I think it counts as super resolution - the source is clearly aliased, and NNEDI + motion compensated temporal smearing is a kind of poor man's super resolution - it's maybe 25%-40% of the real thing - but not 0%. Cheers, David. |
|
10th February 2010, 17:40 | #7 | Link | ||
Registered User
Join Date: May 2007
Posts: 220
|
Quote:
Quote:
|
||
10th February 2010, 10:42 | #9 | Link | |
Registered User
Join Date: Dec 2002
Location: UK
Posts: 1,673
|
Quote:
No, I meant the mathematical algorithms proposed in a few papers (see earlier super resolution threads here on doom9) which work in the specific cases Didée mentioned - and certainly we have such a case here. Problem is the maths assumes that the underlying scene does not change - only the camera moves. That's fair enough for background details, but useless and someone walks across the frame swinging their arms! Cheers, David. |
|
8th February 2010, 19:01 | #10 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,389
|
"real super resolution" - the myth that some enterprising people claim it would exist.
It has been discussed so often. Search for "(temporal) superresolution" and {my name}, or by MfA as well, who IIRC in the past has elaborated on that topic too. Short version: "superresolution" is basically possible when the input is aliased due to decimated resolution. Motion compensated (Bob-) Deinterlacers are one special case where TSR is actually working. But for the typical, average, non-aliased input with non-decimated resolution, there is no super-resolution to achieve. Also you script has very little to do with super-resolution. The biggest worker is NNEDI2, which softens edge-aliasing. Then you hit after it with some rather weak, rather cautious temporal averaging filter that mostly works in areas with very weak signal. NNEDI2 on its own is not a true superresolution filter - it does not bring any new information into the frame, it "only" interpolates edge oriented. Smoother edges, but no new information. The next step - temporal averaging only of areas that are highly similar anyway - is no superresolution either. Superresolution would require to appropriately use adjacent information that is quite different to the current information. (Don't judge just by the housewall texture - result appears nicer, yes, but to some extend that's just lucky coincidence. E.g.: on the trees in the background there is apparent *loss* of detail. On the house's chimney there is apparent *loss* of detail. How comes, since both parts are without 'incremental' motion, in the same way as the house wall is?) What your script does is upsizing with smooth edges, plus some denoising. There's nothing wrong with that. If the result seems pleasing, then it's fine. But better don't call that "super resolution". Because it isn't.
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
8th February 2010, 20:49 | #11 | Link |
Registered User
Join Date: May 2007
Posts: 220
|
Ok, I changed the name of the thread. It now better reflects the function of the script, to improve temporal stability by selectively averaging frames to reduce aliasing, moire, and compression artifacts, for the specific case of low-resolution, shaky video produced by a handheld digicam.
Last edited by um3k; 8th February 2010 at 20:55. |
10th February 2010, 13:02 | #13 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,389
|
@ Terka
Do you have a working copy of the famous Apply_PDF_Claims() filter? No? A pity! For what Filip Sroubek can or can not do, the same restrictions as noted in the past do apply. His method relies on subtile subpixel-differences - the problem is that in real-world sources, those tiny differences are strongly distorted by compression. (You can safely assume that the distortions are many-times as big as the original usable subpixel-diff has been. Hence, probably unusable.) His method may work for showcases. For general practical appliance, chances are rather poor. Also, his examples of "webcam video" are ... well, not particularly impressing. That we can do with NNEDI2+sharpening, too. Modern nuclear science has shown that it is by all means possible to turn lead into gold. The old alchemist's dream has become true!! - But wait, why isn't there any industrial appliance of that method? Because the result does not justify the effort. @ 2Bdecided Referring to the posted sample "Picture0013.avi", there is not much aliasing going on. (A little bit aliasing is needed for such small resolution to make it appear sharp. 320x240 without any aliasing would be not a picture, but a blurry mess.) A bit of aliasing, yes, but not very much. In particular, no aliasing as in "decimated resolution". *** Just remembered some sport that was done one year ago ... source - --- result - Not really earthshattering, it's just spatial interpolation (NNEDI2) combined with dedicated multistage sharpening. The good thing is that this is not a myth. It is real, and practically doable.
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
10th February 2010, 15:31 | #15 | Link | |
Registered User
Join Date: Dec 2002
Location: UK
Posts: 1,673
|
Quote:
FWIW the way these cameras work when capturing video - grabbing only every n-th pixel from a large sensor - is the very definition of decimated resolution IMO, assuming the image on the sensor is in-focus. I've shot hours of video at 320x480 on 4MP, 6MP and 8MP digital still cameras over the years - plenty of severe aliasing! Cheers, David. |
|
10th February 2010, 17:44 | #16 | Link | |
Registered User
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 1,197
|
Quote:
I'd like to try it on my aliasing-prone 'pseudo-progressive' (frame mode) DV material
__________________
Nostalgia's not what it used to be Last edited by WorBry; 10th February 2010 at 17:46. |
|
10th February 2010, 19:26 | #17 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,389
|
Here you go. Just yesterday it was exactly one year ago.
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
26th November 2010, 17:15 | #20 | Link | ||
Registered User
Join Date: Dec 2002
Location: UK
Posts: 1,673
|
Didée,
Do you remember exactly how you used this script... Quote:
Quote:
Using NNEDI2 (twice; both dimensions!) then that script gives a large image - if that's what you did, then I can't figure out how you downscaled it to 800x600 while maintaining such sharpness. I've tried scaling before or after the sharpening - I've even tried sharpening, scaling, then sharpening again. I can't make it match your result at all - I either get a softer result, or more halos. I know this stuff comes easy to you, but some of us mortals struggle to keep up - what did you do?! Cheers, David. |
||
Thread Tools | Search this Thread |
Display Modes | |
|
|