Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 6th February 2010, 19:22   #1  |  Link
um3k
Registered User
 
Join Date: May 2007
Posts: 220
Temporal-Spatial Anti-Aliasing for Shaky Digicam Videos

Some of you may know about scharfis_brain's long-ago attempt at temporal-spatial super resolution. http://forum.doom9.org/showthread.ph...solution+depan

It's certainly the most successful attempt I've seen at avisynth-based super resolution. In a quest to improve some low-quality digicam videos, I've crudely modified his script to preserve differential motion.

Code:
AVISource("Picture0013.AVI")
ConvertToYUY2()

# for comparision
i=lanczos4resize(640,480).converttoyv12()

# for rendering
#~ j=pointresize(640,480).converttoyv12()
j=nnedi2_rpow2(2).converttoyv12()

data=i.depanestimate(range=2,pixaspect=1.0,zoommax=1,improve=false,trust=0)
r=4
j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=15,inputlog="0013.log")

m=mt_makediff(last,j.ChangeFPS(last.Framerate)).Grayscale()
m=Overlay(m, m.Invert(), x=0, y=0, opacity=1.0, mode="lighten", greymask=true, ignore_conditional=false, pc_range=True).Levels(145, 1.000, 160, 0, 255, coring=false)

mt_merge(last, j.ChangeFPS(last.Framerate), m, luma=true)
temporalsoften(r,255,255)
selectevery(2*r+1,r)

#~ Unsharp(varY=1.0,strength=2)

stackhorizontal(last.subtitle("temporal super resolution"),i.subtitle("original (320x240)"))
As you can probably see, my Avisynth scripting ability is pretty basic. But the results I get from this are definitely a step in the right direction. A couple things to note:
1. I've replaced the pointresize with NNEDI2. The pointresize probably would be a little sharper, but it doesn't get along very well with my crude masking.
2. I've replaced the DepanEstimate results with a Deshaker log, which I found to result in more accurate correction.

Here is a still example of its efficacy. Notice that there is a huge reduction in JPEG artifacts, as well as aliasing, particularly on the house in the background. (Click for full size)

For comparison, here is the output of scharfis_brain's original script:


Unfortunately, there are still motion artifacts, primarily ghosting on differential motion, as well as some chroma artifacts. I'll continue working on it, but am not sure how far I'll be able to go. And, of course, there is the additional limitation of differential motion not receiving any enhancement.

If you want to take a stab at it, or just see the results for yourself, here is the original video file as well as the log file:
Picture0013.AVI
0013.log

So, take a look, and toss in your two cents.

Last edited by um3k; 8th February 2010 at 20:24. Reason: Changed Title
um3k is offline   Reply With Quote
Old 8th February 2010, 17:48   #2  |  Link
davidhorman
I'm the Doctor
 
Join Date: Jan 2004
Posts: 1,490
Code:
j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=15,inputlog="0013.log")
Others will need to remove the inputlog parameter, since we don't have your log file...

Edit: Doh, it's right there in the post

David

Last edited by davidhorman; 8th February 2010 at 20:02.
davidhorman is offline   Reply With Quote
Old 8th February 2010, 17:54   #3  |  Link
um3k
Registered User
 
Join Date: May 2007
Posts: 220
I included the log file in my post. Look at the links near the end.

BTW, I've been working on this some more, and have made some improvements. I'll post an update when I feel it is ready.
um3k is offline   Reply With Quote
Old 8th February 2010, 17:55   #4  |  Link
2Bdecided
Registered User
 
Join Date: Dec 2002
Location: Yorkshire, UK
Posts: 1,673
I've used this in the past...
PHP Code:
nnedi(field=0,dh=true).turnleft().nnedi(field=0,dh=true).turnright()

vf=last.mvanalyse(idx=1,pel=1,isb=false,blksize=16truemotion=true)
vb=last.mvanalyse(idx=1,pel=1,isb=true,blksize=16truemotion=true)

xf=last.mvflow(idx=1,vf)
xb=last.mvflow(idx=1,vb)

interleave(xf,last,xb)
temporalsoften(1,20,20,mode=2)
selectevery(3,1
There's also a cripplingly slow script here...
http://forum.doom9.org/showthread.ph...569#post812569
...which looks like a newer / better version of the one you started with.

I've tried various attempts to mask badly predicted and unpredictable movement and not merge those parts. Nothing worth sharing - and glacially slow!

TBH, it's nothing like real super resolution (I think we all know this) - I have clips with fine details that flicker horribly - real super resolution would fix these nicely.

Cheers,
David.
2Bdecided is offline   Reply With Quote
Old 8th February 2010, 18:05   #5  |  Link
um3k
Registered User
 
Join Date: May 2007
Posts: 220
I just gave those both a try. The script in your post didn't create motion artifacts, but it also didn't really improve the image much. The script in the link caused horrific motion artifacts, and still didn't provide as much improvement as the current version of the script I posted. So I'll keep trying to improve what I've got here. And true, it isn't really super resolution--but it does make it easier to watch.
um3k is offline   Reply With Quote
Old 8th February 2010, 18:06   #6  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 3,514
2Bdecided - what products/software are you referring to with "real super resolution" ? Do you mean like video enhancer , topaz enhance , instanthd , etc... or something else?
poisondeathray is offline   Reply With Quote
Old 8th February 2010, 19:01   #7  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,390
"real super resolution" - the myth that some enterprising people claim it would exist.

It has been discussed so often. Search for "(temporal) superresolution" and {my name}, or by MfA as well, who IIRC in the past has elaborated on that topic too.

Short version: "superresolution" is basically possible when the input is aliased due to decimated resolution. Motion compensated (Bob-) Deinterlacers are one special case where TSR is actually working.

But for the typical, average, non-aliased input with non-decimated resolution, there is no super-resolution to achieve.


Also you script has very little to do with super-resolution. The biggest worker is NNEDI2, which softens edge-aliasing. Then you hit after it with some rather weak, rather cautious temporal averaging filter that mostly works in areas with very weak signal.
NNEDI2 on its own is not a true superresolution filter - it does not bring any new information into the frame, it "only" interpolates edge oriented. Smoother edges, but no new information. The next step - temporal averaging only of areas that are highly similar anyway - is no superresolution either. Superresolution would require to appropriately use adjacent information that is quite different to the current information.
(Don't judge just by the housewall texture - result appears nicer, yes, but to some extend that's just lucky coincidence. E.g.: on the trees in the background there is apparent *loss* of detail. On the house's chimney there is apparent *loss* of detail. How comes, since both parts are without 'incremental' motion, in the same way as the house wall is?)

What your script does is upsizing with smooth edges, plus some denoising. There's nothing wrong with that. If the result seems pleasing, then it's fine.

But better don't call that "super resolution". Because it isn't.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 8th February 2010, 20:49   #8  |  Link
um3k
Registered User
 
Join Date: May 2007
Posts: 220
Ok, I changed the name of the thread. It now better reflects the function of the script, to improve temporal stability by selectively averaging frames to reduce aliasing, moire, and compression artifacts, for the specific case of low-resolution, shaky video produced by a handheld digicam.

Last edited by um3k; 8th February 2010 at 20:55.
um3k is offline   Reply With Quote
Old 8th February 2010, 21:45   #9  |  Link
Terka
Registered User
 
Join Date: Jan 2005
Location: cz
Posts: 676
search for "Filip Sroubek" superresolution
Terka is offline   Reply With Quote
Old 10th February 2010, 10:42   #10  |  Link
2Bdecided
Registered User
 
Join Date: Dec 2002
Location: Yorkshire, UK
Posts: 1,673
Quote:
Originally Posted by poisondeathray View Post
2Bdecided - what products/software are you referring to with "real super resolution" ? Do you mean like video enhancer , topaz enhance , instanthd , etc... or something else?
I've tried the first two, and been disappointed.

No, I meant the mathematical algorithms proposed in a few papers (see earlier super resolution threads here on doom9) which work in the specific cases Didée mentioned - and certainly we have such a case here. Problem is the maths assumes that the underlying scene does not change - only the camera moves. That's fair enough for background details, but useless and someone walks across the frame swinging their arms!

Cheers,
David.
2Bdecided is offline   Reply With Quote
Old 10th February 2010, 10:48   #11  |  Link
2Bdecided
Registered User
 
Join Date: Dec 2002
Location: Yorkshire, UK
Posts: 1,673
Quote:
Originally Posted by um3k View Post
I just gave those both a try. The script in your post didn't create motion artifacts, but it also didn't really improve the image much. The script in the link caused horrific motion artifacts, and still didn't provide as much improvement as the current version of the script I posted.
Oh well - I thought maybe you could adapt it, or re-use bits of it - not use it as-is.

e.g. normally block-based motion compensation is going to work better than global motion compensation. Depends on the content, but I think I'd try to use that part, at least.

FWIW I think it counts as super resolution - the source is clearly aliased, and NNEDI + motion compensated temporal smearing is a kind of poor man's super resolution - it's maybe 25%-40% of the real thing - but not 0%.

Cheers,
David.
2Bdecided is offline   Reply With Quote
Old 10th February 2010, 13:02   #12  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,390
@ Terka

Do you have a working copy of the famous Apply_PDF_Claims() filter? No? A pity!

For what Filip Sroubek can or can not do, the same restrictions as noted in the past do apply. His method relies on subtile subpixel-differences - the problem is that in real-world sources, those tiny differences are strongly distorted by compression. (You can safely assume that the distortions are many-times as big as the original usable subpixel-diff has been. Hence, probably unusable.) His method may work for showcases. For general practical appliance, chances are rather poor.
Also, his examples of "webcam video" are ... well, not particularly impressing. That we can do with NNEDI2+sharpening, too.

Modern nuclear science has shown that it is by all means possible to turn lead into gold. The old alchemist's dream has become true!!
- But wait, why isn't there any industrial appliance of that method? Because the result does not justify the effort.


@ 2Bdecided

Referring to the posted sample "Picture0013.avi", there is not much aliasing going on. (A little bit aliasing is needed for such small resolution to make it appear sharp. 320x240 without any aliasing would be not a picture, but a blurry mess.)
A bit of aliasing, yes, but not very much. In particular, no aliasing as in "decimated resolution".

***

Just remembered some sport that was done one year ago ...

source - --- result -

Not really earthshattering, it's just spatial interpolation (NNEDI2) combined with dedicated multistage sharpening. The good thing is that this is not a myth. It is real, and practically doable.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 10th February 2010, 14:03   #13  |  Link
aegisofrime
Registered User
 
Join Date: Apr 2009
Posts: 452
Quote:
Originally Posted by Didée View Post
source - --- result -

Not really earthshattering, it's just spatial interpolation (NNEDI2) combined with dedicated multistage sharpening. The good thing is that this is not a myth. It is real, and practically doable.
OMG! That's really amazing!
aegisofrime is offline   Reply With Quote
Old 10th February 2010, 15:31   #14  |  Link
2Bdecided
Registered User
 
Join Date: Dec 2002
Location: Yorkshire, UK
Posts: 1,673
Quote:
Originally Posted by Didée View Post
@ 2Bdecided

Referring to the posted sample "Picture0013.avi", there is not much aliasing going on. (A little bit aliasing is needed for such small resolution to make it appear sharp. 320x240 without any aliasing would be not a picture, but a blurry mess.)
A bit of aliasing, yes, but not very much. In particular, no aliasing as in "decimated resolution".
You think? I thought the rather random patterns appearing on the house in the background were exactly that (assuming it actually consists of horizontal white boards). I suspect the house is marginally more in focus than the trashed car.

FWIW the way these cameras work when capturing video - grabbing only every n-th pixel from a large sensor - is the very definition of decimated resolution IMO, assuming the image on the sensor is in-focus.

I've shot hours of video at 320x480 on 4MP, 6MP and 8MP digital still cameras over the years - plenty of severe aliasing!

Cheers,
David.
2Bdecided is offline   Reply With Quote
Old 10th February 2010, 15:45   #15  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,390
Those random patterns are just interference. Interference is not necessarily caused by dropping pixels. It can also appear with regular area-based sampling methods.
On the other hand, all straight edges appear more or less anti-aliased. And that's what I (mostly) based my opinion on.

Well, whatever. Looking at that source closely, my guts tell me that this source is hardly suited for temporal superresolution. (I've made best experiences by trusting my guts.)
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 10th February 2010, 17:06   #16  |  Link
um3k
Registered User
 
Join Date: May 2007
Posts: 220
Well, regardless, I went ahead and transformed it into a rather slow function. Let me know if you want an explanation of the arguments, I'm not going to bother typing one out if no one is interested. Keep in mind this is the first avisynth function I've created, and is mostly a result of shooting in the dark. But, I like the results it gives with certain sources.

Code:
#   NoMoreMoire filter by um3k/Justin Phillips                     
#   Based on a "Super Resolution" script by scharfis_brain     
#   Reduces moire, aliasing, noise, and compression artifacts 
#   in shaky, low-quality video from digital still cameras         
#   Alpha 1

function NoMoreMoire_alpha1(clip c, int "scale", int "r", int "freq", int "con", int "mmin", int "mmax", int "sthr", int "srad", string "inputlog", int "mode", bool "chroma")
{
    scale = Default(scale, 2)
    r = Default(r, 4)
    freq = Default(freq, 3)
    sym = 50
    con = Default(con, 72)
    mmin = Default(mmin, 63)
    mmax = Default(mmax, 88)
    sthr = Default(sthr, 0)
    srad = Default(srad, 0)
    mode = Default(mode, 0)
    chroma = Default(chroma, true)

    # for comparision
    i = c.lanczos4resize(c.width*scale,c.height*scale).converttoyv12()

    # for rendering
    j = c.nnedi2_rpow2(scale, cshift="spline36resize").converttoyv12()

    # Motion Comp
    comp = (Defined(inputlog)==true) ? Eval("""
        data = i
        j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=15,inputlog=inputlog)
    """) : Eval("""
        data = i.depanestimate(range=2,pixaspect=1.0,zoommax=1,improve=false,trust=0)
        j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,subpixel=2,mirror=15)
    """)

    # Mask Creation
    m = (chroma==true) ? Eval("""
        m = mt_makediff(F2Quiver(comp, "gcl", freq, sym, degree=4, rescale=true, morph=false) ,F2Quiver(j, "gcl", freq, sym, degree=4, rescale=true, morph=false)\
.ChangeFPS(comp.Framerate), chroma="process").SmoothLevels(0+con, 1.000, 255-con, 0, 255, chroma=100, limiter=1, Lmode=1)
        mc = Overlay(AbsDif(m.UToY()), AbsDif(m.VtoY()), opacity=1.0, mode="lighten", pc_range=false).BicubicResize(m.width,m.height)
        m = AbsDif(m).Greyscale()
        m = Overlay(m, mc, opacity=1.0, mode="lighten", pc_range=True)
        m = mt_hysteresis(m.MT_deflate().MT_deflate().MT_deflate().MT_deflate().Medianblur(3).mt_binarize(mmax), m.mt_binarize(mmin)).mt_expand().Grayscale()
        m
    """) : Eval("""
        m = mt_makediff(F2Quiver(comp, "gcl", freq, sym, degree=4, rescale=true, morph=false) ,F2Quiver(j, "gcl", freq, sym, degree=4, rescale=true, morph=false)\
.ChangeFPS(comp.Framerate), chroma="1").SmoothLevels(0+con, 1.000, 255-con, 0, 255, chroma=0, limiter=1, Lmode=1)
        m = AbsDif(m).Greyscale()
        m = mt_hysteresis(m.MT_deflate().MT_deflate().MT_deflate().MT_deflate().Medianblur(3).mt_binarize(mmax), m.mt_binarize(mmin)).mt_expand().Grayscale()
        m
    """)
    
    # Shadow Mask
    m = (sthr==0) ? Eval("""
        m
    """) : Eval("""
        Overlay(m, comp.mt_expand().YLevels(sthr-srad, 1.000, sthr+srad, 0, 255).invert().mt_expand().mt_deflate().ChangeFPS(comp.Framerate), opacity=1.0,\
 mode="lighten", greymask=true, ignore_conditional=false, pc_range=true).Grayscale()
    """)
    
    # Mask Visualization
    mv = m.temporalsoften(r,255,255).selectevery(2*r+1,r)
    
    # Merge Compensated with Original to Protect Motion
    mix = mt_merge(comp, j.ChangeFPS(comp.Framerate), m, luma=true) 
    
    # The Important Part
    temp = mix.temporalsoften(r,255,255) 
    
    # Back to normal framerate
    dec = temp.selectevery(2*r+1,r) 
    
    # Decide output
    out = (mode==2) ? Eval("""
        Overlay(dec.Levels(0, 1.000, 255, 0, 127, coring=false), mv.Levels(0, 1.500, 255, 0, 255, coring=false), opacity=1.0, mode="add", greymask=true,\
 ignore_conditional=false, pc_range=true)
    """) : \
       (mode==1) ? Eval("""
        i
    """) : Eval("""
        dec
    """)
    
    return out
}

function AbsDif(clip c)
{
    Overlay(c.YLevels(127, 1.000, 255, 0, 255), c.Invert().YLevels(127, 1.000, 255, 0, 255), opacity=1.0, mode="lighten", pc_range=True)
}
The scripting isn't pretty by any definition of the word, and I'm sure there is plenty room for optimization. But it does something, and for that, I'm happy.

If you omit the "inputlog" argument, it will default to depanestimate. However, I recommend creating a log in Deshaker with the slowest settings for the best results. "mode=1" will output lanczos-scaled footage, so you can feed that into VirtualDub, or you can do the scaling yourself.

Also, for best results, you might want to follow up by sharpening and adding some grain--the results tend to be somewhat soft and smooth, but they don't seem to have lost many details that weren't already destroyed by the compression.

Last edited by um3k; 10th February 2010 at 17:24.
um3k is offline   Reply With Quote
Old 10th February 2010, 17:40   #17  |  Link
um3k
Registered User
 
Join Date: May 2007
Posts: 220
Quote:
Originally Posted by 2Bdecided View Post
Oh well - I thought maybe you could adapt it, or re-use bits of it - not use it as-is.
I'll take another look at it at some point--right now I'm seeing how far I can go with what I started with.

Quote:
e.g. normally block-based motion compensation is going to work better than global motion compensation. Depends on the content, but I think I'd try to use that part, at least.
I haven't done extensive testing, but from what I've read and done, it seems MVTools doesn't have the accuracy I need--nor does DepanEstimate, hence my usage of a Deshaker log. However, I'm looking into using MV for the moving bits in order to give them some degree of enhancement, as opposed to the none they have currently. Well, nothing beyond NNEDI2, anyways.
um3k is offline   Reply With Quote
Old 10th February 2010, 17:44   #18  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 950
Quote:
Originally Posted by Didée View Post

Just remembered some sport that was done one year ago ...

source - --- result -

Not really earthshattering, it's just spatial interpolation (NNEDI2) combined with dedicated multistage sharpening. The good thing is that this is not a myth. It is real, and practically doable.
Any chance of the/a script for that?

I'd like to try it on my aliasing-prone 'pseudo-progressive' (frame mode) DV material
__________________
Nostalgia's not what it used to be

Last edited by WorBry; 10th February 2010 at 17:46.
WorBry is offline   Reply With Quote
Old 10th February 2010, 19:26   #19  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,390
Quote:
Originally Posted by WorBry View Post
Any chance of the/a script for that?
Here you go. Just yesterday it was exactly one year ago.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 11th February 2010, 04:52   #20  |  Link
WorBry
Registered User
 
Join Date: Jan 2004
Location: Here, there and everywhere
Posts: 950
Great, thanks. Havent had chance to play with it yet.
__________________
Nostalgia's not what it used to be
WorBry is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 13:04.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.