temporal & spatial super resolution experiments

[scharfis_brain]

I have a lot of low-resolution digicam videos flying around here.

they are mostly 160x120pixels and 15 fps

so I searched for ways improving them.

I found a very good solution for panning-only shots.

It kills all noise and enhances resolution a LOT:

http://home.arcor.de/scharfis_brain/samples/SR.png

this is the script, I've used:


loadplugin("D:\x\depan.dll")

avisource("Mvi_0073.avi")

# for comparision
i=lanczos4resize(480,360).converttoyv12()

# for rendering
j=pointresize(480,360).converttoyv12()

data=i.depanestimate(pixaspect=1.0,zoommax=1,improve=false,trust=0)
r=4
j.depaninterleave(data,pixaspect=1.0,prev=r,next=r,mirror=15)
temporalsoften(r,255,255)
selectevery(2*r+1,r)

stackhorizontal(last.subtitle("temporal super resolution"),i.subtitle("original (160x120)"))


but there are downsides:

- differential motion will completely mess up with that script
- if depan doesn't catch the motion, you'll get a super-motion-blurrer
- it is slow.

[Fizick]

It is interesting, yes. I will try.
BTW, i almost finished coding more fast version of depanestimate.

[Chainmax]

Perfect, this is just what I need for those digicam videos I posted about a while ago. I have a suggestion though: since we're talking about a 900% upsize, wouldn't it be a better idea to use somethinglike SharpResize instead of Lanczos?

[Chainmax]

Hmm, didn't work as well for me. Here's a comparison screenshot:

http://members.arstechnica.com/x/master_burrfoot/Comparison-TSR_Vs_Orig.png

Also, here's the result of the script sh0dan recommended me to use on my thread:

http://members.arstechnica.com/x/master_burrfoot/sh0danrec.png

and the script itself:
loadplugin("wherever\EDIUpsizer.dll")
loadplugin("wherever\MVTools.dll")
loadplugin("wherever\AddGrain.dll")
DirectShowSource(something)
converttoyv12()
selectevery(3,0)
mvback=mvanalyse(isb=true, fth=60, pel=2,sx=4, sy=4)
mvforw=mvanalyse(isb=false, fth=60, pel=2, sx=4, sy=4)
mvconvertfps(last, mvback, mvforw, fps=25.0, fbw=2, thSCD2=255, thSCD1=500)
EDIUpsizer(method=7, window=8)
addgrain(10,0,0)
EDIUpsizer(method=7, window=8)
addgrain(10,0,0)
lanczosresize(480,336)


Maybe the blocking has something to do with the fact that this is the very first frame of the clip?

[scharfis_brain]

chainmax: the first & last frames aren't improved with my method, also I said:

- differential motion will completely mess up with that script
- if depan doesn't catch the motion, you'll get a super-motion-blurrer

so, it won't help much with your kind of clip.

it only helps with clips that
-are not that shaky (your is shaking heavily)
-have a very short shutter speed (your does)
-have no differential motion (yours has a huge amount of diff. motion)
-had been point-downscaled by the cam (yours is better scaled -> unfortunately not that suited for my algo....)

handling your clip with my method will result in an ultra-mess: 9 frames blended into another...

also:

wouldn't it be a better idea to use somethinglike SharpResize instead of Lanczos?

I only have used lanczos for the comparision.
my method internally uses pointresize as working clip
and lanczos as estimation clip.
It was the best combination


EDIT:

the result of the video, I used to build this script:
http://home.arcor.de/scharfis_brain/samples/SR1.avi


hint hint hint....

feed it with fieldseparated true interlaced video...

hint hint hint....

[Chainmax]

Well, I didn't know what differential motion was :o. While we're at it, would you want to give my clip a go?

[scharfis_brain]

the only thingI could think of improving your clip a little bit, would be using depan to denoise it

you can youe my superresolution-script, if you set f=1 and r=3 and temporalsoften(r,12,20) or any other values for 12 and 20 this should remove the blockyness....

but I cannot think of more improvement

maybe a scientist with fundamental mathematical and programming knowledge could help further with this....
(I am everytime astonished by the universitary papers showing extreme improvements but not publicating binaries for testing ourselves...)

[JanBing]

So, technically this should make it possible to take a standard PAL/NTSC clip, and blow it up to full HD-resolution, as long as it's an panning-only shot, right?

[scharfis_brain]

@Janbing: No.
a normal PAL or NTSC video is (beside the interlacing) soft scaled.
this means the moving shot has no flickering pixels (like my AVI-sample shows on its right) inside.
the pixel-flickering (due to nearest neighbor downscaling inside the cam) iscondition for using the super resolution script.

but your may use it for GMC-deinterlacing of interlaced video...

[JanBing]

what exactly is GMC? Apart from General Motors, of course...

[scharfis_brain]

GMC = Global Motion Compensation:

depan searches for the larges moving object within the video
then depan moves the prev. and next frames, so the object of the current frame is overlaid exactly.

'normal' motion compensation is not moving thewhole frame, it either moves blocks of defined size to synthetisize a new frame or it is warping the frame as if it would be as flexible as a rubber...

[morsa]

Amazing results!!!!!
Can't something "simillar" be made using MVTOOLS ???

[Didée]

Sure it can. Wanted to give it a go today evening. However, for the reasons scharfis_brain mentioned, for general content the result will be more about noise reduction and artefact removal, than about "resolution enhancing".

[Chainmax]

morsa, the script that sh0dan cooked up for me uses MVTools among other stuff. If you have any digicam videos lying around, give it a go and report back.

Didée, will your MVTools-based filter be similar to this one then?

[Didée]

Originally posted by Chainmax
Didée, will your MVTools-based filter be similar to this one then?
Hah, that was an optimist speaking.

I ran into the same problem as before, I just had "forgotten" a little about it:

I don't get clean enough temporal interpolations from the MVTools, whatever ME and other parameters I set. Problems with revealing areas, problems with vector coherence - lambda>1500 misses on contradicting motion, lambda<2000 may give vector salad, sometimes, problems with ... lots of things.

Anyone with good parameter suggestions for mvanalyse, mvconvertfps and mvcompensate, aiming for "average natural content"?

[morsa]

@Didee

Yeah, I have the same problems all the time, that's why I posted this:

http://forum.doom9.org/showthread.php?s=&threadid=82015

This ugly mess gives me improved results (it is my one-track-mind way of doing an image segmentation)

vec_back = mvanalyse(last.VagueDenoiser(threshold=4,method=1,nsteps=6,chromaT=0.8,percent=100).convertToYUY2._2DCleanYUY2(0,6,3,3,0,4,4).sharpen(1.0,1.0)._2DCleanYUY2(0,6,3,3,0,4,4).convertToYV12, isb=true, pel=1)

vec_forw = mvanalyse(last.VagueDenoiser(threshold=4,method=1,nsteps=6,chromaT=0.8,percent=100).convertToYUY2._2DCleanYUY2(0,6,3,3,0,4,4).sharpen(1.0,1.0)._2DCleanYUY2(0,6,3,3,0,4,4).convertToYV12, isb=false, pel=1)

MVConvertFPS(last, vec_back, vec_forw, nb=1,fps=50.0, fbw=2,bvt=35000, thSCD1=50)

[Chainmax]

Didée, the script I posted uses mvanalyse and mvconvertfps, why don't you try those values?

[Didée]

Chainmax:
Amongst others, I tried them ...
BTW, those settings are a little obsolete. "fth" has no effect in latest version(s) (and so hasn't "sadt"). These were replaced by the "lambda" parameter (vector field coherence).

morsa:
Right method. The more artefacts the source has, the more important cleaning & enhancing major image features becomes, to give the motion engine a reasonable clean playground.

Perhaps that was the reason why I got so bad results yesterday - I was working with an overcompressed mpeg-2 source of a music concert (DVB ... :rolleyes: )
I remember to have made some 50fps test clips of some chapters of LOTR-TTT some time ago, with quite nice results.

However, having to set up dedicated cleaning for each single source individually can't be the way eiter.

[morsa]

I know....:D
It's just I'm a sucker with this scripting things...
Could you test this method and give an improved version?
May be some other filter does the job easier....
Anyway I hope manao read this and implements some kind of internal preprocessing (may be a bilateral filter with strong settings could do the job)

That thing I've posted above is giving me perfect results, not even a block!!!!!!!

[Didée]

A little bird flew along and sang the song of "Manao is overbusy with other important things these days"

(And I'm sure we all will love ... it ;) )

[Fizick]

I think, that
some kind of internal preprocessing
is not needed, if we can use more flexable external.

I also think about combining MVTOOLS with DEPAN (external)

[morsa]

Well, I´ve used it in my full script to stabilize the clip febore aplying the ME.
What do you mean?

[Fizick]

I am still thinking :)

[Ark]

Damn. I cannot use the Sharfi's script, 'cause i cannot load Depan...

Avisynth gave me an error with incomprensible characters in the line where i put "Loadplugin"...what's wrong??

My system is AthlonXP 2400+, 1 gigabyte RAM, Win XP Pro

[anton_foy]

this super resolution filter is mighty tempting to use on my HDV material. But for starters it takes forever to load and doesnt seem to give my footage more enhancement (comparing to Scharfis picture). Tried it on a static shot and like scharfis said the movement in the footage gets superblurred but the weird thing is that it still looks like no enhancement was made. Source is 1440x1080 anamorphic squeeze and I triple the resolution with this filter and Lanczos it down to 1920x1080.

Anyway that example is really impressive!:)

[scharfis_brain]

as I said earlier: the conditions to use temporal super resolution are rare.
main point: you need ALIASED (this means: footage full of stairstepping, and fringing and flicker) footage. also this script requires the video being progressive.
further normal camcorder video rarely is heavily aliased (besides the interlace).
if you want a semi-TSR result, you may try mvbob(). this is the only solution to obtain true extra resolution from interlaced video.