a concept for improved bobbing with nnedi2: "smoothnnedi2bob"

[shoopdabloop]

(function is further down in the thread)

sorry, i am very bad at explaining things, so hopefully the code and comments will help you understand.

AviSource("Tape 5 - Clip 007.avi").AssumeBFF()

#bob with nnedi2
nnedi2(field=-2)

#separate even and odd frames
edi = last
even = edi.selecteven()
odd = edi.selectodd()

#move even+odd frames forward in time by 50 percent (in new clips)
even2 = even.flowinter()
odd2 = odd.flowinter()

#interleave real frames with mflowinter frames in certain order,
#have to trim even clip because mflowinter cannot go in reverse(?),
#so first frame is effectively lost
interleave(odd,even2,even.trim(1,even.framecount),odd2)

#since each pair of frames are the same, but with slightly
#different spatial information, we blend each pair into one
#frame so they can assist each other to create a better image
merge(selecteven(),selectodd())

#simple helper function to shorten the code when using mflowinter
function flowinter(clip clp,float "tim")
{
tim = default(tim,50)
super = clp.msuper()
filtsuper = clp.blur(1.58).blur(1.58).temporalsoften(4,4,8,15,2).warpsharp(100).msuper()
bw = manalyse(filtsuper,isb=true)
fw = manalyse(filtsuper,isb=false)
return clp.mflowinter(super,bw,fw,time=tim)
}

i've tested the code on some single-scene live action sources, and it works quite well. there is tons of room for improvement however, but i am only a noob, so this is all i know how to do at the moment. cheers!

also, until i figure out how to fix the mflowinter-backwards thing (which might be simpler than i think), it cannot be plugged into tdeint or yadifmod properly, where it would best perform.

[10L23r]

this is interesting...
How's the speed?

[thewebchat]

This looks like MCBob.

[Didée]

Almost. It is like MCBob, just without any error correction. :)

And MFlow surely will produce errors, more than enough...

http://thumbnails20.imagebam.com/4925/16767b49246236.gif (http://www.imagebam.com/image/16767b49246236) http://thumbnails7.imagebam.com/4925/efdf4449246448.gif (http://www.imagebam.com/image/efdf4449246448)


To make those comparisons, I quickly corrected the script's problems with temporal alignment. Now everything is in-phase. (Still causes b0rked interpolation at the very 1st frame, I didn't bother)

nnedi2(field=-2)
fc = framecount(last)

#separate even and odd frames
edi = last
even = edi.selecteven()
odd = edi.selectodd()

#move even+odd frames forward in time by 50 percent (in new clips)
even2 = even.flowinter()
#odd2 = odd.flowinter()
odd2 = odd.selectevery(1,-1).flowinter()

#interleave real frames with mflowinter frames in certain order,
#have to trim even clip because mflowinter cannot go in reverse(?),
#so first frame is effectively lost
#interleave(odd,even2,even.trim(1,even.framecount),odd2)
interleave(even,odd2,odd,even2)

#since each pair of frames are the same, but with slightly
#different spatial information, we blend each pair into one
#frame so they can assist each other to create a better image
merge(selecteven(),selectodd())
trim(0,fc-1)
return(last)

#simple helper function to shorten the code when using mflowinter
function flowinter(clip clp,float "tim")
{
tim = default(tim,50)
super = clp.msuper()
filtsuper = clp.blur(1.58).blur(1.58).temporalsoften(4,4,8,15,2).warpsharp(100).msuper()
bw = manalyse(filtsuper,isb=true)
fw = manalyse(filtsuper,isb=false)
return clp.mflowinter(super,bw,fw,time=tim)
}

[shoopdabloop]

here is some new improved code (in function form). now, instead of applying the pair-merging to the entire image, we only do it on edges that actually need to be smoothed, to reduce visible mflow artifacts. optimizations that do not reduce visual quality much or visual improvements that do not decrease speed would be appreciated. :D
#smoothnnedi2bob v0.01 by shoopdabloop
#Used to attain smoother edges and eliminate most bob flicker when using nnedi2 for bobbing.
#Meant for live-action sources.
#Prerequisites: nnedi2, mvtools2, masktools2

function smoothnnedi2bob(clip clp)
{
edi = clp.nnedi2(field=-2)
even = edi.selecteven().flowfps(59.94)
odd = edi.selectodd().flowfps(59.94)
smooth = interleave(even,odd).selectevery(1,-1)
smooth = merge(smooth.selecteven(),smooth.selectodd())
a = edi.blur(0,0.25)
b = smooth
c = edi.blur(1.58).temporalsoften(4,4,8,15,2).mt_edge("sobel",0,4,0,5).mt_inflate()
return mt_merge(a,b,c)
}

function flowfps(clip clp,float "fps")
{
fps = default(fps,59.94)
super = clp.msuper()
filtsuper = clp.blur(1.58).temporalsoften(4,4,8,15,2).msuper()
bw = manalyse(filtsuper,isb=true)
fw = manalyse(filtsuper,isb=false)
return clp.mflowfps(super,bw,fw,num=int(fps*1000),den=1000)
}
recommended usage?:

MT("smoothnnedi2bob()",0,overlap=8,splitvertical=true)

[Gavino]

even = edi.selecteven().flowfps(59.94)
odd = edi.selectodd().flowfps(59.94)
smooth = interleave(even,odd).selectevery(1,-1)
smooth = merge(smooth.selecteven(),smooth.selectodd())
By hard-coding 59.94 in there, your function will only work on clips with 29.97 fps. Why not change flowfps to just specifically double the framerate of the clip it it is given?

Also, I think the logic isn't quite right. Shouldn't it be:
even = edi.selecteven().flowfps()
odd = edi.selectodd().flowfps()
smooth = merge(even,odd.selectevery(1,-1))

[shoopdabloop]

oops! thanks for catching my mistake, gavino, it was properly set before i changed flowfps to a float-input function.

and also, yes, my logic is "wrong" for that part, but i did so on purpose because otherwise the first frame is messed up. i decided it would be better to leave the frame untouched.

#smoothnnedi2bob v0.02 by shoopdabloop
#Used to attain smoother edges and eliminate most bob flicker when using nnedi2 for bobbing.
#Meant for live-action sources.
#Prerequisites: nnedi2, mvtools2, masktools2

function smoothnnedi2bob(clip clp)
{
edi = clp.nnedi2(field=-2)
even = edi.selecteven().flowfps(edi.framerate)
odd = edi.selectodd().flowfps(edi.framerate)
smooth = interleave(even,odd).selectevery(1,-1)
smooth = merge(smooth.selecteven(),smooth.selectodd())
a = edi.blur(0,0.25)
b = smooth
c = edi.blur(1.58).temporalsoften(4,4,8,15,2).mt_edge("sobel",0,4,0,5).mt_inflate()
return mt_merge(a,b,c)
}

function flowfps(clip clp,float "fps")
{
fps = default(fps,59.94)
super = clp.msuper()
filtsuper = clp.blur(1.58).temporalsoften(4,4,8,15,2).msuper()
bw = manalyse(filtsuper,isb=true)
fw = manalyse(filtsuper,isb=false)
return clp.mflowfps(super,bw,fw,num=int(fps*1000),den=1000)
}

i'm thinking of subtracting a fast-motion mask from the edge mask to prevent bad mflow caused by fast-moving objects (with hard edges) from showing.
do you think that would work?

[Gavino]

What I meant was changing your flowfps function to be something like
function flowfps(clip clp) {
fpsnum = 2*clp.frameRateNumerator
fpsden = clp.frameRateDenominator
super = clp.msuper()
...
return clp.mflowfps(super,bw,fw,num=fpsnum,den=fpsden)
}
Then you can just call it without passing a framerate.

It seems to me that your code
smooth = interleave(even,odd).selectevery(1,-1)
smooth = merge(smooth.selecteven(),smooth.selectodd())
is equivalent to
smooth = merge(odd, even).selectevery(1,-1)
which doesn't seem right, as odd and even are out of phase with each other.

[Didée]

Small technical buzz: as it's posted, the function uses nothing of all the mo'comp stuff on the chroma planes. Chroma is just nnedi2.blur(). Is this intentional? If not, you probably wanted to do

c = edi.blur(1.58).temporalsoften(4,4,8,15,2).mt_edge("sobel",0,4,0,5,U=3,V=3).mt_inflate(U=3,V=3)
return mt_merge(a,b,c,U=3,V=3)


Then ...

Personally, I'm not such a big fan of edge masking. (Not anymore ... the exitment has faded over the years.) Experience has told that edge masking often is not precise enough: usually it won't mask enough of the wanted stuff early enough, with danger of masking not-wanted stuff too early.

Let's see.

a) "Improvement, or no impromement?"

http://thumbnails18.imagebam.com/4936/1357a749352443.gif (http://www.imagebam.com/image/1357a749352443)

Obviously, the problems of the problematic sections are still problematic. It's less now, but it's not good yet.


b) "Are we still as good as we were before?"

http://thumbnails22.imagebam.com/4936/c38c6e49352698.gif (http://www.mediafire.com/download.php?ewmioytzgmy) (MediaFire, video-sample ~7MB)

Obviously, the masking stuff prevents quite some of the "good stuff", as can be seen by the flicker.


So, conclusion is that the masking solves only a part of the problems, while introducing quite some new problems.

The point with edge masking is that there's no safe in-between. You can make it more aggressive / more selective, to prevent more of the motion artifacts - but this will reduce the benefits in places were the results were good. Or you can make it less aggressive, to keep more of the good effects - but then it will let more artifacts pass through.

And there's no magic bullet to make the edge-masking method golden. You can ony shift the compromise in one direction or in the other, but you can't have both at the same time.

To me, edge masking is not a sufficient method. (There's more problems to it, like "detail present only in one field, but not in the other" - but let's spare that out for now...)


subtracting a fast-motion mask from the edge mask to prevent bad mflow caused by fast-moving objects (with hard edges)
Quite possible - but beware, it needs careful tweaking! And from the basics, it's a funny concept. Put simply, it boils down to
1) use motion search+compensation to improve a concept
2) but then, don't use motion compensation when there is motion...

Also, there's another pitfall. Those artifacts occur because the motion estimation was not correct (i.e. the results can be considered "random", sort of).
But, when the motion estimation is not reliable, how can you rely on it to figure where the problematic sections are? You're relying on that very thing that caused the problems in the first place.

***

See, I don't want to discourage you in any way. It's just that I've spent more than enough time with exactly these problems ... and I'm just taking the liberty to point them out. :)

Point is: when you have a rather slow concept, and have to add safety stuff (making it even more slow) to get a good result, then you should come to a really good balance. Because, when you end with a seriously slow process that still has serious issues, then it's a bad compromise.

[shoopdabloop]

i was thinking, how about forgetting all the edge and motion-mask stuff and only merging the real frames with the MFlow frames on areas where they mostly agree? (minimal difference)
would that work?

[Didée]

For some sources yes, for some sources no.

The goal is to interpolate missing lines. Most interesting areas are areas with detail. "Detail" is presented by pixels that are different from their neighbors. For "sharp" detail, this means that a pixel may be *very* different from its neighbors.

So, how would you define "minimal difference" in a way that won't prevent detailed areas to be correctly reconstructed? (detailed areas<--> big differences can be acceptable.)
Or how would you prevent a flat shade from appearing in a wrong area, just because an erroneous vector happens to say so? (flat areas <--> small differences may be very noticeable.)

[shoopdabloop]

do you know how to create a difference mask from two clips with masktools?
(highest values at areas with most difference)

also, i've changed the structure of the filter a little bit, but it's unfinished.

#smoothnnedi2bob v0.03 by shoopdabloop
#Used to attain smoother edges and eliminate most bob flicker when using nnedi2 for bobbing.
#Meant for live-action sources.
#Prerequisites: nnedi2, mvtools2, masktools2

function smoothnnedi2bob(clip clp)
{
edi = clp.nnedi2(field=-2)
even = edi.selecteven()
odd = edi.selectodd()
even2 = even.flowinter()
odd2 = even.trim(0,-1)++odd.flowinter()
flow = interleave(odd2,even2)
#for unclean result
#return interleave(even,odd2,odd,even2).merge(selecteven(),selectodd())
}

function flowinter(clip clp)
{
super = clp.msuper()
filtsuper = clp.blur(1.58).temporalsoften(4,4,8,15,2).msuper()
bw = manalyse(filtsuper,isb=true)
fw = manalyse(filtsuper,isb=false)
return clp.mflowinter(super,bw,fw,50,blend=false)
}

[Didée]

do you know how to create a difference mask from two clips with masktools?

simple "signed" difference, with "zero point" at mask=128:

mt_makediff( clip1, clip2 )


(highest values at areas with most difference)

simple absolute difference:

mt_lutxy( clip1, clip2, "x y - abs" )


Reading the documentation of MaskTools is a good idea.