Motion filter : +1st MC FPS-change script for real usage

[Mug Funky]

@ clouded: thanks heaps for this. i've got a head full of bad audio at the moment, but i'll give this a shot on a suitable source ASAP.

@ foxyshadis: it'd be awesome if you could pull this off (especially without artefacts), as detecting different framerates and treating them accordingly is something not available easily. and depending on how stable the framerate conversion is, it might even be able to replace speed-up to PAL (this is good if one wants to re-use audio).

[edit]

while waiting for a compile to finish, i gave this a try on some 60i stuff. it's awesome! and fast! thanks heaps

you don't have any problem with my using this in production now do you? this could be quite useful for DVD extras (they're very often 60i), and this thing runs almost as fast as vanilla kernelbob+convertfps+resize...

[scharfis_brain]

For Standards Conversion I suggest you to use this deinerlacing script:

function securedeint(clip i, int "th", int "l", int "mode")
{   th=default(th,6)
    l = default(l,1)
    mode=default(mode,2)

   function staticmask(clip e, int th, int l)
	{e0=e.motionmask(thy1=0,thy2=th,thc1=0,thc2=th,thSD=255,u=3,v=3).temporalsoften(6,1,1)
	e1=e0.trim(1,0)
	e2=e0.trim(2,0)
	e3=e0.trim(3,0)
	e4=e0.trim(4,0)
	e5=e0.trim(5,0)
	e6=e0.trim(6,0)
	e7=e0.trim(7,0)
	ea0=logic(e0,e1,"OR")
	ea1=logic(e2,e3,"OR")
	ea2=logic(e4,e5,"OR")
	ea3=logic(e6,e7,"OR")
	eb0=logic(ea0,ea1,"OR")
	eb1=logic(ea2,ea3,"OR")
	ec=logic(eb0,eb1,"OR")
	(l==0) ? ec .duplicateframe(0).duplicateframe(0).duplicateframe(0) :\
	(l==1) ? eb0.duplicateframe(0) :\
	(l==2) ? ea0 : e0
}

i.converttoyv12(interlaced=true).separatefields()
e=selecteven().staticmask(th, l)
o=selectodd().staticmask(th, l)
mi=interleave(e,o)
m=mi.bob() 
#interleave(mi.weave().tomsmocomp(-1,0,0), mi.doubleweave().selectodd().tomsmocomp(-1,0,0))
m=merge(m.trim(1,0),m,0.5).binarize(upper=false,u=3,v=3)


eedi = getparity(i) ? i.separatefields().eedi2(field=3) : i.separatefields().eedi2(field=2)
toms = interleave(i.tomsmocomp(-1,0,0), i.doubleweave().selectodd().tomsmocomp(-1,0,0))
krnl = i.leakkernelbob(order=(getparity(i) ? 1 : 0), threshold=0)
dumb = i.bob(0,1)

b = (mode==0) ? dumb : (mode==1) ? krnl : (mode==2) ? toms : eedi

d=i.doubleweave()

overlay(d,b,mask=m)
#interleave(min,m,mex)
getparity(i) ? assumetff() : assumebff()
}

I practically has no artifacts caused by the motionmask (holes)
And it is pretty fast.

Parameters:
th - deinterlacing threshold 6 seems to be a good value and is default

l - temporal radius of the combmask:
0 -> 16+2 fields are used for motion masking
1 -> 8+2 fields (default)
2 -> 4+2 fields
any other value -> 2+2 fields

mode - interpolation type
0 - bicubic
1 - kernel
2 - tomsmocomp's ELA (default)
3 - triticals EEDI2 (slooooow)

[mg262]

Originally Posted by Mug Funky

you don't have any problem with my using this in production now do you? this could be quite useful for DVD extras (they're very often 60i),

Of course not -- it's free software, you can use it for what you like! If I can make specific changes to make it better for the task,* I'm happy to. But, for the love of God, make sure it's good enough for the task and doesn't cause excessive weird glitches before you use it...

*i.e. it is really useful to know what kind of footage it fails on, but that doesn't always translate to a concrete change I can make to the thing. Plus, some specific changes are a lot of work to implement or won't work well with the motion algorithm (e.g. different block sizes). But, time willing, I'll enjoy trying to get it to run better if you can think how, esp. if it's going to be used on "real" stuff. If you want, if it ever reaches the point where you can use it heavily, you could always talk your company into making a subscription to a children's charity I'm a trustee of... but you don't have to; I'll be pleased just to have it working well.

and this thing runs almost as fast as vanilla kernelbob+convertfps+resize...

It's not running as fast as it can, yet, mainly because I left everything possible in script to let you all play with it. Currently it's about three times as slow as (convertto... + convertFPS + convertto...) -- 70 FPS vs 200 FPS here. If we reach a point where you're all happy with the script, I can start filterising bits... I'd like to hit 100 FPS. (NB BlendFPS is also pretty fast, so you can use it in script for additional artefact protection.)

I'm unlikely to make many changes to this for a fair while -- hitting the script-tweakable mark was the milestone I was aiming at. But there are a few things I have in mind further down the line. First is a couple of new compensation modes, which split the blocks into 4x4 blocks before compensation (I talked about this above)... should decrease artefacts a bit. Second is inbuilt GMC, to track zooms, etc better. Third if I have the energy is proper warping, not bilinear... though this will be very slow. Last is lots of speedups!

[Mug Funky]

hehe... charitable donations, huh? we're not above such things (though it'll probably be me that does the donating...).

i've tried the script on 1 clip so far, but it had a fair range to play with. it was one of the "group discussion" extras from Otogi Zoshi. it's ex-composite (ew), has fairly high contrast, some pans, some fast hand-gestures (these tend to be hell for mocomp, but your script handles them flawlessly), and some clips out of the actual show (ie animated).

your script seems to fall back on blending for the anime bits, and mocomp everything else. very nice - just the behaviour i like. i'll have to throw something more challenging at it

btw, i tried it on another source - a mix of 25p and 50i, with NTSC as it's final destination (yes, i make stuff for PAL land, but sometimes PAL land produces something that NTSC land might like), and it seems it blends progressive most of the time, but sometimes mocomps it. i imagine that's a matter of tweaking the script... of course it doesn't bother me at all, as IMHO progressive PAL should be blended to NTSC otherwise you get a disconcerting "jerky-slidy" look that probably nobody notices but me...

one thing i saw was that scenechanges weren't always being caught, but seeing as the transitions were over 1 field it was barely visible (certainly not catastrophic due to the robustness of the motion-algo you're using).

anyhoo, very nice stuff. maybe i'll send you a copy of the first disc i use this on (haven't used it on anything that's going out the door just yet).

[mg262]

Minor issue: the script as it currently stands upsamples the 1/64-size mask and then removes the padding, like this.

bilinearresize(width*8, height*8)
crop(0, 0, input.width(), input.height())

This is a bad idea on my part because the resamplers will use padding values to interpolate at the right hand and bottom edges of the frame. Unfortunately the obvious solution, namely

bilinearresize(input.width(), input.height(), 
\0, 0, input.width()/8, input.height()/8)

doesn't solve things. From testing and from glancing at the resampling code, I think that it is accessing values from the original frame base outside the cropped region. (So, I think the manual may be wrong when it states that this is exactly equivalent to crop + resize).

There isn't a really satisfactory solution to this until Y8 is available [and I'm very much coding everything with Y8 in mind]. For bilinear resampling, I can work around it by copying the padding values from the nearest pixel values... and I will release a version that does that soon. That will be close enough to the right answer for bicubic, etc to be indistinguishable -- actually, the way the masks are used, its unlikely that the current method is indistinguishable from the correct one, but it should of course be fixed anyway!

Sorry about this...

Proper replie(s) also coming soon.

[mg262]

Padding copied from border pixels to work around crop/resize issue (see last post):

Motion, 9 December 05
MotionProtectedFPS script (unchanged)
Edit:Documentation

Edit: Scene changes aren't being dealt with at all... IMO detecting + dealing with them properly takes much more work than a AverageLumaDifference, and sorting it out will take a couple of weeks at some point. (Although I'll do it in such a way that I can use it for all my filters at once.)

But in the meantime, you need something to work with... how about this: a postprocessing filter that detects scene breaks using a dumb-as-hell method (AverageLumaDifference), and duplicates frames to avoid interpolation-across-scene-breaks? So it would look something like this:

interpolated = MotionProtectedFPS(original, FPS, ...)
FixUpFPSA(original, FPS, interpolated, lumathreshold)

? (I assume duplicating frames is preferable to blending across scene breaks?)

[Mug Funky]

that's an interesting idea. i think kassandro has a filter that does that (improvesceneswitch?)

though i can imagine disturbing jerkiness on flashes and quick pans using this method. i can settle for blended scenechanges though - they'll only last 1 field anyway ... even with the slight warping that the current script gives, it is nearly impossible to actually see at playback speed on an interlaced screen.

[scharfis_brain]

I have a much better Idea for stable scene detection!

Initial step:
- create a motionmask
motionmask(thy1=noise,thy2=noise+1,thc1=noise,thc2=noise+1,thscd=255,u=3,v=3)
- noise is a variable to pass noisy clips
- make a logical AND link between two temporal adjacent Motionmasks
logic(last,last.trim(1,0),"AND",u=3,v=3)

this ANDed motionmask will now show completely filled on scene changes, while normal motion just shows the contoures of moving objects.

Now the decision step:
- calculate the averageluma of that ANDed motionmask.
it has to be factor plus [offset[/i] times greater than the avglume of two predecessing and two following frames.

factor needs to be about 5 to 20
offset about 20.

the offset avoids, that in low motion scenes nearly every frame gets detected as scene change. and the factor defines the minium factor in luma between normal and scene changed motionmask.

ahhh, I finally found my code snipplet:

function scd(clip i, int "factor", int "offset")
{       global fac=default(factor,5)
global off=default(offset,25)

global msk0=i.duplicateframe(0).reduceby2().reduceby2().motionmask(thsd=255,y=3,u=1,v=1)
global msk1=msk0.trim(1,0)
global msk2=msk0.trim(2,0)

global blk=i #blankclip(i)
global wht=i.subtitle("scenechange") #blk.invert()

f1=scriptclip(i,"sc") 
f2=f1.frameevaluate("sc=( (a2 > (a1*fac+off)) && (a2 > (a3*fac+off)) ) ? wht : blk")
f3=f2.frameevaluate("a3=averageluma(msk2)")
f4=f3.frameevaluate("a2=averageluma(msk1)")
   f4.frameevaluate("a1=averageluma(msk0)")
}

[Mug Funky]

yo. i'm getting crashes, but haven't isolated the problem to anything specific yet.

just running mpeg2source and motionprotectedfps(50) to bump a 25p source up to 50p, then playing in mplayer to gauge CPU usage (awesome... realtime and averaged 60% CPU use for PAL DVB source).

only problem is it crashed after about a minute of playback. memory use was a constant 170 megs (about the same in VM too).

i'm running an athlon 64 3500+ venice, at a very cruisy 2200 MHz, with a gig of DDR and winXP (SP2, not x64 unfortunately).

performance is awesome other than that though. really fast and only artefacts in certain cases (i might clip some stuff out where artefacts occur).

[edit]

this could be my source. it crashes _around_ the 7000 frame mark (that's 3500 source frames). i'll take a look at it on another clip.

[mg262]

Damn damn damn. I tried that on a longer clip than I had previously tested + it crashed at the 74000 frame mark. Really sorry about this; CPU is largely tied up with a long encode but I will track it down ASAP.

EDit: rounnding error of 1/16 of a motiion vector... causes read outside frame in some unusual circumstances. I'll fix it tomor row when i have enough cpu to have speech-rec running again.

[mg262]

Motion, 12 December 05
MotionProtectedFPS script
Documentation

I think this should have fixed it -- if not, could you send me a clip on which it fails? FWIW, the motion algorithm converge is pretty fast so if you have an error you can generally trim to about 200 input-side frames before an error without making it disappear.

Also: for reasons that escape me I had set warp to be the same as warpfast... I've fixed it, but also updated the MotionProtectedFPS script to use warpfast as default so that the behaviour is unchanged. If you use the old script with the new DLL, you may see a slowdown (but perhaps better quality... or it may look more blurry due to subpixel compensation). There shouldn't be slowdowns for any other reason... if you see them, please tell me.

scharfis_brain: that sounds pretty robust, but also pretty expensive? I will test it later when I have a bit more free time...

Edit: I've been meaning to say that while FindMotion more or less requires frames to be accessed in linear order, none of the other functions do (so long as they don't implicitly called FindMotion). So, you should be able to request more than one frame from them in parallel using tsp's SetMTMode (in any mode). Unfortunately I only have a single-threaded single-processor box so I can't test this... but I would be very interested in the results if anyone does.

[Fizick]

Originally Posted by mg262

I've tried implementing a MVFlowFPS-like method, and I was very impressed with the results. IIRC Fizick spent a lot of time and effort looking at artefact protection, and it shows. (Thank you, Fizick!)
...

If you use MVTools source (GPL), your source must be availible too.

[mg262]

If you use MVTools source (GPL)

I certainly haven't; not one line of it. What I used was your idea of blending two occlusion masks to create another mask... and a great idea it is, too. As it happens, it's actually implemented in the MotionProtectedFPS script rather than in Motion, so that the scripters can play with it.

FWIW, your implementation scales up the two small masks (you call them something like MaskB and MaskF) and then averages the results... you could probably manage it a little faster by averaging them first and then scaling up the result, (which is what the MotionProtectedFPS script does, by calling CombineFPS and then BilinearResize).

As before, you are more than welcome to the latest source of Motion to read and to pick out ideas from, if you would like?

Incidentally, Merry Christmas all!

[krieger2005]

i have a question how to use this thing because i have here a problem. I use this script:

encode=false
c=last
vf1b=FindMotion(c, from=previous, reset= encode ? 0 : 20, initialise=4)
cf1X=Compensate(c, warp,source=previous, motion=vf1b)
cf1b=cf1X.trim(1,0)

now the problem: It looks, that the compensated Clip cf1X does not fit to the frame of the clip c but the clip cf1b does. But cf1b have the same artifacts like c, so the compensation is not usefull.

So the general problem is, that the "normal compensated" clip seems to be one frame before the original.

Where is the problem?

EDIT: OK, i now remove for the call of the clip cf1X the source=previous and now cf1X give me the right compensated frame, BUT: it has the same artifacts as the original frame (but the frames before and after does not have this artifact). I want to remove these artifacts through the motion-compensation...

[mg262]

I can't reproduce the problem... it may be that the true motion estimation algorithm is not working with some kind of source. (Although removing source = previous shouldn't make any difference. I completely rewrote the relevant code at one point, and although I regression tested it, it may be that some glitch slipped through...)

Would you mind uploading a clip on which both problems occur? I'm particularly keen to see a section where removing source=previous makes a difference. The complete script would also be useful.

The motion algorithm converges pretty fast, so you should probably find that trimming out 50 or 100 frames before the problem frame will give you a clip that reproduces the issue. Although any size download is fine by me...

[krieger2005]

I uploaded a short snip to the net. You can get it here... Sorry for the length but a have only dial-in-connection.
Ok, here the exactly script i use. I used to return closures. Enable one of them...

LoadPlugIn("..\AVS\X\Motion_12Dec05.dll")

avisource("tst.avi")

encode=false

c=last

vf1b=FindMotion(c, from=previous, reset= encode ? 0 : 20, initialise=4)
cf1b=Compensate(c, warp,source=previous, motion=vf1b)

# return 1
return interleave(cf1b.trim(1,0).subtitle("first"),c,cf1b)

# return 2
return interleave(c,cf1b.trim(1,0))

when you enable "return 1" you can see first the trimmed frame, then the original clip and then the original-compensated. Please look at frame 261. Between the trimmed and original clip there is only minimal differences. But the original compensated jumps and looks like the frame before...

Now enable "return 2". Look for example at frame 152. There are much spots. These spots i want remove with despot-filter... But they are also in the compensation. Even better you can see it on frame 178 and so on...

[mg262]

The truemotion algorithm is becoming confused because the foreground object is very fast moving around frame 84 of the input (~frame 261 of your script with return 1). Unfortunately, this means that the compensation isn't working properly. This is a known issue with this kind of recursive algorithm (where the motion found for each frame depends on the motion found for the previous frame), and I've seen a suggested solution, but when I tested it it was pretty useless. I am sorry not to be more helpful. EDIT: Try adding iterate = 4 to FindMotion... results look much better.

For what it's worth, Trim(1, 0) is never necessary -- the compensated clip is just cf1b. In other words, frame 30 of cf1b *should* look like frame 30 of c, when all is working well.

When I try your script with return 1, and look at frames
165 (marked "first")
166
167

then frame 166 is much closer to frame 167 than either is to frame 165, as it should be. It easier to see the behaviour of the script using this return:

p=c.trim(1, 1)+c
interleave(p.subtitle("1. Previous"),cf1b.subtitle("2. Compensated"),c.subtitle("3. Current"))

So here 1 and 3 are consecutive source frames. 2. is made up of bits of 1., but *should* look like 3.
When estimation/compensation isn't working well, 2. will probably look like 1. For the purposes you are using this for, namely preprocessing for despot, that's not a catastrophe... the idea behind fast motion compensation with temporal filters is that while it is not always perfect, it should nearly always be better than using the filter without MC. In other words, we are replacing 1. with 2., and 2. is nearly always better than 1.

I tried to look for the other problem using this addition to your script:

other =Compensate(c, warp, motion=vf1b)
subtract(cf1b,other)
Levels(128 - 15, 1, 128 + 15, 16, 235)

but I couldn't reproduce the issue where omitting source = previous made a difference.

[milh31]

Great work mg262.

[krieger2005]

Originally Posted by mg262

The truemotion algorithm is becoming confused because the foreground object is very fast moving around frame 84 of the input (~frame 261 of your script with return 1). Unfortunately, this means that the compensation isn't working properly.

But the predicted Vectors look ok? But... humen eyes

Originally Posted by mg262

I am sorry not to be more helpful.

Is it possible to add/get an option or a mask, which show how good parts of the frames could be compensated by the filter?

Originally Posted by mg262

EDIT: Try adding iterate = 4 to FindMotion... results look much better.

Thanks, i will try this.

Originally Posted by mg262

For the purposes you are using this for, namely preprocessing for despot, that's not a catastrophe... the idea behind fast motion compensation with temporal filters is that while it is not always perfect, it should nearly always be better than using the filter without MC.

Despotting in general (not only with despot), for my optinion, is dangerous. So i would say, that when one don't know, that a filter can't compensate the frame good enough it is a catastrophe... But this is surely not the problem of your plugin. It make what it can...

Originally Posted by mg262

but I couldn't reproduce the issue where omitting source = previous made a difference.

Sorry.. this was a mistake by myself.

Thanks for your help

[Backwoods]

In the motion.htm, you have MotionCompensatedFPS() listed as a function. Do you mean MotionProtectedFPS()?