Motion filter : +1st MC FPS-change script for real usage

[communist]

I'm mostly testing this plugin with bobbed DV PAL footage that I wish to slowdown extremely - works good for some and doesnt for some other footage (some parts are slowed down constantly while others 'stutter'). But so far looks very promising - Thanks for this great plugin

I can post some problematic scripts / clips if needed but gotta clean it all up a bit

[Velocity 7]

Seems a bit quiet in this thread.

I've read through the entire thing, but could someone remind me if there was anything in regard to motion compensated deinterlacing (as opposed to motion compensated frame conversion)? For example, 30i -> 60p (for video game consoles outputting 60 fps, but squashing to 30i, or for example MiniDV cameras recording in 30i for 60 fps content, etc., see http://www.100fps.com for camera examples)*.

* - just posting as a reminder for those who might not know what the hell I'm talking about

[Mug Funky]

sure. if you can compensate frames, you can use them for deinterlacing.

basically you use a regular smart bobber (leakkernelbob works well for this purpose and is fast), then compensate 1 field onto the other, and use the compensation to fill the missing lines in the original field. then (the slow part) you make sure no artefacts get through.

i think mvbob has been modified already to use the latest mvtools, and i'm sure it can be further modified to support clouded's motion filter (which i prefer conceptually, but it needs tweaking to avoid peculiar artefacts. it's the same deal with fizick's improved mvtools really).

in short, if you can compensate you can deinterlace.

[Velocity 7]

What are the actual parameters to Compensate(), though? The documentation API is kind of off... (e.g., can't just use Compenaste() alone)

[mg262]

I'm sorry about that... I updated the documentation to the new filter version before I should have. The old documentation is still there at

Old Documentation

I haven't finished testing the new version to my satisfaction -- but I've become rather tied up in other things, so it will be a while before I can finish that. If you want to try it anyway, here it is:

Motion, 28 November 2005

Documentation

Old scripts should still work, if you make one change: rename Compensate to SimpleCompensate. [Or alternatively, name the motion argument of Compensate, i.e. source.Compensate(motion =...).]

I haven't abandoned this project, but it may be a bit before there are substantial updates.

I will write proper replies to some of the above shortly...

Edit: On the standard mode (i.e. not warp), any idea which kind of errors you're mainly seeing?

A. Incorrect motion estimation
The wrong motion vector is predicted and so the wrong block is chosen.

B. Occlusion/Revelation
The "right" block simply doesn't exist in the source frame because e.g. one object is moving in front of another.

Type A errors are hard to fix because there's not much leeway to play with the motion estimation algorithm (except by increasing iterate). Type B errors can be prevented by e.g. making use of the previous/next frame or bidirectional compensation; they can also be 'cured' by e.g. using masks to fall back to blending.

[mg262]

A script function (for use with the new version):

CombineBlendFPS changes the frame rate by combining forward and backwards compensation (like CombineFPS), but when the luma difference between the two is large, it gradually falls back to blending (like ConvertFPS).

Arguments:
input,
FPS,
mode -- move or warpfast or warp,
compdiff -- luma difference giving pure compensation,
blenddiff -- luma difference giving pure blending,
show -- set to true to show where blending is being used,

example usage:
CombineBlendFPS(100, warpfast)

A luma difference halfway between compdiff and blenddiff will give a 50:50 mix of compensation (i.e. CombineFPS) and blending (i.e. ConvertFPS). I recommend playing with compdiff and blenddiff ... lowering them will remove more motion artefacts, but increase blending, and vice versa.

function CombineBlendFPS(clip input, float FPS,\
 int "mode", int "compdiff", int "blenddiff", bool "show")
{
	mode = default(mode, move)
	compdiff = default(compdiff, 8)
	blenddiff = default(blenddiff, 14)
	show = default(show, false)

	input
	p=MotionFPS(FPS, mode)
	n=MotionFPS(FPS, mode, source = next)
	b=BlendFPS(FPS)
	c=CombineFPS(FPS, p, n)
	s=c.greyscale().Levels(0, 1, 128, 0, 255)

	yv12subtract (p,n,tol=compdiff)
	yv12lut("255 " + string(blenddiff-compdiff)+" / x *")	
	mask=fity2uv()

	return maskedmerge(c,show?s:b,mask,u=3,v=3)
}

I would be grateful if someone who is good with scripts checks that, especially the chroma processing. (Basic idea is to construct a mask of the luma difference, copy the mask to the chroma planes, and use a masked merge.)

This isn't the end of the story... I don't think luma differences are that reliable. But feedback on this will be useful for future development.

Edit: in addition to other stupidities I pasted in the version with the FPS hardwired to 70...

[Didée]

Seems okay to me, so far. Replacing the "levels()" with "YV12Lut()" would be a tad faster, and the "r2 = ReduceBy2() + YtoUV()" combo could be done more simply by "r2 = FitY2UV()".

Working with plain luma differences is somewhat like tieing up shoes while wearing thick mittens, indeed
It gets better when building a relation between the luma difference and the local detail level - but doing so, things get slower and slower: checks, more checks, even more checks ...

[mg262]

Didée,

Thank you very much! I've changed it as suggested, and also tweaked the defaults.

It gets better when building a relation between the luma difference and the local detail level - but doing so, things get slower and slower: checks, more checks, even more checks ...

I'm hoping that something more intelligent can be pulled out of the motion vectors... for example looking for discontinuities or for sudden changes from one frame to the next. But this script was lower-brainpower effort .

[Didée]

yv12lut("255" + string(blenddiff-compdiff)+"/ x *")

must be

yv12lut("255 " + string(blenddiff-compdiff)+" / x *")
            ^                                ^

[mg262]

Rrrr... I don't seem to be very good at scripts. Thank you!

I can speed this all up more with filterlets (e.g. scale, scaleandmerge, scalediffandemerge) if anyone finds it useful. + other FPS scripts will appear...

[mg262]

I've tried implementing a MVFlowFPS-like method, and I was very impressed with the results. IIRC Fizick spent a lot of time and effort looking at artefact protection, and it shows. (Thank you, Fizick!)

Motion, 6 December 05 (revised)
MotionProtectedFPS script

MaskTools is needed (but you should have it anyway ). Simple usage is just

MotionProtectedFPS(100)

This runs reasonably fast (AVSTimer measured it at 70 FPS).

You might get slightly higher quality by increasing iterate:

MotionProtectedFPS(100, iterate = 4)

(Runs at 60 FPS on my machine). Increasing protection may decrease the number of artefacts, at the cost of increased jerkiness -- but the default works pretty well, so don't increase it for the sake of it. The other arguments probably don't need fiddling with.

[ONLY INTERESTING TO SCRIPTERS]
I haven't documented SumStretchFPSMask yet, but it just creates an occlusion mask using the same method as Manao/Fizick. I've tried to write the script in such a way that it is easily readable by scripters (even the spacing ).

Mug Funky, scharfis, Didée (if you aren't too buried in other things), and anyone else interested in this stuff: if you have a moment try looking at the intermediates, especially maskp, maskn, maskn.invert(), and the output of CombineFPS. All these masks are small (1/64th the original, plus padding up to be legal resolution), so you need to use the bilinearresize/crop combo (see script) to look at them usefully.

The core of Fizick's method relies on using both temporal information like CombineFPS (i.e. if you are trying to construct a frame at time 63.1, you want to use frame 63 much more than frame 64) and also occlusion information (where frame 63 is useless, use frame 64 anyway). Try single-stepping maskedmerge(p, white_clip, ??) with each mask -- or maybe all 4 at once reduced -- and it should become pretty obvious what is going on. [But poke me if anything is unclear.]

Now: because the masks are so small, operations on them are essentially free. So you can try expands, inflates, etc -- also you can replace bilinearresize -- to your hearts content, to try and build better occlusion/combination masks. I have faith in you guys [/ONLY INTERESTING TO SCRIPTERS]

[scharfis_brain]

I get a problem with unmatchable objects, that are getting blended - obviously.
Their chroma isn't in place!

try this video: http://scharfisbrain.no-ip.info/doom9.avi

the script I used:

avisource("doom9.avi")
assumetff()
tdeint(mode=1,type=3,link=0,mthreshl=7)
converttoyv12()
Motionprotectedfps(framerate*4)

you'll notice, that the yellow moving flag has wrong chroma

[Didée]

In the last line of MotionProtectedFPS( ), write

maskedmerge(p, n, last, U=3,V=3 )

?

[scharfis_brain]

tried that. didn't make things better

[mg262]

Leaving aside the stupid bug (thank you, Didée!) I'm not sure I have got the right end of the stick... is it this thing (circled in red)?

Edit: picture removed.

That's a frame from the source, i.e. from

directshowsource("doom9.avi")
assumetff()
tdeint(mode=1,type=3,link=0,mthreshl=7)
converttoyv12()

[I can try to sort out use of AVISource if that makes a difference.]

[scharfis_brain]

Directshowsource messes up the chroma here.

please use AVISource and ensure that the codec delivers the native YUY2 of the source video.

The source has no temporal chroma problem.

but the video after motion interpolation looks like in your screenshot

[mg262]

I'm really sorry I keep making so many silly script bugs... as well as U=3,V=3 change, there needs to be a FitY2UV applied to the mask before the MaskedMerge (because the mask is just a luma mask). I've updated the script. I hope that fixes it (it seems to at my end)...

[scharfis_brain]

this simple script

avisource("doom9.avi")
assumetff()
#securedeint(6,1,2)
bob()
converttoyv12()
Motionprotectedfps(50)

stops interpolating new frames after frame 376 and shows the first frame unti the end of the video.

EDIT: seems to be an 1000/1001 thing again, cause interpolating to framerate*n works without problems.

then I have a question, how to avoid those jumpy texture movements by one pixel? is there some sore of subpixel accuracy?

mode = default(mode, warp(bilinear,halfpixel))

didn't show any improvement

[mg262]

Safety check became over-eager... now fixed:

Motion, 6 December 05 (revised)
MotionProtectedFPS script

Thank you for all the testing, scharfis.

Edit: try warp(bilinear, bilinear). [The second argument is used in the case where the motion of the four corners is the same, so the block can just be copied. But, general bilinear translation is still in C, so for the moment it's pretty slow.] If that doesn't work, I will have to think much harder about how to change the algorithms; qpel estimation may be necessary, although it's a nightmare to implement.

Part of the problem I have is that my eye for non-animated material is not very good at all -- could you give me a frame number where this effect is particularly distracting?

[foxyshadis]

While editing the vfr page on avisyth, I've had a bit of a revelation: Wouldn't this be the perfect filter to replace decimate modes 1 & 3? Mode 1 translates 24 fps segments to 30 by stretching 4 frames into 5 by blending, and mode 3 compresses 30 fps into 24 the same way. If someone wants a constant framerate video, motion compensation would be the best compromise between visual stability and compression efficiency. =D

The problem is that something would have to developed to detect the segments. Of course it would be fairly easy to make a bare-bones conditional function to do it, it just wouldn't be as fast or as featureful as Decimate/TDecimate. Still, I think I should give it a shot.