Motion filter : +1st MC FPS-change script for real usage

[krieger2005]

thanks for showing of the typo... too many changes in the script, as allways

I must corrent what i said above... Not the function FundMotion make the probelm but SWC, because if i let show the Vectors with "DrawMotion" on the original-clip it works ok. When i put out the compensated frames made by SWC VDub close.
However, it seems to be something with the "iterate"-Vaiable. Also the failure is when "source=previous" and "source=next". With "bilinear=false" the failure occur also.

[marcellus]

Hi, Clouded
Speaking of Motion... you are moving realy fast! :)
With the new function CombineFPS things are looking FAR better. :thanks: for your work!

In the week-end I had time to play with DePan and I didn't got good enough results, until I discovered Deshaker for VirtualDub (shame on me discovering it so late, I'm more accustomed to Avisynth than VDub processing). AntiFlicker and Deshaker do wonders with my clips, putting them in much better shape for a FPS conversion.

And now you come with this CombineFPS which works much better than anything else I tried (even on un-deshaked clips), and it's fast too. Thanks a lot! I'm begining to hope I will get where I want.

[scharfis_brain]

@clouded: thanks for blendfps.
May I suggest a boolean parameter that lets the user decide the blending mode:

option 1: behaviour like convertfps:
- never blend more that two frame together when doing framerate downconversions
- duplicate frames except for the frame to frame transition with extreme framerate upconversions.
- but fps paramter unrestriced

option 2:
- blend as many frames as needed together when the user defines a framerate downconversion.
- make a slow travelling blend weigth movement with extreme framerate upconversions. (so one can simulate slowmotion with blending)

[mg262]

krieger2005,

There was a typo in the warp code (copy/paste bug as ever... h in place of v) that caused a slight inaccuracy in the warping; luckily for me, you found a case where this caused it to access off-screen. Thank you very much!

I replaced the DLL...
Motion, 14 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_14Nov05.dll)
Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

Edit: I'm not sure the server has the new file... give it a few minutes
... it should be there now.

And, please, anyone using warping, download this version. I added more safety checks to the warping which may slow it down somewhat... but they are needed in some unusual situations. Please tell me if the slowdown is large.

marcellus,

I'm glad it's working well! It is only because I have incisive feedback from yourself and others that I can improve things. So, if you spot other reasons/circumstances where it doesn't work well, please tell me!

By the way, I think it is possible (but fiddly) to import Deshaker into AVISynth if needed; at least, I saw a thread on this once.

scharfis_brain,

Thanks for the suggestion. I will certainly put it in, when I have thought about it a little and thought of sensible parameter names (I dislike mode = 1, mode = 2, etc). Edit: what about a variable "aperture" for motion blur?

Grab Motion, 14 November 2005 (revised) (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_14Nov05C.dll) and try BlendFPS(float targetframerate[, float "aperture" = 1]). Increasing aperture will blend more frames; decreasing aperture will blend fewer.

[krieger2005]

Okey,
i tested the new version (not the c-version) and the old from 10Nov. Here my speed-results for 2 clips with 1000 frames:

#14Nov05:
#ca. 37-42 fps, 26 sec / ca. 37-40 fps, 26 sec
#10Nov05:
#ca 37-40 fps, 25 sec / ca. 36-40 fps, 26 sec

I have a P4 3000MHZ, HP enabled. The new version of the motion-filter does now not quit Vdub. thanks.

I would say, that if there is a speed drop, than not a big one, since both times are equal for a 1000 frame-clip, so i guess for a larger clip it would be the same...

But still VDub close. But this is one thing of my avs-scripts... they are too complex. But maybe you can add something in, so that your filter stop to allocate memory when a maximum is arrived because i had situations, where vdub take 2GB (swap-file).

BTW. I think also it would be great to use a global motion/MV-Format... Fizicks' MVFlow and your Vector-Analyzer would, i think, work great together.

Can you add (but i guess you have it) the scene-change-detection in your to-do-list?

great work.

[mg262]

Thank you for checking it.
maybe you can add something in, so that your filter stop to allocate memory when a maximum is arrived because i had situations, where vdub take 2GB (swap-file).I'm not allocating memory at all; I'm relying on AVISynth to do it for me. There is a mechanism to provide hints to AVISynth about what information should be kept, and I will try that to see if it helps. Could you give me a script that causes large memory usage?

As far as I know, the MVFlow???? functions and my warp functions do exactly the same thing (bilinear mesh compensation)... so there's not much point in mixing and matching.

Scene detection: will do. Also see here (http://forum.doom9.org/showthread.php?p=736198#post736198).

[krieger2005]

Here is such a script:
f=4
#analyze1 = last.FFT3DFilter(sigma=6).BilinearResize(m4(width*f), m4(height*f))
analyze1 = last.BilinearResize(m4(width*f), m4(height*f))
analyze2 = last.BilinearResize(m4(width*f), m4(height*f))
motionVB = FindMotion(analyze1, reset=0, initialise=20, iterate=6, from=previous)
motionCB = SWC(analyze2,motionVB, source=previous, bilinear=true).LanczosResize(width,height)
motionVF = FindMotion(analyze1, reset=0, initialise=20, iterate=6, from=next)
motionCF = SWC(analyze2,motionVF, source=next, bilinear=true).LanczosResize(width,height)
interleave(motionCB,last,motionCF)
temporalsoften(1,5,5,10,2)
SelectEvery(3,1)
i know that fact that with such supersampling one must have much memory, but i thought that there would be an error-message, so i thought you allocate the memory internal.


As far as I know, the MVFlow???? functions and my warp functions do exactly the same thing (bilinear mesh compensation)... so there's not much point in mixing and matching.
I don't know anything... i'm a noob :) The results looks just a little bit different. Maybe because of different motion-informations given by two different Algorithms...

Your idea for scene-change-detection-extra-filter is, i think, good, since very much filters need such an information (as you written in that thread). Do you want make it based on motion-informations made by, for example, "FindMotion" or a simple "luma-difference"-thing? But... i will see it when you release it :)

[mg262]

Careful when supersampling... you are effectively making everything move much faster, and I don't know how well the filter will cope. At least, please check vs the non-supersampled version. Also, as it says in the docs, iterate isn't really meant for general use... you are welcome to try it, but be aware that I haven't checked its effectiveness yet.

I've read all of Fizick's thread, but not all the documentation updates... so I will go and read up when I have a bit of time.

Scene detection: The idea is to have a standard scene representation (a scene type) and many different scene-detection filters which all output that type. Certainly SAD (= luma-difference?) and SSD will be in there.

I am dragging my feet on this because once a representation is shared between many filters, it becomes much harder to change -- so I need to get close to the right answer first time.*

*The "much harder" is for the case where all the filters are mine... the alternative is much worse. This is precisely what you see in formats like AVI... they can't be changed because they are too widely used, so you get all sorts of hacks being put in place and the situation ends up in a total mess. Same goes for AVISynth 2.5. Simulating associative arrays, etc. helps a bit but is far from a real solution.

[Fizick]

Stupid question:
What is SSD ?

[Manao]

SSD: Sum of Square Differences
SAD: Sum of Absolute Differences

[mg262]

I intend to allow auto-generation of motion data if motion isn't specified. So these

Compensate(FindMotion(from = next,initialise = 4), source = next)
SimpleFPS(100, FindMotion(initialise = 4))
SimpleWarpFPS(100, FindMotion(initialise = 4, from = next), from = next)

will become

Compensate(source = next)
SimpleFPS(100)
SimpleWarpFPS(100, from = next)

But first, FindMotion needs better defaults...

In particular: shall I set reset or not? And what kind of value?

[krieger2005]

For my taste i can't see any differences when i set reset. I set it mostly to 100-300. I think a value about 150 should work good enough for the most content and give a fast startpoint.

Just an other question. Since the filter compensate just a frame before/after to an actual frame there does not exists enough informations to compensate the picture correctly. For example on a left-shifted scene the filter can't reconstruct the left side. I am right till now?
Now it would be nice, if these informations could get from somewhere. Since this is a complex search in most cases a simple assumtation could be made:

A B C D E

is are 5 frames. If the frame C should get compensated then B could be used. For all unknown content frame E could be used. On the other side D and A can be used.

Is there a possibility to get something like that or is this all nonsense?

krieger2005

[foxyshadis]

Of course once you have a scenechange mechanism, you can truncate reset there everytime.

Is there planned to be any support for global luminance changes? Maybe there already is and it just causes a lot of extra hiccups or something.

krieger: That's what CombineFPS does at the moment (though clouded is working on a less cumbersome method). The 'not enough information' areas are called occulsions, and you're right, that is generally the best way to deal with them.

I've been testing out 120 fps on a few sources, mostly out of curiosity and to see whether I could break it. Sometimes it looks ghosty and ripply as hell, especially on fast motion (it gets trippppy), but at 120 it's not that big a deal, and the times when it looks amazingly smooth make up for that.

[mg262]

@krieger2005,

Setting lower values of reset will give you faster seeking -- but at a cost of either time or quality. If initialise is high, quality won't suffer -- but processing time increases by about (initialise/reset*100)%... so with your settings of 20 and 150, you get a 13% increase. I tend to use initialise = 4 and reset = 50, but I haven't yet run enough checks to be sure whether 4 is high enough (20 seems rather high).

The rest of what you said is definitely not nonsense -- in fact it contains the spirit of the methods used in some papers I've read. Adapting such things to warping is a little more work, though (but definitely possible).

@foxyshadis,

I'm definitely going to reset on scene changes! Global luminance changes... atm only fades come to mind; could you give me better examples?

@all

I'm going to quote something from the docs...[SimpleFPS] is not meant as a usable replacement for ConvertFPS; it is a tool to help find situations which are difficult for motion compensated frame rate change to deal with.FPS change is a battle I have not yet begun to fight... think of this as reconnaissance.* Complicated strategies can break things as well as fix them... so you need a simple benchmark that is not clever in any way to check them against. That's what SimpleFPS is for. It is also a speed reference; it may also be useful in your scripts (e.g. with masking)... but I'm not suggesting it as a ConvertFPS replacement.

*(And the two functions, SimpleWarpFPS and CombineFPS, I built on request because they were easily added.)

Edit: but it suits my purposes very well that you're playing with it :cool: ... because then you'll spot when other methods break things!

[foxyshadis]

Fades are really 95% of global changes... otherwise, brief flashes, drops, or auto contract correction.

One of the weird things I notice (I have a specific very regular panning clip I like for this) is that using combinefps, occusions will leave a rippling pattern in their wake that extends well beyond the passing of the object. Usually on the second original frame after the occulsion it settles down, rather than the first or even before. (Maybe it could use confidence weighting for mvs to decide whether forward or backward gets the most weight?)

Also many objects seem to move via a trippy blendfps, the blendweight changes but the object also... shrinks and melts toward the new position in big chunks. I'm not sure how to describe it. Protrusions (tops of logs) and thin objects suffer from it the most, I guess because there isn't enough... meat? to grab on to.

And unfortunately it doesn't seem to work too well with solid-color areas, when checking out the vectors it often seems to get confused and point to similar nearby spots instead. I can understand that as a limitation of the area matching, so I'm not bitching, just letting people know even for animation with well-defined motion it's to e used cautiously.

Oh, and concerning your earlier comment about mvflow, I noticed almost exactly the same artifacts in a clip in both methods, though not as pronounced in motion, so your algorithms are definitely very similar.

Last question, is seach-radius configurable? Or would that be silly for this application?

[mg262]

This is really useful stuff, foxyshadis. :thanks:

Fades first: I can't see a way to tweak the filter, but you could detect the fade in script (e.g. search for many consecutive blends), and treat it sensibly. E.g. luma-equalise the fade, motion detect on that, and compensate the original. Or: fade-compensated interleave, like this (0 0)(0 1)(1 2)(2 3)(3 4)… with each () pair luma-equalised, FindMotion with reset = 1, high initialise, and SelectOdd to get MVs. Similarly for drops (n.b. Mug Funky's point on using Compensate to fix them) and flashes. Auto-contrast... :scared:. I'll keep thinking about these.

The method always tries the motion vector from the previous frame, which is why the ripple can persist. There's no search radius, but increasing iterate has a similar effect... but e.g. iterate = 4 -> FindMotion is 4x slower.

Maybe it could use confidence weighting for mvs to decide whether forward or backward gets the most weight?'Confidence' is a better term than I had -- I shall steal it shamelessly :D. I was thinking of this:

CombineFPS(forwards, backwards, fmask, bmask, fallback)

using a weighted combination of forwards and backwards, with weights depending both on temporal closeness to original and the two confidence masks; if both masks were low, it would blend in fallback as well. That should also deal with screen edges and small/thin objects. It won't fix flat areas... I have some ideas on that and on faster methods, but that's for later.

For now, the right thing to do is to play with confidence masks until we build up intuition about them. After I finish tidying, I will throw out a few masks (maybe 2 filters + 2 scripts) and rely on yous to tell me which ones work well for what kind of content :).

[mg262]

I'm soon going to put in the new interface, but I wanted opinions on a couple of things first. I'll illustrate it with FPS conversion (compensation will look almost identical).

SimpleFPS and SimpleWarpFPS are going to be replaced* with one function FPS, that can be used like this:

FPS(100, move) #works like SimpleFPS(100,...)
FPS(100, warp) #works like SimpleWarpFPS(100,...)
FPS(100, warp(bilinear)) #works like SimpleWarpFPS(100,bilinear = true,...)
FPS(100) # default mode = move, so works like SimpleFPS(100,...)

*well, I'll leave them around for a bit so as not to break scripts.

Here's the full argument list

(source) clip
float targetFPS
mode = move/warp/warp(bilinear)/... [default = move]
chromamode = move/warp/warp(bilinear)/... [default = same as mode]
clip motion [default = autogenerated]
source = previous/next [default = previous]

I hope that's clear. If motion isn't specified, it will be auto-found from the source; otherwise, you can use FindMotion(...) as before OR you can pass another clip of the same size as the source, and motion will be found from that. So a more complicated example would be this:

denoised = input.SpatialSmooth(...)
input
FPS(100, mode = warp(bilinear), chromamode = move, motion = denoised)

This applies warp-compensation to the luma plane with bilinear accuracy, but constructs a chroma plane by moving blocks; motion is automatically found from the clip denoised.

[Mode arguments can be passed through functions... so CombineFPS(100, ???) will call FPS(100, ???, source = previous) and FPS(100, ???, source = next) and amalgamate the results. So you will be able to write e.g. CombineFPS(100, warp)... a bit more compact than the current version! Or, I may add the option source = both.]

How's it sound? In particular, I'd welcome any thoughts on sensible argument order -- targetFPS + mode should be probably be first, but after that I'm not sure...

[marcellus]

I'm soon going to put in the new interface
Great news! :cool:
How's it sound?
Changes are looking good.
One little thing:
warp(bilinear) looks weird when I read the script, like bilinear is a clip.
Maybe warp_bilinear or something like that would be better. Or you could keep bilinear=true with no effect when mode is 'move'.

I hope you will keep 'iterate' as a parameter, I obtain better results when I increase it, the result pays the speed decrease. And maybe you could 'unveil' more parameters (regarding especially the motion search), in general I have no problem with a lot of parameters as long they are well explained in the documentation.

I'd welcome any thoughts on sensible argument order
Personally I like to use in scripts named arguments like:
FPS(targetFPS=100, mode=warp)
instead of
FPS(100,warp)
because I have a poor memory so when I read the script after half a year the first version looks more intuitive, so I can remember what I was doing. But the natural order seems IMHO:
FPS(clip,targetFPS,mode,chromamode,source,motion)

[foxyshadis]

The parenthesis are kind of weird... "warpbilinear" would work as well. As for argument order it seems great now, the last two could swing either way. Source=both sounds more coherent overall than a separate combinefps function, even if they'd be exactly the same (ie, source=both just calls source=next/last).

The main thing I can see is that maybe it should be MotionFPS; I'm sure there's no plain "FPS" now but it does seem like namespace pollution. =p

[mg262]

Warp is currently a function... syntax like this:

Warp(accuracy = pixel, moveaccuracy = halfpixel)
[each accuracy can be set to pixel/halfpixel/bilinear... but I may add more modes]

So you could write
FPS(100, mode = warp(accuracy = bilinear))

Is it less confusing like that?

More advanced modes will need the function syntax [e.g. mode = rigidwarp(rigidity = 3.5)] so it should probably stay in for consistency -- and it allows finer control from script... but I could also add lots of constants like warppixelpixel, warppixelhalfpixel, warppixelbilinear, (and maybe warppixel, warphalfpixel, warpbilinear) for those that prefer them?

I felt names like SimpleWarpCompensate were getting out of control, so I've been trying to push the other way. I wasn't sure about FPS either... but if I use MotionFPS then I should use MotionCompensate for consistency... and that seems too long to inflict on people. Plus, the syntax for FPS is such that it is unlikely to clash with another FPS function (though I can't see what one would be). Maybe I could add both FPS and MotionFPS, Compensate and MotionCompensate... :confused:

Namespace issues worry me a lot... but OTOH that comes from the bit of my brain that works on properly-typed extensively structured languages, and often such intuitions turn out to be v. inappropriate for AVISynth. I don't know... :confused: :( .

[Fizick]

May be I am wrong, but you can not use mode=warp, it must be mode="warp", i.e.string.

[marcellus]

I felt names like SimpleWarpCompensate were getting out of control, so I've been trying to push the other way.
As long the function does it's job I have no reason to complain for function names being too long, as long they are intuitive (we all do copy/paste anyway). A clear and detailed documentation is more important. Having multiple names for same function might be confusing though, IMHO.


So you could write
FPS(100, mode = warp(accuracy = bilinear))

Is it less confusing like that?

More advanced modes will need the function syntax [e.g. mode = rigidwarp(rigidity = 3.5)] so it should probably stay in for consistency -- and it allows finer control from script... but I could also add lots of constants like warppixelpixel, warppixelhalfpixel, warppixelbilinear, (and maybe warppixel, warphalfpixel, warpbilinear) for those that prefer them?
Wouldn't be better, for example:
FPS(100, mode = warp,accuracy = bilinear,rigidity=3.5,subpixel =true)
and explain every parameter and their relationship in the documentation? And when, somebody writes by mistake:
FPS(100, mode = move,accuracy = bilinear)
the accuracy parameter should be ignored because has no signification in mode "move" (or throw an error message).
That's the kind of synthax I'm used to.
(Previewing my post I noticed that while I was writing Fizick made a post too and I think he's right.)

[Fizick]

IMHO, FPS() or WARP() is not appropriarte names (too general).
(Do you remember recent V.C.Mohan's plugin with too general name "Denoise" ?)

But by the way, first alpha versions of MVTools were named "Motion" (Motion.dll") :)

[foxyshadis]

I see! I was confused, I thought mode= was always a string and that was just a weird way to put it. Now it makes a lot more sense. (Certainly "profiles" would be a great idea, deen proves their popularity, but they can come later if it's a hassle.)

I forgot again that names can overlap. I don't think there's enough of a standard in avisynth to say "it has to be prefixed" or vice versa, but in some way it should identify its source (like MDeblock vs. just Deblock).

[mg262]

I don't use strings because AVISynth doesn't copy them properly -- so certain planned features would require leaking memory or something else unpleasant. Profiles -- definitely! I was thinking of defining HQ and LQ as arguments to be used anywhere... like this

FPS(100, HQ)
FPS(100, warp(HQ))
FPS(100, mode = HQ, chromamode = LQ)

I have taken all the remarks on name conflicts on board... it's just that something like
MotionFPS(100, mode = motionwarp(accuracy = motionbilinear, moveaccuracy = motionhalfpixel), from = motionprevious)
seems to me to become completely unreadable.

At root, it comes down to the fact that the language doesn't have implicit or explicit namespaces or (productive) types,* and I at least find it very difficult to work without these. Type-based function overloading definitely isn't the right mechanism for dealing with namespace conflicts, but in the absence of a proper mechanism, it's better than most alternatives. (I.e.: I'm not so worried about using warp because it doesn't take a clip as an argument -- so it won't conflict with other functions.) And, many natural names have already been taken... consider Invert, Deblock, RemoveGrain, DeGrain, UnBlend, DeBlend, RemoveBlend; although in theory it could cause trouble, I haven't seen it happen often if ever. [Warpsharp is the only exception that comes to mind.]

Having been heavily criticised (not here) for making things too inaccessible, I'm trying to make things as un-intimidating for newbies as possible... and making things short is one of the main ways to do this.

*There is a pseudo-namespace feature, in that you can write DLLName_functionname(...).

Edit: I forgot to say anything about iterate. I'll certainly leave it in if you want it... but I haven't yet put in the tweaks that might make it more effective. (Plus I think of this as a faster, lower quality alternative to MVTools... ) I'd like to have a way to set reset, iterate, etc in the new syntax... my preferred method would be this:

FPS(100,..., motion = auto(iterate = 2),...)

i.e. have auto be a function just like warp, with overrideable defaults.

[tsp]

I don't use strings because AVISynth doesn't copy them properly -- so certain planned features would require leaking memory or something else unpleasant. Profiles -- definitely! I was thinking of defining HQ and LQ as arguments to be used anywhere... like this
mg262: Did you use env->SaveString(const char* s, int length = -1); when using strings? or when does the memory leak happend?

[mg262]

No... but as far as I know that doesn't deallocate until everything is unloaded. It's not that I have a resource problem due to losing a few hundred bytes due to leakage; I just find the idea of things persisting beyond their appropriate lifetime... deeply unpleasant. It makes me want to wash my hands ;).

[tsp]

you could say that by calling SaveString you link the string with the IScriptEnvironment so it is okay that it is first destroyed when the last reference to it disappear :) In fact most avisynth filters that handles strings uses SaveString meaning if you uses them with ScriptClip or FrameEvaluate they will allocate memory for the string argument for every frame because the constructor is called each frame and the memory is first freed when the IScriptEnvironment is destroyed. I don't know if that is what you mean with memory leak but it only happends when using ScriptClip or FrameEvaluate?

[mg262]

tsp,

:( I'm sorry, I've given you the wrong impression. I haven't measured a leak. I'm not concerned about resources. I don't like this for aesthetic reasons, just as I don't like global variables, or typecasting pointers to ints, or goto statements, or non-constant static variables for aesthetic reasons. Working in these ways... just makes me feel horrible inside :( .

(Please don't let me give the wrong impression. I'm not trying to be rude or criticise the way anyone else chooses to work. @all, do as you see fit :) .)

Edit: I missed this too
FPS(100, mode = warp,accuracy = bilinear,rigidity=3.5,subpixel =true)The thing about this is that it doesn't let you pass the mode around easily. So script functions might want to e.g. call FPS twice (once with previous, once with next), mask the results in some way, and then combine them. Currently they can take mode as an argument and pass it into both calls... that becomes much, much messier & harder if the mode info. is scattered across many parameters.

[mg262]

1. You're all right, MotionFPS it is.

2. Two alternatives for mode = warp(accuracy = bilinear) syntax have been suggested:
-- Add all the arguments (accuracy, moveaccuracy, rigidity, chromaaccuracy, chromamoveaccuracy, chromarigidity etc) to every function

A simple script that e.g. generates forwards and backwards motion (with a specified mode), creates masks and combines motion will need a line like "accuracy = default(accuracy,...)" for each argument.
An editor like AVSEdit will end up displaying every argument for every mode, which becomes unreadable.
It's harder to write newbie-friendly docs, because they have to be split up into lots of cases.

-- Use strings instead (leaving aside my distaste for these...)

Unless I write a proper parser, the syntax will be more restrictive than AVISynth.
Numeric arguments (e.g. rigidity = 3.5) will need to be stuffed into strings and extracted again (ugly).
Scripts will have to write things like "warp[" + mode+"]" rather than warp(mode).

So I'm sticking with the current syntax. BUT

3. There will be plenty of presets
...so no one has to delve into this syntax unless they want to try something advanced.

Just for fun, the graph below shows relative time/frame of each mode. (Warp has two arguments because it can often move a block without stretching it; the second argument tells it what interpolation to use when doing this.) You can see that moving things with half-pixel accuracy is just as fast as moving them with pixel accuracy... so there are just 3 sensible modes at present. I was thinking move, warpfast and warp? (And bilinear move-accuracy will give moveslow and warpslow.) Any better ideas?

http://people.pwf.cam.ac.uk/mg262/posts/Motion/FPS 100 times.PNG

Apart from CombineFPS/both, it's ready to release once I have names for everything. [BTW, I might use "method" instead of "mode"... I think it's more descriptive.]

[communist]

I'm mostly testing this plugin with bobbed DV PAL footage that I wish to slowdown extremely - works good for some and doesnt for some other footage (some parts are slowed down constantly while others 'stutter'). But so far looks very promising - Thanks for this great plugin :)

I can post some problematic scripts / clips if needed but gotta clean it all up a bit ;)

[Velocity 7]

Seems a bit quiet in this thread.

I've read through the entire thing, but could someone remind me if there was anything in regard to motion compensated deinterlacing (as opposed to motion compensated frame conversion)? For example, 30i -> 60p (for video game consoles outputting 60 fps, but squashing to 30i, or for example MiniDV cameras recording in 30i for 60 fps content, etc., see http://www.100fps.com for camera examples)*.

* - just posting as a reminder for those who might not know what the hell I'm talking about

[Mug Funky]

sure. if you can compensate frames, you can use them for deinterlacing.

basically you use a regular smart bobber (leakkernelbob works well for this purpose and is fast), then compensate 1 field onto the other, and use the compensation to fill the missing lines in the original field. then (the slow part) you make sure no artefacts get through.

i think mvbob has been modified already to use the latest mvtools, and i'm sure it can be further modified to support clouded's motion filter (which i prefer conceptually, but it needs tweaking to avoid peculiar artefacts. it's the same deal with fizick's improved mvtools really).

in short, if you can compensate you can deinterlace.

[Velocity 7]

What are the actual parameters to Compensate(), though? The documentation API is kind of off... (e.g., can't just use Compenaste() alone)

[mg262]

I'm sorry about that... I updated the documentation to the new filter version before I should have. The old documentation is still there at

Old Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion_old.html)

I haven't finished testing the new version to my satisfaction -- but I've become rather tied up in other things, so it will be a while before I can finish that. If you want to try it anyway, here it is:

Motion, 28 November 2005 (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_28Nov05.dll)

Documentation (http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html)

Old scripts should still work, if you make one change: rename Compensate to SimpleCompensate. [Or alternatively, name the motion argument of Compensate, i.e. source.Compensate(motion =...).]

I haven't abandoned this project, but it may be a bit before there are substantial updates.

I will write proper replies to some of the above shortly...

Edit: On the standard mode (i.e. not warp), any idea which kind of errors you're mainly seeing?

A. Incorrect motion estimation
The wrong motion vector is predicted and so the wrong block is chosen.

B. Occlusion/Revelation
The "right" block simply doesn't exist in the source frame because e.g. one object is moving in front of another.

Type A errors are hard to fix because there's not much leeway to play with the motion estimation algorithm (except by increasing iterate). Type B errors can be prevented by e.g. making use of the previous/next frame or bidirectional compensation; they can also be 'cured' by e.g. using masks to fall back to blending.

[mg262]

A script function (for use with the new version):

CombineBlendFPS changes the frame rate by combining forward and backwards compensation (like CombineFPS), but when the luma difference between the two is large, it gradually falls back to blending (like ConvertFPS).

Arguments:
input,
FPS,
mode -- move or warpfast or warp,
compdiff -- luma difference giving pure compensation,
blenddiff -- luma difference giving pure blending,
show -- set to true to show where blending is being used,

example usage:
CombineBlendFPS(100, warpfast)

A luma difference halfway between compdiff and blenddiff will give a 50:50 mix of compensation (i.e. CombineFPS) and blending (i.e. ConvertFPS). I recommend playing with compdiff and blenddiff ... lowering them will remove more motion artefacts, but increase blending, and vice versa.

function CombineBlendFPS(clip input, float FPS,\
int "mode", int "compdiff", int "blenddiff", bool "show")
{
mode = default(mode, move)
compdiff = default(compdiff, 8)
blenddiff = default(blenddiff, 14)
show = default(show, false)

input
p=MotionFPS(FPS, mode)
n=MotionFPS(FPS, mode, source = next)
b=BlendFPS(FPS)
c=CombineFPS(FPS, p, n)
s=c.greyscale().Levels(0, 1, 128, 0, 255)

yv12subtract (p,n,tol=compdiff)
yv12lut("255 " + string(blenddiff-compdiff)+" / x *")
mask=fity2uv()

return maskedmerge(c,show?s:b,mask,u=3,v=3)
}I would be grateful if someone who is good with scripts checks that, especially the chroma processing. (Basic idea is to construct a mask of the luma difference, copy the mask to the chroma planes, and use a masked merge.)

This isn't the end of the story... I don't think luma differences are that reliable. But feedback on this will be useful for future development.

Edit: in addition to other stupidities I pasted in the version with the FPS hardwired to 70...

[Didée]

Seems okay to me, so far. Replacing the "levels()" with "YV12Lut()" would be a tad faster, and the "r2 = ReduceBy2() + YtoUV()" combo could be done more simply by "r2 = FitY2UV()".

Working with plain luma differences is somewhat like tieing up shoes while wearing thick mittens, indeed ;)
It gets better when building a relation between the luma difference and the local detail level - but doing so, things get slower and slower: checks, more checks, even more checks ...

[mg262]

Didée,

Thank you very much! I've changed it as suggested, and also tweaked the defaults.
It gets better when building a relation between the luma difference and the local detail level - but doing so, things get slower and slower: checks, more checks, even more checks ...I'm hoping that something more intelligent can be pulled out of the motion vectors... for example looking for discontinuities or for sudden changes from one frame to the next. But this script was lower-brainpower effort :).

[Didée]

yv12lut("255" + string(blenddiff-compdiff)+"/ x *") must be
yv12lut("255 " + string(blenddiff-compdiff)+" / x *")
^ ^ ;)

[mg262]

Rrrr... I don't seem to be very good at scripts. Thank you!

I can speed this all up more with filterlets (e.g. scale, scaleandmerge, scalediffandemerge) if anyone finds it useful. + other FPS scripts will appear...

[mg262]

I've tried implementing a MVFlowFPS-like method, and I was very impressed with the results. IIRC Fizick spent a lot of time and effort looking at artefact protection, and it shows. (Thank you, Fizick!)

Motion, 6 December 05 (revised) (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_06Dec05B.zip)
MotionProtectedFPS script (http://people.pwf.cam.ac.uk/mg262/posts/Motion/MotionProtectedFPS.avsi)

MaskTools is needed (but you should have it anyway ;)). Simple usage is just

MotionProtectedFPS(100)

This runs reasonably fast (AVSTimer measured it at 70 FPS).

You might get slightly higher quality by increasing iterate:

MotionProtectedFPS(100, iterate = 4)

(Runs at 60 FPS on my machine). Increasing protection may decrease the number of artefacts, at the cost of increased jerkiness -- but the default works pretty well, so don't increase it for the sake of it. The other arguments probably don't need fiddling with.

[ONLY INTERESTING TO SCRIPTERS]
I haven't documented SumStretchFPSMask yet, but it just creates an occlusion mask using the same method as Manao/Fizick. I've tried to write the script in such a way that it is easily readable by scripters (even the spacing :p).

Mug Funky, scharfis, Didée (if you aren't too buried in other things), and anyone else interested in this stuff: if you have a moment try looking at the intermediates, especially maskp, maskn, maskn.invert(), and the output of CombineFPS. All these masks are small (1/64th the original, plus padding up to be legal resolution), so you need to use the bilinearresize/crop combo (see script) to look at them usefully.

The core of Fizick's method relies on using both temporal information like CombineFPS (i.e. if you are trying to construct a frame at time 63.1, you want to use frame 63 much more than frame 64) and also occlusion information (where frame 63 is useless, use frame 64 anyway). Try single-stepping maskedmerge(p, white_clip, ??) with each mask -- or maybe all 4 at once reduced -- and it should become pretty obvious what is going on. [But poke me if anything is unclear.]

Now: because the masks are so small, operations on them are essentially free. So you can try expands, inflates, etc -- also you can replace bilinearresize -- to your hearts content, to try and build better occlusion/combination masks. I have faith in you guys :) [/ONLY INTERESTING TO SCRIPTERS]

[scharfis_brain]

I get a problem with unmatchable objects, that are getting blended - obviously.
Their chroma isn't in place!

try this video: http://scharfisbrain.no-ip.info/doom9.avi

the script I used:

avisource("doom9.avi")
assumetff()
tdeint(mode=1,type=3,link=0,mthreshl=7)
converttoyv12()
Motionprotectedfps(framerate*4)

you'll notice, that the yellow moving flag has wrong chroma

[Didée]

In the last line of MotionProtectedFPS( ), write

maskedmerge(p, n, last, U=3,V=3 )

?

[scharfis_brain]

tried that. didn't make things better

[mg262]

Leaving aside the stupid bug (thank you, Didée!) I'm not sure I have got the right end of the stick... is it this thing (circled in red)?

Edit: picture removed.

That's a frame from the source, i.e. from

directshowsource("doom9.avi")
assumetff()
tdeint(mode=1,type=3,link=0,mthreshl=7)
converttoyv12()

[I can try to sort out use of AVISource if that makes a difference.]

[scharfis_brain]

Directshowsource messes up the chroma here.

please use AVISource and ensure that the codec delivers the native YUY2 of the source video.

The source has no temporal chroma problem.

but the video after motion interpolation looks like in your screenshot

[mg262]

I'm really sorry I keep making so many silly script bugs... as well as U=3,V=3 change, there needs to be a FitY2UV applied to the mask before the MaskedMerge (because the mask is just a luma mask). I've updated the script. I hope that fixes it (it seems to at my end)...

[scharfis_brain]

this simple script

avisource("doom9.avi")
assumetff()
#securedeint(6,1,2)
bob()
converttoyv12()
Motionprotectedfps(50)

stops interpolating new frames after frame 376 and shows the first frame unti the end of the video.

EDIT: seems to be an 1000/1001 thing again, cause interpolating to framerate*n works without problems.

then I have a question, how to avoid those jumpy texture movements by one pixel? is there some sore of subpixel accuracy?

mode = default(mode, warp(bilinear,halfpixel))

didn't show any improvement

[mg262]

Safety check became over-eager... now fixed:

Motion, 6 December 05 (revised) (http://people.pwf.cam.ac.uk/mg262/posts/Motion/Motion_06Dec05B.zip)
MotionProtectedFPS script (http://people.pwf.cam.ac.uk/mg262/posts/Motion/MotionProtectedFPS.avsi)

Thank you for all the testing, scharfis.

Edit: try warp(bilinear, bilinear). [The second argument is used in the case where the motion of the four corners is the same, so the block can just be copied. But, general bilinear translation is still in C, so for the moment it's pretty slow.] If that doesn't work, I will have to think much harder about how to change the algorithms; qpel estimation may be necessary, although it's a nightmare to implement.

Part of the problem I have is that my eye for non-animated material is not very good at all -- could you give me a frame number where this effect is particularly distracting?

[foxyshadis]

While editing the vfr page on avisyth, I've had a bit of a revelation: Wouldn't this be the perfect filter to replace decimate modes 1 & 3? Mode 1 translates 24 fps segments to 30 by stretching 4 frames into 5 by blending, and mode 3 compresses 30 fps into 24 the same way. If someone wants a constant framerate video, motion compensation would be the best compromise between visual stability and compression efficiency. =D

The problem is that something would have to developed to detect the segments. Of course it would be fairly easy to make a bare-bones conditional function to do it, it just wouldn't be as fast or as featureful as Decimate/TDecimate. Still, I think I should give it a shot.