Motion filter : +1st MC FPS-change script for real usage

[Leak]

Originally Posted by foxyshadis

A friend actually came to me last week with a handful of PAL dvds that he'd bought, asking if I could do anything with them because they don't work with his DVD player (or tv? I don't know anything about that). Sure he can watch them on his laptop, but...

How about using DGPulldown (combined with reencoding the video to the correct resolution) for this? At least if you want to get the conversion done this year...

np: Blockhead - Crashing Down (Downtown Science)

[mg262]

This...

Originally Posted by marcellus

A----B----C will become
AMNPQBRSTUC

Now, I observed that the new generated frames M,N,P,Q are warped versions of the frame A, and the more far we go (from M to Q) the more distorted they are, Q being very distorted but still, much closer to A than to B. When I play the movie this is giving a "pulsing" effect...

... is really useful feedback. It makes perfect sense that this occurs, but my eyes aren't good enough to see it -- so I have to rely on you guys! Thank you very much.

My suggestion would be to make M and N frames by warping the A frame but P and Q warping the B frame. Or even better, a weighted warping that take into consideration both A and B.

I considered the latter but abandoned it on speed grounds -- there are less time-expensive ways of improving the picture quality. OTOH, it's quicker to add and doesn't mess with my basic framework, so...

Motion, 13 November 2005
Documentation

Example usage:

#toconvert = AVISource...
toconvert
fromprevious = SimpleFPS(100, FindMotion(initialise = 4))
fromnext = SimpleFPS(100, FindMotion(initialise = 4, from = next), source = next)
toconvert.CombineFPS(100, fromprevious, fromnext)

NB new argument order for frame rate conversion functions; it's now source.simple[warp]FPS(framerate,...). (IMO much more readable!) If you're using simplewarpFPS, and you are more concerned about quality than speed, I recommend using bilinear = true.

Please note that I'm by no means a programmer so take my suggestion accordingly...

Suggestions from non-programmers are particularly valuable. When deciding what to do (rather than how to do it), programming doesn't come into it at all... and it's easy for a programmer to e.g. not consider something because it is slow, even if it is the right solution. So please, suggest away!

[Alain2]

Hi,

Thank you mg262 for this new filter, it seems to be a very interesting work, and a fast filter if people get 44fps speed with it

Originally Posted by mg262

For denoising, I remember you saying that FFT3D worked particularly well with motion compensation...

I read in the documentation the example for "motion compensated denoising", in which you place a temporal smoother radius = 1 after the introduction of motion compensated frames; Can this temporal smoother be fft3d as you seem to suggest? Is there a particular reason for a radius 1? Can you or Mug Funky points me to the post where the motion compensated + fft3d combination is discussed?

Thanks

[psme]

Hi scharfis_brain and Clouded,

Will there be a faster mvbob using this new MC filter? Or these are not related issue? I always heard that MC deinterlace is the best method for real 60i video content. Thanks in advance.

regards,

Li On

[Mug Funky]

Originally Posted by Alain2

Can you or Mug Funky points me to the post where the motion compensated + fft3d combination is discussed?

it's pretty simple. fft3d has a temporal radius of 1 in bt=3 mode.

so you do something like this:

interleave(forward_compensated, clip, back_compensated)
fft3dfilter(... , ... , ... , bt=3)
selectevery(3,1)

obviously fill the holes with your favourite settings, and use any method you prefer for getting the forward and backward compensated clips (motion or mvtools).

no particular reason to have a temporal radius of 1 - just use as many compensated frames as you need (if you're going 1x forward and 1x backward compensated frames, then temporal radius is going to be 1. it's not wise to have a temporal radius larger than the distance you're compensating).

[mg262]

Alain2,

Good to hear from you again! Mug Funky has answered your question, but you might also find it useful to open up the documentation and scroll down to the heading Example.

psme,

I think that interest is definitely running in this direction... but it needs me to write a robust frame rate conversion function. Or else an effective mask showing where the compensation is working well and where it isn't. On that note...

@all,
For a mask showing how effective the motion compensation during frame rate compensation is, what about trying the (absolute) difference between forwards and backwards compensation? I.e. between these two things:

SimpleFPS(100, FindMotion(initialise = 4))
SimpleFPS(100, FindMotion(initialise = 4, from = next), source = next)

I've had a quick look at it and it seems interesting. I'm going to try coding something else as well... but let me know what you think of this one. Even if it messes up, it would be useful to know why it messes up...

[Mug Funky]

i liked the idea in the mvtools thread about getting a more traditional metric from a full compensation (time=100 or whatever), and compensating that mask to the desired time value.

but difference between 2 compensations seems pretty straightforward too. but it doesn't allow for the possibility that both compensations are wrong. this shouldn't come up too often though (generally with occlusion type situations when one compensation fails the other succeeds in the same areas and vice versa).

[mg262]

i liked the idea in the mvtools thread about getting a more traditional metric from a full compensation (time=100 or whatever), and compensating that mask to the desired time value.

Sounds good... seems possible with the existing functions? If not, tell me what is needed. Maybe it would be useful to have some kind of difference function other than just Overlay(mode = "difference")?

I added a quick function to help when playing with masks:

Motion, 14 November 2005
Documentation

BlendFPS -- works more or less like ConvertFPS, but supports YV12 -- also should deal with arbitrary colour spaces and arbitrary frame rates.* Unlike ConvertFPS, it will average more than two frames (mainly applies when decreasing FPS by a large factor)... so it might be useful for motion blur.

*It ain't 100% clear what the "correct" weights for general blend conversion are; I've picked values that have some physical motivation. If they don't look right, let me know.

Don't worry if mask-related scripts become slow... there's plenty I can do to speed them up. But "premature optimisation is the root of all evil" .

Last thing: the DLL seems to have shrunk considerably without any very clear explanation... all my standard tests seem to be working okay, but please let me know if anything unusual happens.

[krieger2005]

I had a problem with your plugin and want tell it... maybe it is a bug. I use your filter of 10Nov05. I used it in this way:

# last, in high motion precise, motion builder
motionVB = FindMotion(last, reset=300, initialise=20, iterate=6, from=previous)
motionCB = SWC(last,motionVB, source=previous, bilinear=true)
motionVF = FindMotion(last, reset=300, initialise=20, iterate=6, from=next)
motionCF = SWC(last,motionVB, source=next, bilinear=true)

interleave(motionCB,last,motionCF)
Despot(mscene=10, pwidth=260, minpts=20, maxpts=260, pheight=160, p1=23,p2=10,mthres=16, merode=19, color=true, fitluma=true)
Despot(mscene=10, pwidth=140, maxpts=50, minpts=9, pheight=140, p1=22,p2=10,mthres=12, merode=18, color=true, fitluma=true)
SelectEvery(3,1)

I tried it and it works well, but on a special situation VDub crashed (closed without any message). I looked at this position (it was frame 4232 but however). It is a scene where a static picture was shifted... like a panorama-picture, which was shifted from left-to-right side. The shift was very fast...

I looked at the code and the problem seems to be in the FindMotion-Function. Especialy if the variable "iterate" decrease the position, when VDub crashes, is later. if "itearate=1" VDub does not crash on these scene...

If youwant these clip a have to manage the upload but maybe you does not need it.

Thanks for development

[mg262]

krieger2005,

thanks for your message!

motionVF = FindMotion(last, reset=300, initialise=20, iterate=6, from=next)
motionCF = SWC(last,motionVB, source=next, bilinear=true)

There is a typo here -- I think you mean motionVF. I'm not sure whether that caused the crash though -- I will look into it.

[krieger2005]

thanks for showing of the typo... too many changes in the script, as allways

I must corrent what i said above... Not the function FundMotion make the probelm but SWC, because if i let show the Vectors with "DrawMotion" on the original-clip it works ok. When i put out the compensated frames made by SWC VDub close.
However, it seems to be something with the "iterate"-Vaiable. Also the failure is when "source=previous" and "source=next". With "bilinear=false" the failure occur also.

[marcellus]

Hi, Clouded
Speaking of Motion... you are moving realy fast!
With the new function CombineFPS things are looking FAR better. for your work!

In the week-end I had time to play with DePan and I didn't got good enough results, until I discovered Deshaker for VirtualDub (shame on me discovering it so late, I'm more accustomed to Avisynth than VDub processing). AntiFlicker and Deshaker do wonders with my clips, putting them in much better shape for a FPS conversion.

And now you come with this CombineFPS which works much better than anything else I tried (even on un-deshaked clips), and it's fast too. Thanks a lot! I'm begining to hope I will get where I want.

[scharfis_brain]

@clouded: thanks for blendfps.
May I suggest a boolean parameter that lets the user decide the blending mode:

option 1: behaviour like convertfps:
- never blend more that two frame together when doing framerate downconversions
- duplicate frames except for the frame to frame transition with extreme framerate upconversions.
- but fps paramter unrestriced

option 2:
- blend as many frames as needed together when the user defines a framerate downconversion.
- make a slow travelling blend weigth movement with extreme framerate upconversions. (so one can simulate slowmotion with blending)

[mg262]

krieger2005,

There was a typo in the warp code (copy/paste bug as ever... h in place of v) that caused a slight inaccuracy in the warping; luckily for me, you found a case where this caused it to access off-screen. Thank you very much!

I replaced the DLL...
Motion, 14 November 2005
Documentation

Edit: I'm not sure the server has the new file... give it a few minutes
... it should be there now.

And, please, anyone using warping, download this version. I added more safety checks to the warping which may slow it down somewhat... but they are needed in some unusual situations. Please tell me if the slowdown is large.

marcellus,

I'm glad it's working well! It is only because I have incisive feedback from yourself and others that I can improve things. So, if you spot other reasons/circumstances where it doesn't work well, please tell me!

By the way, I think it is possible (but fiddly) to import Deshaker into AVISynth if needed; at least, I saw a thread on this once.

scharfis_brain,

Thanks for the suggestion. I will certainly put it in, when I have thought about it a little and thought of sensible parameter names (I dislike mode = 1, mode = 2, etc). Edit: what about a variable "aperture" for motion blur?

Grab Motion, 14 November 2005 (revised) and try BlendFPS(float targetframerate[, float "aperture" = 1]). Increasing aperture will blend more frames; decreasing aperture will blend fewer.

[krieger2005]

Okey,
i tested the new version (not the c-version) and the old from 10Nov. Here my speed-results for 2 clips with 1000 frames:

#14Nov05:
#ca. 37-42 fps, 26 sec / ca. 37-40 fps, 26 sec
#10Nov05:
#ca 37-40 fps, 25 sec / ca. 36-40 fps, 26 sec

I have a P4 3000MHZ, HP enabled. The new version of the motion-filter does now not quit Vdub. thanks.

I would say, that if there is a speed drop, than not a big one, since both times are equal for a 1000 frame-clip, so i guess for a larger clip it would be the same...

But still VDub close. But this is one thing of my avs-scripts... they are too complex. But maybe you can add something in, so that your filter stop to allocate memory when a maximum is arrived because i had situations, where vdub take 2GB (swap-file).

BTW. I think also it would be great to use a global motion/MV-Format... Fizicks' MVFlow and your Vector-Analyzer would, i think, work great together.

Can you add (but i guess you have it) the scene-change-detection in your to-do-list?

great work.

[mg262]

Thank you for checking it.

maybe you can add something in, so that your filter stop to allocate memory when a maximum is arrived because i had situations, where vdub take 2GB (swap-file).

I'm not allocating memory at all; I'm relying on AVISynth to do it for me. There is a mechanism to provide hints to AVISynth about what information should be kept, and I will try that to see if it helps. Could you give me a script that causes large memory usage?

As far as I know, the MVFlow???? functions and my warp functions do exactly the same thing (bilinear mesh compensation)... so there's not much point in mixing and matching.

Scene detection: will do. Also see here.

[krieger2005]

Here is such a script:

f=4
#analyze1 = last.FFT3DFilter(sigma=6).BilinearResize(m4(width*f), m4(height*f))
analyze1 = last.BilinearResize(m4(width*f), m4(height*f))
analyze2 = last.BilinearResize(m4(width*f), m4(height*f))
motionVB = FindMotion(analyze1, reset=0, initialise=20, iterate=6, from=previous)
motionCB = SWC(analyze2,motionVB, source=previous, bilinear=true).LanczosResize(width,height)
motionVF = FindMotion(analyze1, reset=0, initialise=20, iterate=6, from=next)
motionCF = SWC(analyze2,motionVF, source=next, bilinear=true).LanczosResize(width,height)
interleave(motionCB,last,motionCF)
temporalsoften(1,5,5,10,2)
SelectEvery(3,1)

i know that fact that with such supersampling one must have much memory, but i thought that there would be an error-message, so i thought you allocate the memory internal.

As far as I know, the MVFlow???? functions and my warp functions do exactly the same thing (bilinear mesh compensation)... so there's not much point in mixing and matching.

I don't know anything... i'm a noob The results looks just a little bit different. Maybe because of different motion-informations given by two different Algorithms...

Your idea for scene-change-detection-extra-filter is, i think, good, since very much filters need such an information (as you written in that thread). Do you want make it based on motion-informations made by, for example, "FindMotion" or a simple "luma-difference"-thing? But... i will see it when you release it

[mg262]

Careful when supersampling... you are effectively making everything move much faster, and I don't know how well the filter will cope. At least, please check vs the non-supersampled version. Also, as it says in the docs, iterate isn't really meant for general use... you are welcome to try it, but be aware that I haven't checked its effectiveness yet.

I've read all of Fizick's thread, but not all the documentation updates... so I will go and read up when I have a bit of time.

Scene detection: The idea is to have a standard scene representation (a scene type) and many different scene-detection filters which all output that type. Certainly SAD (= luma-difference?) and SSD will be in there.

I am dragging my feet on this because once a representation is shared between many filters, it becomes much harder to change -- so I need to get close to the right answer first time.*

*The "much harder" is for the case where all the filters are mine... the alternative is much worse. This is precisely what you see in formats like AVI... they can't be changed because they are too widely used, so you get all sorts of hacks being put in place and the situation ends up in a total mess. Same goes for AVISynth 2.5. Simulating associative arrays, etc. helps a bit but is far from a real solution.

[Fizick]

Stupid question:
What is SSD ?

[Manao]

SSD: Sum of Square Differences
SAD: Sum of Absolute Differences