Motion Compensation

[scharfis_brain]

I have lots of problems with mvtools:

- v0.9.4. doesn't work at all.
it always claims : genericmotionfilter needs yv12 (at the line with mvinterpolate, mvblur, mvconvertfps or whatever), but the source is already yv12

- v0.9.2.x the nb parameter is useless, I do not see any difference.
the motionblur isn't getting better with higher nb
this means, on high motion, the motionblur is always ghosted instead of being one connected smoothed area..


btw. IMO it would give a MUCH better motionblur, if the image gets blurred directionally using 'simple' pixel mixing instead of block moving.
I mean, if you could just blur the pixels underlying a vector(field) into its direction, it should be THE ultimative mvblurrer...

[Manao]

Scharfi : can you provide the scripts you used ? I think you're messing up with the vector's stream.

I mean, if you could just blur the pixels underlying a vector(field) into its direction, it should be THE ultimative mvblurrer...

I don't think so. First, you would need to interpolate the vector's field for all pixels, which would be slow. Then, you would have to copy & move a pixel along its vector, which would mean interpolate its position for each time we copy it. Slow again. And finally, if the vector's field was diverging, you would get strange results ( blocks tend to lessen that issue, they don't solve it all )

[mg262]

Manao - thanks for the great tools.
_______________________

I'm trying to use the motion compensation to set up the context for a temporal blur as follows:

Get a frame order like this:

10, 8, 10, 9, 10, 10, 10, 11, 10, 12,
11, 9, 11, 10, 11, 11, 11, 12, 11, 13,
...
n, n-2, n, n-1, n, n, n, n+1, n, n+2

Then forward search + MVCompensate, and select the odd frames so you get

8, 9, 10, 11, 12 (all mocomped to match 10 as closely as possible)
9, 10, 11, 12, 13 (all mocomped to match 11 as closely as possible)
etc.

____________________________________

(Then you can apply any blur with radius two and throw away 4/5 of the frames.)

Unfortunately the results aren't great... looking at each run (e.g. 8,9,10,11,12), there doesn't seem to be less motion than in the original. The script is below and if anyone has any insights into what's going wrong I'd appreciate them...

(I have tried playing with the fth parameter. Also, the frame numbers coming out of the output are correct.) Have I understood forwards/backwards correctly? Flipping to backwards doesn't help...
________________________________________

o = last

Interleave( o, blankclip(o, 2) + o, o, blankclip(o, 1) + o, o, o, o, trim(1,0), o, trim(2,0))#, o, blankclip(o))

v = MVAnalyse(isb=false, fth=0 )
MVCompensate(last, v)

selectodd()

_____________________________________

By the way, I'm wasn't sure if this should be a new thread, but all the information on these filters seems to be here, so I've stuck with it...

Mohan

[sh0dan]

Wow - this is probably the best stuff I've seen released for ages.

It has a LOT of potential for framerate conversion, deinterlacing and denoising!

Wow! Great work!

@scharfi: You need to explicitly give the first video parameter, etc:

vec = mvanalyse(fth=30, isb=true)
mvshow(last, vec)

This is because the vector clip is optional, and therefore your vector information is assumed to be the clip you'd like to apply your motion stuff on.

A bit about the implementation (feel free to ignore)

- I really like the idea of having motion information as a clip.

- Forward backward considerations. It seems a bit strange you need to do both backward and forward prediction. Why not simply alsways do forward prediction, and request the previous (cached) motion frame?

- Binary format. A separate index file, and a motion vector file would probably be the nicest implementation. When frame 'n' is requested:

1) Check if present in index. If yes, read MV's from file.
2) Calculate MV's.
3) Append MV information to binary file.
4) Write index of frame 'n' to index file.

That way you only need to keep the index (two integers per frame) in memory, and you get random access.

[MfA]

Motion compensating using the per block forward motion vectors (in terms of the direction in time) from the last frame leaves gaps in the present frame if you try to reuse it ... which complicates things.

[sh0dan]

Oh, I see - it makes sense using current frame as reference to the previous.

I noticed a very slow, but also very good motion blur by using:

vec_back=mvanalyse(fth=30, isb=true)
vec_forw=mvanalyse(fth=30, isb=false)

MVConvertFPS(last, vec_back, vec_forw, fps=100, fbw=2, thSCD2=255, thSCD1=200)
temporalsoften(2, 255, 255, 0, 2)
selectevery(4,0)

Assuming 25fps material. Works very well for my CG material.

[scharfis_brain]

and again,a feature request:

I want to repair framedrops using this script: http://forum.doom9.org/showthread.php?s=&threadid=79771

but, mvinterpolate returns garbage when trying to interpolate the missing frame.

I am going this way:

for progressive Video:

normal Sequence:

A B C D E F G

framedrop at frame D

A B C C E F G

my function interpolates the 50:50 position between C & E, but somehow the C C E F sequence messes up mvinterpolate because C is a dupe.

the same with laced video:

normal Sequence:

Aa Bb Cc Cc Ee Ff Gg

here, I interpolate between c and E to get two frames with the weights
c*0.66+E*0.33 = D
and
c*0.33+E*0.66 = d

but like said before the dupes (or with interleced: forward-backwards stutter) makes mvinterpolate going messy.

would there be any solution for that?

maybe included in the plugin itself (dropped frames are almost indentcal, so detection is very easy!)

the function I posted in the thread works for some low motion video, but completely fails on high motion / short shutter speed video...

also burst framedrops (several framedrops in one sequence: A B C C C C G H) could be handled, too.

what do you think manao?

[Manao]

Detecting dup is fairly easy with the motion vectors and the sad. However, the rest of the processing would take a long time to code ( not complicated however ), and I don't have a much time as before ( internship, not a student anymore ).

I'm still working on the motion estimation ( I'm adding a padding of the reference frame, to better catch objects that come into the frame, as it is done in MPEG-4 ). Halfpel works again, it is faster than before ( fast enough to make useless the saving to file feature ), but no filter is using it yet.

[morsa]

Can this be thought in a more general way?
I mean, thinking about interpolating dropped frames, I got to the idea of interpolating missing frames from old films.
You know those portions of a film where 3, 5, 10, frames have been lost.In this case whe don't have a dupe frame, So I guess we should give the filter parameters like first frame, last frame and how many frames to be interpolated.
Is this posible?

At this moment we have:
DeScratch
Despot
Equlines
RemoveGrain
RemoveDirt

So what we lack is what Scharfis is asking (sorry Scharfis if I miss something)

[Manao]

morsa : it's already possible ( just not quite straightforward ) :

function MVInterpolateBetween(clip c, int "first", int "last", int "number")
{
  number = Default(number, last - first)
  beginbase = (first > 0 ) ? first - 1 : 0
  endbase = ( last < framecount(c) ) ? last + 1 : framecount(c)
  
  newfps = framerate(c) * number
  
  beginning = c.trim(0, first)
  ending = c.trim(last, 0)
  base = c.trim(beginbase, first) + c.trim(last, endbase)

  mvf = base.MVAnalyse(isb = false, lambda = 2000)
  mvb = base.MVAnalyse(isb = true, lambda = 2000)

  reconstructed = base.MVConvertFPS(mvb, mvf, fps = newfps)
 \.trim(number + 1, 2 * number - 1).AssumeFPS(framerate(c))

  return beginning + reconstructed + ending
}

Edited : first version was not working correctly.

[Dee]

to Manao: do you plan to release sources of most recent version (and future versions)?

[Wilbert]

What do you mean? The sources are included in 0.9.4.

[Manao]

I've released a testing version because E-Male was making a test between several temporal denoisers ( http://forum.doom9.org/showthread.php?s=&threadid=79706 ). But the code was being heavily modified at that time, so I decided not to release the source for that particular un'official' version. The next 'real' release will have the source code, of course.