Apply motion data from a clip to something else - MVTools2

[Redsandro]

Hi,

I am pretty sure you've all seen this:
Some mpeg4 movie has a keyframe gone bad and you see something completely different distorting and moving by the motion estimation data from the animation that was supposed to start in that keyframe.

A few years ago, I think when codecs were more buggy, you could also see this effect right after fast forwarding.

I am looking for a way to deliberately do this. Move the contents of an image according to the motion detected in other footage.

I once saw a combination of filters for Avisynth that made objects look like they were growing or shrinking because of a motion vector filter reacting on luminance of the footage. That's probably the closest thing I saw in this direction, so my best bet would be that Avisynth is capable of something like that.

Can anyone point me in the right direction for realizing this, or tell me that it is not possible?

[videoFred]

I once saw a combination of filters for Avisynth that made objects look like they were growing or shrinking because of a motion vector filter reacting on luminance of the footage. That's probably the closest thing I saw in this direction, so my best bet would be that Avisynth is capable of something like that.

Averaging multiple frames. You need MVTools for this..

Fred.

[wonkey_monkey]

You might be able to splice non-keyframe data (say, from Xvid) onto to a different key frame to achieve the same effect. I thought I could do this in Virtualdub in Direct Stream Copy mode, but somehow it seems to encode an extra keyframe!

David

[Redsandro]

Hey that MVTools2.dll looks very promising!
The functionality definitely has it's origin in the same motion compensation tricks that mpeg does that would cause the stuff I mentioned in the OP.

I am going to toy around with it to see if it reaches the amount of precision I need. MFlow() looks very cool for that.

First question, why does the target(footage) (that is going to be distorted/compensated) require a superclip? It is just getting it's pixels moved.
So either I don't get it, or I am using this wrong. I read the documentation (http://avisynth.org.ru/mvtools/mvtools2.html#functions) but I don't get what superclip is good for.

To illustrate:

LoadPlugin("D:\Program Files (x86)\AviSynth 2.5\plugins\mvtools\mvtools2.dll")
source = ImageSource("motion%04d.png", start=0, end=100, fps=25, pixel_type="RGB24").ConvertToYUY2()
mSuper = MSuper(source, pel=2, rfilter=2)
mVector = MAnalyse(mSuper, blksize=8)

target = ImageSource("test.jpg", end=100, fps=25).ConvertToYUY2()
tSuper = MSuper(target, pel=2, rfilter=2) # why?

MCompensate(target, tSuper, mVector)


Footages are RGB but converted because I read that MVTools cannot use RGB footage. Sounds weird for someone working in RGB most of the time but it's probably very logical.

And how can I update the target so the next frame is the previous compensated frame?
MCompensate(target, tSuper, mVector)
Here every next frame has one frame worth of compensation applied, but I want the compensation to be cumulative. So for example target frame 10 has not only the compensation for 9-10, but for 0-10.
Right now when I substitute MFlow() it looks really bad, like there's a cheap water effect on my target footage. Also with blksize=4, search=5. But maybe I cannot see the movement clearly because every new compensation doesn't use the previous compensation as a basis like it should to get closer to the effect I want.

PS - There's a typo in the latest mvtools2.dll:
MCompensate : wring source or super frame size :)

-edit-

While googling for help, I also stumbled across this (http://spench.net/drupal/research/mv), it illustrates my OP. :)

[Redsandro]

Dammit they stole my idea but didn't bother to tell me how to do it.

Chairlift - Evident Utensil
http://www.youtube.com/watch?v=3pU9oAIFLcY

I am still looking for a way to do something like that without going into the decoder source code to mess up I-frames or something. I think it is possible with MVtools if I can calculate the integral from the motion vectors so that for example on frame 10 you have the vectors relative to frame 1 in stead of to the previous frame.

[Stephen R. Savage]

The difference between MCompensate and MFlow is that MCompensate moves blocks and MFlow moves individual pixels. That is what causes the wavy effect. MFlow is more for when you want to convert the framerate or something similar, as it produces a smoother result.

The target needs a super clip because the motion vector can have subpixel precision (e.g. MOVE {23 36} BY {30.5 20.25}). Basically, MSuper is a speed optimisation, because if it weren't there, each time you called MVTools, it would have to generate the supersampled data again.

I think what you want is the recursion parameter to MCompensate.

[Redsandro]

I think what you want is the recursion parameter to MCompensate.

Yes, yes that is it. Too bad it doesn't work for mflow. The mcompensate results are very blocky and noisy. I cannot fake a movement this way because after a few frames the target image looks like it was painted with the spraycan from the classic ms-paint. Remember that? ;)

[Leak]

Dammit they stole my idea but didn't bother to tell me how to do it.

Chairlift - Evident Utensil
http://www.youtube.com/watch?v=3pU9oAIFLcY
Can't watch this video (thanks, YouTube) but here's another one:

Triptych Part 3 (http://www.youtube.com/watch?v=S07Xm3JIhZU) by Blockhead...

[Redsandro]

Yes, good one, the effect in that one is a LOT better than what I manage to simulate using these simple lines:
LoadPlugin("D:\Program Files (x86)\AviSynth 2.5\plugins\mvtools\mvtools2.dll")

#Load sequence
source = ImageSource("PC03_%04d.png", start=0, end=80, fps=25, pixel_type="RGB24")
source = source.ConvertToYUY2().Reverse()

mSuper = MSuper(source, pel=1, rfilter=2) # pel=2 not supported for recursion
mVector = MAnalyse(mSuper, blksize=4, search=5)

target = ImageSource("test2.png", end=80, fps=25)
target = target.ConvertToYUY2()

tSuper = MSuper(target, pel=1, rfilter=2) # pel=2 not supported for recursion

# Debug
#MShow(mSuper, mVector)

#MVFlow()

# Motion Compensation output
MCompensate(target, tSuper, mVector, recursion=100)

Can anyone help with a more descent approach using AviSynth filters?