How to warp static image with vectors from video?

[Mike3]

Bellow there is code from doom9 forum many years ago .
How should the script be modified so that:
- after analyzing the first and second frames of the avi video and the static jpg is distorted with just that small amount of motion the result is saved as let's say 1_dist0001.jpg
and
- for the next set of motion vectors resulted from analyzing the second and third frames of the avi video is applyied to image 1_dist0001.jpg
and the result is saved as 1_dist0002.jpg
... and so on until the last two frames of the video and their motion vectors



LoadPlugin("mvtools2.dll")
o_clip = AVISource("1.avi")
o_clip =o_clip.ConvertToYV12
image = ImageSource("1.jpg", end = 100, fps = 30)
image = image.ConvertToYV12
super_clip = MSuper(o_clip, hpad = 0, vpad = 0, pel = 1)
mv_clip = MAnalyse(super_clip, blksize = 4, truemotion = true, overlap = 2)
super_image = MSuper(image, hpad = 0, vpad = 0, pel = 1)
result_clip = MCompensate(image, super_image, mv_clip, recursion = 440)
return result_clip

[wonkey_monkey]

Depending on what kind of source video you have, I have a plugin in the works which may do exactly what you want (and, if the video is a suitable one, give a much better result than MVTools, because you'll suffer from both compounding rounding errors and image degradation):

https://i.imgur.com/uHtG7Rw.gif

Can you share the video and image?

[Mike3]

Hello Wonkey_monkey

That plugin you mentioned would be very useful.

You actually pointed out the drawbacks with the current form of the script.
I am looking for a way to improve the script to overcome the drawbacks you mentioned
That is why I want to save the resulted image after each set
of motion vectors and distortion and apply the next slight distortion on the previously saved result.

I am looking forward to see the plugin you are making.
And also to ways that people here find the script could be modified to work around the rounding errors.

[wonkey_monkey]

The plugin isn't suitable for every video, so if you can share it I can give you an idea of how well it might work.

[Mike3]

I uploaded them here.
That small tracking gif made with your plugin isn't it :)?
I didn't notice at first what it is.
https://files.fm/u/58c7asph

[wonkey_monkey]

The video may be too dark and grainy, but I'll try it out later. I'll also have to find a way to break it into sections as my tracking doesn't handle points which go out of frame.

[Mike3]

1.a https://youtu.be/W_jo1a-DTtE?t=189
1.b https://youtu.be/vJlUhjSaqp4?t=681
2 https://www.youtube.com/watch?v=AAn_yKEFRj0
These here are the most "working around errors" tech I know.
Well I think the Avisynth Tracking gif from you looks very nice.
From what you tell me I could try a lot of types of preprocessing to obtain motion vectors.
If you intend to make the plugin comercial I understand .
If not I would try a lot of things and return to you the best workflow I find for low feature video.
I am thinking about hard "hdr" type compression , multi size detail enhancement.
That gif looks good and there definetly is something good with your software and I like finding ways to get best results.

[wonkey_monkey]

As I thought the video is too dark and smeary for feature tracking.

[poisondeathray]

I am thinking about hard "hdr" type compression , multi size detail enhancement.

Can you explain in more detail what you want / what context this is for ?

Is this in the context of motion tracking and compositing ? Or are you trying to use motion vectors for another reason? The thread title is misleading

There are other free /open source applications that have tracking ability e.g. blender, natron , blackmagic fusion
(and commercial ones of course)


These here are the most "working around errors" tech I know.

Those are research, not production ready...



If not I would try a lot of things and return to you the best workflow I find for low feature video.


Tracking is more difficult when the image distorts (eg. cmos rolling shutter "jello") , motion blur, lens distortions - because the things your trying to track effectively change shape - it no longer "registers" as the same feature area, or point, or whatever you're method you're using to track. Or compression artifacts can interfere and knock your trackers off. The lower the quality, the more source problems => the more difficult to track accurately

Often you can use techniques to optimize for pre-tracking.eg. If it's too dark, brighten it. If it's too noisy, denoise it etc... You can track a filtered video and apply the results to a different one. Different types of motion trackers have various pros/cons and situations that they are suited for.

Your sample has a reference ground plane that is mostly a "flat" 2D plane. wonkey_monkey's example is mostly 2 flat planes too. A specific type of tracker exists called a "planar tracker." It is specially designed to track...you guessed it ... 2D planes. Mocha is the most famous for this, but blender has a planar tracker too, as do other tools

Here mocha was used to track the ground, and the motion data was linked to colored patches . You could replace it with anything, like a high res ground texture etc... there is a slight bit of wobble, but given the condition of the source footage, it's probably ok and sticks well enough. It could be fixed with minor adjustments if you needed to, and the pro version has track adjust features

https://www.mediafire.com/file/o9td70c9n59lc6j/mocha_ground_track_15fps.mp4/file


But compare that situation to a moving expressive face with contours. Or a tree with waving branches and leaves. If you're trying to enhance textures on a bumpy rock with irregular shape, or project a gaping wound or place cyborg implants on a face in motion, and while camera moves - those are not simple flat planes . Those situations are more complex and require other techniques. So your 2d example is not as technically challenging as those research presentations, or other scenarios

In practice, the way it's done today - is with 3d match moving, and 3d projection mapping . Textures are projected onto "proxy" 3d geometry . Any modern Hollywood movie or high budget TV series with special FX in the last 10-15 years uses this when you need to "stick" something onto something else . But most software that does this is specialized and expensive

For a typical video - it will not be possible to do this with some avisynth script alone, automatically. It requires some user intervention to define areas to track and set parameters. Mvtools or avisynth won't be accurate enough for this type of work - it's the wrong tool for the job. You need it to be very accurate if you wanted it to "stick" correctly, or if you wanted to enhance resolution . Otherwise textures will "fall off". But I'm seriously interested in wonkey_monkey's plugin . If you need a tester , I'll take it for a spin

[Mike3]

Hi Poisondeathray
The hdr type compression and multi size detail enhancing are mentioned as way I thought about as good ways to prepare the video for motion vector extraction.

In my opinion there also is a way to perfectly prepare the video for Mvtools motion vectors extraction but I did't find it yet.

I also think that even a 160 by 120 resolution version of the video would be enough to extract perfect motion vectors that would translate well for sticking 2d sprites on the full resolution video.

If the human eyes and brain can track it there must be a reasonable way to track every pixel without artificial neural networks. There must be something slipping the bright mind of programmers. I have this strong feeling.
There is a simpler solution noone thought about yet.

[Mike3]

I return to my first post asking for scripting help about doing the following :
(in short how to modify the frame interpolation mvtools script below in order to incrementally apply the frame to frame motion vectors to a previously "interpolated" result)


How to warp static image with vectors from video?
Bellow there is code from doom9 forum many years ago .
How should the script be modified so that:
- after analyzing the first and second frames of the avi video and the static jpg is distorted with just that small amount of motion the result is saved as let's say 1_dist0001.jpg
and
- for the next set of motion vectors resulted from analyzing the second and third frames of the avi video is applyied to image 1_dist0001.jpg
and the result is saved as 1_dist0002.jpg
... and so on until the last two frames of the video and their motion vectors



LoadPlugin("mvtools2.dll")
o_clip = AVISource("1.avi")
o_clip =o_clip.ConvertToYV12
image = ImageSource("1.jpg", end = 100, fps = 30)
image = image.ConvertToYV12
super_clip = MSuper(o_clip, hpad = 0, vpad = 0, pel = 1)
mv_clip = MAnalyse(super_clip, blksize = 4, truemotion = true, overlap = 2)
super_image = MSuper(image, hpad = 0, vpad = 0, pel = 1)
result_clip = MCompensate(image, super_image, mv_clip, recursion = 440)
return result_clip