Doom9's Forum - View Single Post

pbristow · 10th May 2011, 00:03

I second what Dark Shakiri says. By creating an AVIsynth plugin, the result can be used by several different applications (including VirtualDub, of course), and be more powerfully combined with other tools and techniques.

Myself, I'd love to see a new approach to motion estimation & interpolation, based on 3d object recognition and tracking. At present, for motion estimation we have:
- In Virtualdub, the free MSU_Motion_Estimation.vdf plugin, thanks to you guys.
- In AviSynth, MSU_FRC - I don't know what method or algorithm this is based on, possibly 'cos you guys are keeping it secret for commercial exploitation...?

(Or maybe it's 'cos I'm too lazy to try to read the source code. [EDIT: No, I was right first time. There's no source code published for this filter.])
- In AViSynth, the excellent MVTools2, which uses a 2d block-based algorithm. It's very good for most simple types of motion, which account for maybe 90% or 95% of what happens in real video, but there are cases it can't handle, e.g.: objects that are rotating or tumbling so quickly that, although their *topology* remains near enough the same from frame to frame, their shape as seen in 2d doesn't; Also layered motion, for example where fog or smoke or a reflection on a window is drifting one way, in front of an object or person that moves the other way (or remains still).
- Several lesser tools, mostly to do with simple de-shaking/de-panning.

I can see two possible ways forward for motion estimation from MVtools2:

1. Stick with 2d analysis, but move from detecting and matching similar blocks, to detecting clusters of features that move together as an "object". This means being able to take into account that a set of features may have been rotated or skewed between frames, as well as translated across or up/down the frame.

2. Switch to using 3d object recognition and tracking. This brings in the areas of study you refer to above.

The result of the tool needs to be not just a pre-determined action, such as frame-rate conversion, but rather a video clip (or a logfile, but a video clip is better, especially if you're working AVisynth) containing the encoded motion data IN A PUBLICLY DOCUMENTED FORMAT, so that other developers/researchers can then develop new applications (or alternative motion detection tools) using the same format. (This is the approach MVtools2 takes: Although it provides a complete toolkit for motion interpolation and frame rate conversion, it also allows others - by examining the code - to develop their own tools using the same data.)

10th May 2011, 00:03	#4 \| Link
pbristow Registered User Join Date: Jun 2009 Location: UK Posts: 263	I second what Dark Shakiri says. By creating an AVIsynth plugin, the result can be used by several different applications (including VirtualDub, of course), and be more powerfully combined with other tools and techniques. Myself, I'd love to see a new approach to motion estimation & interpolation, based on 3d object recognition and tracking. At present, for motion estimation we have: - In Virtualdub, the free MSU_Motion_Estimation.vdf plugin, thanks to you guys. - In AviSynth, MSU_FRC - I don't know what method or algorithm this is based on, possibly 'cos you guys are keeping it secret for commercial exploitation...? (Or maybe it's 'cos I'm too lazy to try to read the source code. [EDIT: No, I was right first time. There's no source code published for this filter.]) - In AViSynth, the excellent MVTools2, which uses a 2d block-based algorithm. It's very good for most simple types of motion, which account for maybe 90% or 95% of what happens in real video, but there are cases it can't handle, e.g.: objects that are rotating or tumbling so quickly that, although their topology remains near enough the same from frame to frame, their shape as seen in 2d doesn't; Also layered motion, for example where fog or smoke or a reflection on a window is drifting one way, in front of an object or person that moves the other way (or remains still). - Several lesser tools, mostly to do with simple de-shaking/de-panning. I can see two possible ways forward for motion estimation from MVtools2: 1. Stick with 2d analysis, but move from detecting and matching similar blocks, to detecting clusters of features that move together as an "object". This means being able to take into account that a set of features may have been rotated or skewed between frames, as well as translated across or up/down the frame. 2. Switch to using 3d object recognition and tracking. This brings in the areas of study you refer to above. The result of the tool needs to be not just a pre-determined action, such as frame-rate conversion, but rather a video clip (or a logfile, but a video clip is better, especially if you're working AVisynth) containing the encoded motion data IN A PUBLICLY DOCUMENTED FORMAT, so that other developers/researchers can then develop new applications (or alternative motion detection tools) using the same format. (This is the approach MVtools2 takes: Although it provides a complete toolkit for motion interpolation and frame rate conversion, it also allows others - by examining the code - to develop their own tools using the same data.) Last edited by pbristow; 10th May 2011 at 00:16.