Motion compensation reached its limit?

[Terka]

With respect and thank you to all who spend a lot of time with development, MVtools and SVPflow dont seem to have bigger progress now.
Some filters on Cross corelation by V.C. Mohan and StainlessS appeared. Suggestions by Didee what to improve...
But temporal filtering needs more powerful tool.
Could someone please help Avisynth world move forward again?

[Mystery Keeper]

I've tried to ask the same question. Got no help here. Searched for different solutions. One thing I can say for sure is many modern methods use segmentation. But my math understanding is not good enough to comprehend how it works.

I might have made mistakes in my trials, but here's my experience:
I tried phase correlation. Works like wonder on global scale (whole frame), but I don't know how to make it work on smaller blocks properly. When I said I might make a stabilization plugin - I had phase correlation in mind.
I tried Gunnar Farneback’s algorithm from OpenCV. It sucked.
I tried Lucas-Kanade method from OpenCV. It worked well on bright frames and failed completely on dark ones.

This work looks like a possible solution. Right now I feel like I need to take a break from MC research and focus on more pressing matters. Would be grateful if someone else looked into it.

Another thing I can say about motion estimation is that much research on it was done by Microsoft. Probably related to Kinect. But I can't imagine the outcome if someone bugs them about it.

[Adub]

A large amount of motion compensation algorithms have been evaluated by the x264 devs, and they chose to implement the algorithms that actually work consistently and well.

Now, a plugin that exposed x264's motion compensation algorithms certainly seems like a worthy endeavor, especially considering the code quality and optimizations that the devs have put into their MC code.

I currently don't have the time nor understanding to implement such a plugin, and my efforts have been focused on GPU support in Vapoursynth.

[Mystery Keeper]

When I think about how our brain perceives and handles motion information, I realize we do it based on our knowledge of objects. Looking at two frames we almost always can say exactly what moved and how. Does it make AI the best estimator? How can such AI be implemented and trained?

Also, I wonder if MVTools would do better with segmentation.

[wonkey_monkey]

Now, a plugin that exposed x264's motion compensation algorithms certainly seems like a worthy endeavor

Isn't that an almost entirely different thing from interpolating frames, though? When compressing, you've already got your target frame to aspire to, so when mocomp "fails" - there's no decent match - you can fallover to encoding a new macroblock, or whatever it is x264 really does. It also doesn't matter if x264 makes an incorrect estimation of motion for any particular block, because it will be the match that most closely reconstructs the target pixels, not necessarily the match that most closely follows the real-life motion.

David

[mandarinka]

For filtering purposes, you want to catch the "true" motion - whereas an encoder goes for "cheapest to code" motion vectors. That means that it can get things quite wrong. So just porting over x264 methods doesn't help you unless you adapt them a lot.

[Mystery Keeper]

Another option to try if someone can reverse engineer Mathlab code.
http://ps.is.tue.mpg.de/theme/Scenes%20in%20Motion

[Terka]

It looks like there is also C code.