Motion Compensation

[Manao]

Fizick : Which compensation functions ?

In the class BlockData, there is still MotionCompensateBlock(). It copies the block pointed by the vector in the array given in argument.

But I don't use it in MVBlur() and in MVConvertFPS, because in these filters, I move the block which possess the vector along this vector.

Scharfi : I encountered your bug ( choppy motion with high fps conversion ). It's mainly because - for the moment -, I compensate with a pel precision only. It happens also because when I compute by how much I have to move the block, I round a float number with flooring instead of rounding. I'll change that, which will partly solve the problem.

Later, I'll be able to move a block along non integer vectors.

[scharfis_brain]

nice to hear, You've found it, too!

what happens on a Zoom? will your fiter 'only' move the blocks or will they be zoomed, too?

the same on rotations: will it move the blocks or are they rotated, too?

[Manao]

Blocks are not rotated nor zoomed. I only search the translation that minimizes the distorsion. Taking into account rotation and zoom would mean minimizing a function of 5 parameters instead of 2, which is by itself a lot more complex. Moreover, trying such a transformation implies computation far more complex.

That means of course that you'll see artifacts if there is a fast zoom or a fast rotation. But its unavoidable for the moment ( maybe not in 5 or 6 years ).

For filling uncompensated areas, I just tried another trick which performs interestingly ( better results but segfault ) : I want to interpolate a frame between frame n and frame n+1 :
- I compute vectors between frame n-1 and frame n, frame n and frame n+1, frame n+1 and frame n, and finally frame n+1 and frame n+2. Vectors from decond and third computation are used in priority. Vectors from first and fourth are inversed, areas filled by these compensation are added to the already filled areas. I also increase the size of the block moved ( making the added borders less prioritary than the core )
- Finally, if there is still some areas unfilled, they are filled with a blend of frame n and frame n+1. I'll try later to fill them spatially, it will be better.

[scharfis_brain]

Another Idea:
what about using a global motion compensation, like fizick depan for moving/rotating/zooming large uniform areas and afterwards compute the remaining diferential motion on this global compensated frame.

Could this improve precision on moving objects and global rotation and zooms (oh-oh zooms ARE global, hehe)?

is this implementable?

I hope, that I don't bother you with my thoughts.
I cannot estimate the programming work your doing. I assume it is much...

[Manao]

It would be possible, but I don't think if it would worth the amount of work required to implement it. It will not be me who would implement that algorithm, because there is still a lot of work on other parts of the motion engine, and on other filters ( MVDenoise, mainly, because a good denoising would be really great )

I hope, that I don't bother you with my thoughts.

Not at all, I asked for ideas and I'm glad you're exposing yours.

I cannot estimate the programming work your doing. I assume it is much...

I'm still a student, so I have plenty of time.

[Fizick]

I want to try to use your plugin similar as Scharfis spoke,
to make local motion compensation after global compensation (by my DePan plugin).

In previous versions, the ShowMotion function had option to make such local compensation.
But now it is removed. Can you restore it? It is very general function, anybody could use it for denoising with any temporal filter.

Yes, I found 2 internal functions in source code:
void BlockData::MotionCompensateBlock(unsigned char *block)
void BlockData::MotionCompensateBlock(unsigned char *block, int dpitch)
(what is correct ?) But right now I can not modify and compile your plugin.

[Manao]

Both were correct a long time ago. But since I changed a lot of things inbetween, the second one (( block, pitch )) is now useless.

I'll will put back the motion compensation part of MVShow.

There will be another release during the week, with updated MVBlur / MVinterpolate / MVConvertFPS / MVShow.

[Fizick]

Thanks for answer.
I hope, you will include both forward and backward compensation (option).
May I ask for make file for project?

[violao]

I managed to compile mvtools using VisualC++ 6.0 and nasm (GPL) and right now I'm trying to create a filter using motion vectors from SearchMVs (like in MVShow). The problems are comming mostly from occlusion areas, mainly uncovering regions around moving object and edges. It appears that SearchMVs produces 'spurious' motion vectors in and around these regions and all further actions are worthless since just by looking at vector we cannot know if it's spurious or not. Having those false vectors around we also cannot properly detect moving edges in order to provide for different treatment for occlusion areas. OTOH this should be possible if we had 'zero' vectors in uncovering regions or anything else that matches the movement of surrounding not-occlusion area. If we find a way to produce 'better' matches for those vectors, then most, if not all further filtering will be better.

Luckily, there seems to be a way to approximate those vectors and I would like to kindly ask Manao, the author, to consider implementing this as either an extension to SearchMVs, or even better a separate function for calculating motion vectors. This should be very similar to existing function, but would need to work on 3 frames: previous, current and next. While searching for vectors it should look in BOTH previous and next frames and choose whatever displacement in either previous or next frame delivers the lowest SAD. This way if we have uncovering region in previous->current frames it will (hopefully) become fully visible region in current->next frames so it's motion vectors will likely be similar to neigbouring not occluded area.

This should probably be sufficient *i think* for various filtering purposes, but even for interpolation this may eventually solve 'edge problems'. Since we should now have completely qualified (even not the best) vectors for the whole frame we might try to detect occlusion areas by finding horizontal and vertical moving edges. Then we may regionalize the frame and apply different interpolation strategies to different regions, for example we may try to interpolate in uncovering regions from next frame only, hopefully producing less artifacts in those regions.

[Manao]

New version : MVTools v0.9.2.

Changelog :

 * MVInterpolate doesn't exist anymore.
 * MVBlur and MVConvertFPS have been improved. They also have got new parameters, have a look at the documentation.
 * MVShow gets back its compensate mode ( MVShow(cm = true) )

@violao : I'm glad to hear you manage to compile my code, expecially with MSVC 6.0.

Your idea is indeed a good one, but you have to slightly extend it since a covering / uncovering happens between two frames, not exactly on a frame. Hence, you have to fetch motion vectors between frames n-1, n, n+1 and n+2.

But your idea amounts to make a postprocessing on motion vectors, and such processing must not occurs in SearchMV, but elsewhere. Moreover, with the source code I provided with the latest version, you can implement it quite easily. Look at the code in MVConvertFPS to see how to search for vectors between different couples of frames. Once the search is made, it's up to you to find and apply a good algorithm related to what you want to do with motion vectors.

I may write some utilitarian functions to ease that process, but, at the end, the process is too dependant of the use of the motion vectors, so there will not be a function 'MakeMotionVectorsGood'.

@all : you'll see that MVConvertFPS works better, especially on slow motion scenes ( closeup ). There should not be anymore slutering when trying to slow down a lot the video.

But, on high motion scenes, don't expect wonders. The concept of motion vectors shows clearly its limits in such scenes.

When scenechange is detected ( either real scenechange, or too much motion for the filter to cope with ) the frame which is built is not interpolated. Hence, on really high motion scene where everything is detected as scenechange, motion will slutter.

Covering / uncovering is slightly better interpolated, except when the covering object is the border of the frame ( that's the first place to search for artifacts ).

[scharfis_brain]

* MVInterpolate doesn't exist anymore.

why this?

I hoped you had made some improvements to it, too. But now you've removed it

mvinterpolate was very good for:
- repairing dropped & duped frames (it worked very cool)
- trying to build some motioncompensated deinterlacing (that was working too but with the underlying issue, i mentioned earlier)
- building a high configurable 60i to 24p conversion.
- and lot more per-frame repairing stuff

could you Pleeeeeeeze reinclude it?

a function that does custom motioncompensation (with your el &bl params) is really important, IMO....

[violao]

Originally posted by Manao
Your idea is indeed a good one, but you have to slightly extend it since a covering / uncovering happens between two frames, not exactly on a frame. Hence, you have to fetch motion vectors between frames n-1, n, n+1 and n+2.

I don't really get this. The idea of covering/uncovering is based on a premise that uncovered regions in (n-1, n) sequence will be fully visible in (n, n+1) sequence. Since they are already fully visible in the latter case, we may rely on vectors produced by (n, n+1) and I fail to see how can additional (n+1, n+2) vectors help in resolving original problem introduced in (n-1, n). Searching in (n-2, n-1) also wouldn't help since if something was occluded in n-1 it has probably been occluded in n-2 too.

But your idea amounts to make a postprocessing on motion vectors, and such processing must not occurs in SearchMV, but elsewhere.

Of course, but I was not talking about post-processing, but about the search itself. Post-processing cannot help solving original problem. Just by comparing 2, 3 or more vectors we cannot tell which one is 'false'. What I proposed was a modification of search scheme, just for this purpose, of course, not generally. The idea is that if we have covered->uncovered->visible sequence in 3 frames, then min SAD search will likely find better match for uncovered region in the next frame. That motion vector (n+1, n) is in no correlation with previous (n-1, n) so simple comparison or other operations with them are useless. Instead we need to use (n-1, n),(n, n+1) SAD as a criterion, not the final vectors from (n-1, n), (n, n+1) or whatever. At least I think so.

I'll check on your latest sources later. BTW, thanks for your great work.

[Manao]

violao : I grouped covering / uncovering together, one of which happened between frames n and n+1, that's why I spoke of frames n-1 to n+2.

And for implementing the algorithm outside SearchMV : don't forget that you have access to a lot of things once SearchMV has been call :
- GetFinalMV() gives you the motion vector
- GetFinalSAD() gives its SAD
- GetBestMV(int i) gives you the ith best motion vector found, SADwise. ( GetBestMV(0) <> GetFinalMV() if fth > 0 )
- GetBestSAD(int i)

So with all that, I think you can implement your algorithm.

scharfi : I thought it was useless with MVConvertFPS, but if you need it, I will make it come back.

@all : the last frames of MVConvertFPS are not returned. I'll correct it.

[violao]

Thanks. Somehow I missed the SAD part od blockdata

[Manao]

Alright, new version : MVTools 0.9.2.1

Changelog :

 * MVInterpolate makes its come back.
 * MVConvertFPS should work on the last few frames of the clip

violao : I'll try to make a good documentation on the API, with doxygen. It will be easier for you.

[scharfis_brain]

Many thanks!

I will test that out asap.

[Leak]

Originally posted by Manao
MVTools 0.9.2.1

Thanks for the new version - I just hope you won't add another number at the end with each one from now on, that could get a bit unwieldy...

When playing around with MVConvertFPS I've found that actually setting wf to "hat" produces an image only consisting of garbage blocks in red, yellow, green, blue, violet and cyan instead of the expected picture, which probably isn't what you intended...

I'm just curious why it seems to work if I don't specify wf at all, since "hat" is the default - but for generic functions not being able to specify "hat" even if it works as the default might be a bit problematic...

np: Autechre - C/Pach (Tri Repetae)

[Manao]

Well spotted, Leak. It seems that "hat" in fact never worked well. Hopefully, default was changed ( not intentionnally ) to "uniform", and I forgot to change documentation. Both will be updated ASAP, "hat" in order to work, documentation in order to report well default values.

So here is MVTools 0.9.2.2

[scharfis_brain]

manao: the default should stay on uniform blending. THis is the natural way.

look at moblurred Film or Video. you'll only see uniform motionblur. no hat or other ramps there.

[Manao]

It stays with "uniform" as default, don't worry. I may have made me misunderstood in the previous post ( "not intentionnally" was meaning not related to the bug ).