Motion Compensation

[mf]

Manao has made the bold step of starting motion compensation filters. I would like to use this thread for two things, for users like me to present ideas, and for developers like Manao to explain the practical side, what is possible, how hard it is, etc. And of course, for developers to share ideas on the implementation of these filters.

This morning I have started to play with MVTools, and am currently encoding a supersampled (sorry, couldn't help myself :rolleyes:) motion blur, to see how it looks in realtime. I am impressed by the results so far.

[Wilbert]

Manao has made the bold step of starting motion compensation filters.
Small question. I don't have the time to look at it yet, but maybe someone of you can say something about it. Is it possible to do the following: hide someone's face with blocks, and keep it hided while he moves using this motion compensation stuff?

[Abond]

The problem was disscused before, maybe in the usage forum. Found it: http://forum.doom9.org/showthread.php?threadid=72895
If I remember the aim was different, but the approach...

[Mug Funky]

i'm attempting a motion-compensated deinterlacer a-la tomsmocomp.

not getting good results yet, but there's a glimmer of nice coming (pans are leaving no combing or stairstepping after just 4 mins of playing)

[edit]

argh! where's my brain? this is tough.

[scharfis_brain]

Do you mean this:

function mvdeint(clip x, float "moblur")
{
mbl=default(moblur,0.1)

x.separatefields()

evn0=selecteven().converttoyv12()
odd0=selectodd().converttoyv12()

# compensate the even field to the temporal position of the odd field
evn1a=evn0.mvinterpolate(nb = 5,bl = 0.5-mbl, el = 0.5+mbl, wf = "hat")
evn1b=evn0.duplicateframe(1).reverse().mvinterpolate(nb = 5,bl = 0.5-mbl, el = 0.5+mbl, wf = "hat").reverse()
evn1=overlay(evn1a,evn1b,opacity=0.5)

# compensate the odd field to the temporal position of the even field
odd1a=odd0.mvinterpolate(nb = 5,bl = 0.5-mbl, el = 0.5+mbl, wf = "hat")
odd1b=odd0.duplicateframe(1).reverse().mvinterpolate(nb = 5,bl = 0.5-mbl, el = 0.5+mbl, wf = "hat").reverse()
odd1=overlay(odd1a,odd1b,opacity=0.5)

# chain original and compensated field
evn2=interleave(evn1,evn0).trim(1,0)
odd2=interleave(odd1,odd0)

# reinterlace the strea,
interleave(evn2,odd2)
weave()
}

[Manao]

Wilbert : it can be done, but it's not very pratical with Avisynth ( no easy way to define where is the face at the beginning, since there is no GUI )

@all : I'm waiting for your ideas, mainly on how to treat uncovering of areas, how to use motion vectors to decide whether there is a scene change ( I have access to their length and the SAD assiociated with them ).

Right now, vectors have to point inside the frame ( which means, that a block that move toward the border gets a wrong motion vector ). I'll try to change that, but that's not an easy thing.

[scharfis_brain]

I'm waiting for your ideas, mainly on how to treat uncovering of areas

Ideas working out hidden areas.

Try to imagine a static background. Maybe a landscape.

now small object drives between camera and background from left to right

everyframe, the object moves from left to right, you have to replace the leaved space with the background (this would the way, our brain is working)

you could do that with a motionmask, that says: "this area was static a frame before / is static the next frame)"

now imagin the background is panning (direction irrelevant, for now)
a normal motionmask will fail, BUT a motioncompensated motionmask will work!

compensate, lets say 5 frames, to the same temporal postion, build a motionmask, and figure out the areas, gone static within those 5 frames. Those pseudostatic areas can then be used to fill the uncovered areas.

I hope that wasn't to irritating, trying to make the best out of my crappy english knowledge

[Mug Funky]

Scharfi:

that's very cool indeed. bit of a dud on scenechanges i'm afraid :)

i get 2 combed frames for 1 uncombed scenechange.

[scharfis_brain]

this mvdeint puts out the naked uncorrected motioncompensated video.
correcting that will require using masks. A lot of masks.

But I do not really like the output. even a whole frame isn't compensated correctly. there is almost stairstepping.
much more than with kerneldeint.

[Wilbert]

Wilbert : it can be done, but it's not very pratical with Avisynth ( no easy way to define where is the face at the beginning, since there is no GUI )
Can't you draw a circle in which the face is located (given that the user provides the location of the circle)? Or do you need the location of the edge of the face for this to work?

[Manao]

Stair stepping is to be expected : the motion during three consecutive frames isn't exactly constant, so the interpolation created from frame 1 and 3 won't be exactly aligned with frame 2.

The method should be the following :
- fieldseparate
- aligning fields ( moving the second field half a pixel up or down, if needed )
- compute motion
- motion compensating odd frames on even frames, and merging a spatial interpolation with the result of the motion compensation.

It can be done using only avisynth scripting, but I can't implement it today.

Wilbert : I do not need the edges, a circle may be enough. But it will have to fit rather well ( ratio area of the face / area of the disk close to 1 ), in order to follow closely the face ( it will even be able to discard blocks inside the circle that don't belong to the face :) ). I provide the source code with the filter, there is a class GenericMotionFilter from with all my filters inherit. You could code it yourself, if you can make my code to compile properly. Fetching the motion vectors is quite easy ( look at MVBlur, for example ), and then the algorithm would be to make a mean of the motion vectors inside the circle, and then decide that all blocks whose vector is close to the mean are inside the face.

You know where these blocks went since you have their motion vectors. So now, you have a set of blocks which should overlay the face on the next frame. You repeat the algorithm and it should work.

[scharfis_brain]

Oh, I think, scripting this within avisynth on my 500MHz Machine will be pain...
I do not have acces to the faster one next days :(

but: what about a motioncompensated 60i to 24p conversion?


function mvconvert60ito24p(clip x, int "mode")
{

mode = default(mode,2)
mbl=0.1

ya=x.mvinterpolate(nb = 4,bl = 0.5-mbl, el = 0.5+mbl, wf = "hat")
yb=x.duplicateframe(1).reverse().mvinterpolate(nb = 4,bl = 0.5-mbl, el = 0.5+mbl, wf = "hat").reverse()
y=overlay(Ya,Yb,opacity=0.5)
interleave(y,x)
mode0=selectevery(5,2)
mode1=overlay(selectevery(5,3),selectevery(5,2),opacity=0.5)
mode2=overlay(overlay(selectevery(5,1),selectevery(5,3),opacity=0.5),selectevery(5,2),opacity=0.3)
mode3=overlay(overlay(selectevery(5,0),selectevery(5,3),opacity=0.5),overlay(selectevery(5,1),selectevery(5,2),opacity=0.5),opacity=0.5)

(mode==0) ? mode0 : (mode==1) ? mode1 : (mode==2) ? mode2 : mode3

}

if the video has a shutter speed of 1/60 sec the modes result in following simulated 24p shutter speeds:

mode0 1/60 sec
mode1 1/40 sec
mode2 1/30 sec
mode3 1/24 sec

If 1/120 sec shutter has been used while shooting, you'll get those simulated shutter speeds:

mode0 1/120 sec
mode1 1/60 sec
mode2 1/40 sec
mode3 1/30 sec

the higher the mode (max 3), the lesser the mv-artifacts

have fun.

[DDogg]

scharfis_brain, the script above would require kernelbob(7), correct? Like -

avisource("G:\final.avi").converttoyv12(interlaced=true)
kernelbob(7)
mvconvert60ito24p()

/Add: Maybe you could start a thread in Avisynth user to discuss?

[scharfis_brain]

jep. exactly this.

[Manao]

Scharfi : concerning the process of uncovering areas, I would prefer if it wasn't involving more than two frames ( because searching for motion vectors is a rather slow process yet, and I would prefer if the filter was running at a few fps, not some mfps :) ).

Now, for the MVConvertFPS, I got an idea which works quite nicely. As you have notice, the interpolated frames have got quite a lot of artifacts, because the moving blocks don't pave well the frame. However, if I compute motion vectors on 8x8 blocks, nothing forbid me from moving 12x12 blocks, or even 16x16 one :). That trick performs very well, and allows some artifacts removal.

It now needs the SAD decision, as well as vectors able to point outside of the frame, and it will become usable.

Wilbert : I gave a try for face tracking. It doesn't work well on close up, because the face then doesn't have a uniform movement. I'll try it on other things, it may prove better.

@all : I'll try to make a good documentation on how to use the API I made for fetching the motion vectors. I don't know what you need in order to make a filter without having to compile my code : obj files, headers, and what else ?

[scharfis_brain]

Manao: what about a server/client structure?

a filter analyes the stream and gives back a data stream with all motionvectors and blocksizes needed to do a compensation.

the other filter will to the compensation then.

this would allow us, to write our own postprcessing routines using avisynth's masking...

I am thinking about a structure similar to fizicks depan().

[Manao]

Scharfi : whatever the method ( client / server or using the API ), you still have to write a filter in order to do use the motion vectors. Then, the postprocessing needs to know what the filter did, so it has to be included with the filter.

It will require for an outside programmer the same amount of work to use the API, or to fetch the stream of vectors. So for the moment, it will stay like this. Later, if I see that some filters could be chained without interfering, and that they both would use the motion vectors from the original clip, I will implement the client / server solution.

[Fizick]

Manao:
Can we use VC6 or free VCToolkit2003 for compiling?

[Manao]

I use the VCtoolkit2003 + Nasm. If you don't succeed, I'll put the .obj files online ( I don't know if it's enough, but I guess it is ).

[Fizick]

Why you remove Compensation function from new version?

[Manao]

Fizick : Which compensation functions ?

In the class BlockData, there is still MotionCompensateBlock(). It copies the block pointed by the vector in the array given in argument.

But I don't use it in MVBlur() and in MVConvertFPS, because in these filters, I move the block which possess the vector along this vector.

Scharfi : I encountered your bug ( choppy motion with high fps conversion ). It's mainly because - for the moment -, I compensate with a pel precision only. It happens also because when I compute by how much I have to move the block, I round a float number with flooring instead of rounding. I'll change that, which will partly solve the problem.

Later, I'll be able to move a block along non integer vectors.

[scharfis_brain]

nice to hear, You've found it, too!

what happens on a Zoom? will your fiter 'only' move the blocks or will they be zoomed, too?

the same on rotations: will it move the blocks or are they rotated, too?

[Manao]

Blocks are not rotated nor zoomed. I only search the translation that minimizes the distorsion. Taking into account rotation and zoom would mean minimizing a function of 5 parameters instead of 2, which is by itself a lot more complex. Moreover, trying such a transformation implies computation far more complex.

That means of course that you'll see artifacts if there is a fast zoom or a fast rotation. But its unavoidable for the moment ( maybe not in 5 or 6 years ).

For filling uncompensated areas, I just tried another trick which performs interestingly ( better results but segfault :) ) : I want to interpolate a frame between frame n and frame n+1 :
- I compute vectors between frame n-1 and frame n, frame n and frame n+1, frame n+1 and frame n, and finally frame n+1 and frame n+2. Vectors from decond and third computation are used in priority. Vectors from first and fourth are inversed, areas filled by these compensation are added to the already filled areas. I also increase the size of the block moved ( making the added borders less prioritary than the core )
- Finally, if there is still some areas unfilled, they are filled with a blend of frame n and frame n+1. I'll try later to fill them spatially, it will be better.

[scharfis_brain]

Another Idea:
what about using a global motion compensation, like fizick depan for moving/rotating/zooming large uniform areas and afterwards compute the remaining diferential motion on this global compensated frame.

Could this improve precision on moving objects and global rotation and zooms (oh-oh zooms ARE global, hehe)?

is this implementable?

I hope, that I don't bother you with my thoughts.
I cannot estimate the programming work your doing. I assume it is much...

[Manao]

It would be possible, but I don't think if it would worth the amount of work required to implement it. It will not be me who would implement that algorithm, because there is still a lot of work on other parts of the motion engine, and on other filters ( MVDenoise, mainly, because a good denoising would be really great )
I hope, that I don't bother you with my thoughts.Not at all, I asked for ideas and I'm glad you're exposing yours.I cannot estimate the programming work your doing. I assume it is much...I'm still a student, so I have plenty of time.

[Fizick]

I want to try to use your plugin similar as Scharfis spoke,
to make local motion compensation after global compensation (by my DePan plugin).

In previous versions, the ShowMotion function had option to make such local compensation.
But now it is removed. Can you restore it? It is very general function, anybody could use it for denoising with any temporal filter.

Yes, I found 2 internal functions in source code:
void BlockData::MotionCompensateBlock(unsigned char *block)
void BlockData::MotionCompensateBlock(unsigned char *block, int dpitch)
(what is correct ?) But right now I can not modify and compile your plugin.

[Manao]

Both were correct a long time ago. But since I changed a lot of things inbetween, the second one (( block, pitch )) is now useless.

I'll will put back the motion compensation part of MVShow.

There will be another release during the week, with updated MVBlur / MVinterpolate / MVConvertFPS / MVShow.

[Fizick]

Thanks for answer.
I hope, you will include both forward and backward compensation (option).
May I ask for make file for project?

[violao]

I managed to compile mvtools using VisualC++ 6.0 and nasm (GPL) and right now I'm trying to create a filter using motion vectors from SearchMVs (like in MVShow). The problems are comming mostly from occlusion areas, mainly uncovering regions around moving object and edges. It appears that SearchMVs produces 'spurious' motion vectors in and around these regions and all further actions are worthless since just by looking at vector we cannot know if it's spurious or not. Having those false vectors around we also cannot properly detect moving edges in order to provide for different treatment for occlusion areas. OTOH this should be possible if we had 'zero' vectors in uncovering regions or anything else that matches the movement of surrounding not-occlusion area. If we find a way to produce 'better' matches for those vectors, then most, if not all further filtering will be better.

Luckily, there seems to be a way to approximate those vectors and I would like to kindly ask Manao, the author, to consider implementing this as either an extension to SearchMVs, or even better a separate function for calculating motion vectors. This should be very similar to existing function, but would need to work on 3 frames: previous, current and next. While searching for vectors it should look in BOTH previous and next frames and choose whatever displacement in either previous or next frame delivers the lowest SAD. This way if we have uncovering region in previous->current frames it will (hopefully) become fully visible region in current->next frames so it's motion vectors will likely be similar to neigbouring not occluded area.

This should probably be sufficient *i think* for various filtering purposes, but even for interpolation this may eventually solve 'edge problems'. Since we should now have completely qualified (even not the best) vectors for the whole frame we might try to detect occlusion areas by finding horizontal and vertical moving edges. Then we may regionalize the frame and apply different interpolation strategies to different regions, for example we may try to interpolate in uncovering regions from next frame only, hopefully producing less artifacts in those regions.

[Manao]

New version : MVTools v0.9.2 (http://jourdan.madism.org/~manao/MVTools-v0.9.2.zip).

Changelog : * MVInterpolate doesn't exist anymore.
* MVBlur and MVConvertFPS have been improved. They also have got new parameters, have a look at the documentation.
* MVShow gets back its compensate mode ( MVShow(cm = true) )
@violao : I'm glad to hear you manage to compile my code, expecially with MSVC 6.0.

Your idea is indeed a good one, but you have to slightly extend it since a covering / uncovering happens between two frames, not exactly on a frame. Hence, you have to fetch motion vectors between frames n-1, n, n+1 and n+2.

But your idea amounts to make a postprocessing on motion vectors, and such processing must not occurs in SearchMV, but elsewhere. Moreover, with the source code I provided with the latest version, you can implement it quite easily. Look at the code in MVConvertFPS to see how to search for vectors between different couples of frames. Once the search is made, it's up to you to find and apply a good algorithm related to what you want to do with motion vectors.

I may write some utilitarian functions to ease that process, but, at the end, the process is too dependant of the use of the motion vectors, so there will not be a function 'MakeMotionVectorsGood'.

@all : you'll see that MVConvertFPS works better, especially on slow motion scenes ( closeup ). There should not be anymore slutering when trying to slow down a lot the video.

But, on high motion scenes, don't expect wonders. The concept of motion vectors shows clearly its limits in such scenes.

When scenechange is detected ( either real scenechange, or too much motion for the filter to cope with ) the frame which is built is not interpolated. Hence, on really high motion scene where everything is detected as scenechange, motion will slutter.

Covering / uncovering is slightly better interpolated, except when the covering object is the border of the frame ( that's the first place to search for artifacts ).

[scharfis_brain]

* MVInterpolate doesn't exist anymore.

why this?

I hoped you had made some improvements to it, too. But now you've removed it :(

mvinterpolate was very good for:
- repairing dropped & duped frames (it worked very cool)
- trying to build some motioncompensated deinterlacing (that was working too :) but with the underlying issue, i mentioned earlier)
- building a high configurable 60i to 24p conversion.
- and lot more per-frame repairing stuff

could you Pleeeeeeeze reinclude it?

a function that does custom motioncompensation (with your el &bl params) is really important, IMO....

[violao]

Originally posted by Manao
Your idea is indeed a good one, but you have to slightly extend it since a covering / uncovering happens between two frames, not exactly on a frame. Hence, you have to fetch motion vectors between frames n-1, n, n+1 and n+2.
I don't really get this. The idea of covering/uncovering is based on a premise that uncovered regions in (n-1, n) sequence will be fully visible in (n, n+1) sequence. Since they are already fully visible in the latter case, we may rely on vectors produced by (n, n+1) and I fail to see how can additional (n+1, n+2) vectors help in resolving original problem introduced in (n-1, n). Searching in (n-2, n-1) also wouldn't help since if something was occluded in n-1 it has probably been occluded in n-2 too.
But your idea amounts to make a postprocessing on motion vectors, and such processing must not occurs in SearchMV, but elsewhere.
Of course, but I was not talking about post-processing, but about the search itself. Post-processing cannot help solving original problem. Just by comparing 2, 3 or more vectors we cannot tell which one is 'false'. What I proposed was a modification of search scheme, just for this purpose, of course, not generally. The idea is that if we have covered->uncovered->visible sequence in 3 frames, then min SAD search will likely find better match for uncovered region in the next frame. That motion vector (n+1, n) is in no correlation with previous (n-1, n) so simple comparison or other operations with them are useless. Instead we need to use (n-1, n),(n, n+1) SAD as a criterion, not the final vectors from (n-1, n), (n, n+1) or whatever. At least I think so.

I'll check on your latest sources later. BTW, thanks for your great work.

[Manao]

violao : I grouped covering / uncovering together, one of which happened between frames n and n+1, that's why I spoke of frames n-1 to n+2.

And for implementing the algorithm outside SearchMV : don't forget that you have access to a lot of things once SearchMV has been call :
- GetFinalMV() gives you the motion vector
- GetFinalSAD() gives its SAD
- GetBestMV(int i) gives you the ith best motion vector found, SADwise. ( GetBestMV(0) <> GetFinalMV() if fth > 0 )
- GetBestSAD(int i)

So with all that, I think you can implement your algorithm.

scharfi : I thought it was useless with MVConvertFPS, but if you need it, I will make it come back.

@all : the last frames of MVConvertFPS are not returned. I'll correct it.

[violao]

Thanks. Somehow I missed the SAD part od blockdata :(

[Manao]

Alright, new version : MVTools 0.9.2.1 (http://jourdan.madism.org/~manao/MVTools-v0.9.2.1.zip)

Changelog : * MVInterpolate makes its come back.
* MVConvertFPS should work on the last few frames of the clip

violao : I'll try to make a good documentation on the API, with doxygen. It will be easier for you.

[scharfis_brain]

Many thanks!

I will test that out asap.

[Leak]

Originally posted by Manao
MVTools 0.9.2.1 (http://jourdan.madism.org/~manao/MVTools-v0.9.2.1.zip)

Thanks for the new version - I just hope you won't add another number at the end with each one from now on, that could get a bit unwieldy... ;)

When playing around with MVConvertFPS I've found that actually setting wf to "hat" produces an image only consisting of garbage blocks in red, yellow, green, blue, violet and cyan instead of the expected picture, which probably isn't what you intended... :)

I'm just curious why it seems to work if I don't specify wf at all, since "hat" is the default - but for generic functions not being able to specify "hat" even if it works as the default might be a bit problematic...

np: Autechre - C/Pach (Tri Repetae)

[Manao]

Well spotted, Leak. It seems that "hat" in fact never worked well. Hopefully, default was changed ( not intentionnally ) to "uniform", and I forgot to change documentation. Both will be updated ASAP, "hat" in order to work, documentation in order to report well default values.

So here is MVTools 0.9.2.2 (http://jourdan.madism.org/~manao/MVTools-v0.9.2.2.zip)

[scharfis_brain]

manao: the default should stay on uniform blending. THis is the natural way.

look at moblurred Film or Video. you'll only see uniform motionblur. no hat or other ramps there.

[Manao]

It stays with "uniform" as default, don't worry. I may have made me misunderstood in the previous post ( "not intentionnally" was meaning not related to the bug ).

[avih]

Manao, this is sick :D

as some of u might know, i much prefere 50fps playback over 25fps when it comes to video (i.e. live sport events, etc). i usually capture <something>x576 interlaced with ffdshow with some processing thrown in, and i don't re-process my caps as i mainly use it as pvr (playback with ffdshow, deinterlacing using a dscaler filter).

now, i tried taking one of my encoded clips (400x576), take one field only, and MVConvertFPS (0.9.2.2) to 50fps, and it's awesome :) i reduces the threshold a bit (quite a bit), and i truely get a 'video-like' expreience :) still few glitches here and there, but generally, it rocks.

i'm now thinking of using ffdshow to capture, with deinterlacing filter (encode 25fps 'progressive'), and then use MVConvertFPS during playback to record full res at 25fps and playback interpolated 50fps. might save quite some bits during encoding i think.

although so far, MVConvertFPS doesn't play good in zoomplayer (both direct avs and through ffdshow with ebmedded avisynth script). vdub plays it well. i don't know what's the cause yet. i'll try to update the thread if i have new conclusions.

cheers Manao, great work :)

[violao]

While I was playing with motion vectors calculated by SearchMVs I noticed the following problem that sometime makes vectors found in a certain area useless. Suppose you have an object moving in front and near the edge of a very dark static background area. Now suppose that beside that edge there is another static area with the brightness level similar to that of the moving object and that the distance between the object and this similar area is less than displacement of the object between n and n-1 frames. It appears that the vector search algorithm finds this closer background area more similar to the original object than the object itself in n-1 frame so the resulting vectors point from surrounding background to the object, instead from object in n-1 to object in n. Is it possible to do something about it? Perhaps increasing vector search space?

Another question, how is it possible to fetch vectors from higher levels? I'm familiar now to using BlockData class after SearchMVs, but I fail to see how to get field of blocks for higher levels. Manao?

EDIT: Fetching higher level vectors solved (I think).

[Manao]

First, the easy question :
Another question, how is it possible to fetch vectors from higher levels? I'm familiar now to using BlockData class after SearchMVs, but I fail to see how to get field of blocks for higher levels.You access them by using a higher PlaneOfBlock. If GOP is a GroupOfPlane, GOP[x] return the xth PlaneOfBlock, counting from the lowest, and GOP[x][i] then return ith BlockData of the xth PlaneOfBlock.

Now, for the other one. There are several reasons that may lead to such results, could you provide a screenshot, it would help determine whether it's a failure of the search algorithm, or something inherent to block matching.

For the principle of the search algorithm, there is no such thing as "space search". The multilevel analysis allows to fetch long vectors, while the use of a non exhaustive recursive search ( One Time Search, Diamond Search ) tries to find the local minimum around the best predictor ( either the one from the multilevel analysis, or the motion vector from surrounding blocks ).

It you want an exhaustive search, in PlaneOfBlocks::SearchMVs(), replace blocks[i]->SearchMV(predictedMotionVectors[i],1,1,ONETIME, CURRENT);byblocks[i]->SearchMV(predictedMotionVectors[i],1,radius,EXHAUSTIVE, CURRENT);Where radius is the half-width of the search square. Also, set 'fth' to zero. But it will be very slow then. You'll then be able to know if the 'wrong' vector you're getting is due to the search algorithm or to the block matching itself.

avih : thank you for the kind words. I don't know what's wrong with ZoomPlayer, but if I had to make a guess, I would say it's the DirectShow interface.

[violao]

Originally posted by Manao
It you want an exhaustive search, in PlaneOfBlocks::SearchMVs(), replace by...Where radius is the half-width of the search square. Also, set 'fth' to zero. But it will be very slow then. You'll then be able to know if the 'wrong' vector you're getting is due to the search algorithm or to the block matching itself.[/B]
Thanks. I'll try various search algorithms and will let you know what happens. Radius is given in block units, I suppose?

[Manao]

No, it's given in pixel unit. The search is made for all the vectors whose coordinates don't differ from the best predictor by more than 'radius'.

But be warned, a radius as small as 5 will imply 121 SAD computations by blocks, so it will be slow.

[violao]

Originally posted by Manao
But be warned, a radius as small as 5 will imply 121 SAD computations by blocks, so it will be slow. [/B]
OK, I tried with radius up to 16 with no success. Other search methods give similar results. I suppose this clip I'm playing with violates basic assumption that is 'intensity consistency hypotesis'. It seems that whenever an object moves between different illumination conditions (various shades and similar) there are some funny vectors around. Another complication seems to be the bluring of faster moving objects. Bluring is itself a blending of an moving object with static background, that results in changing both objects shape and it's luminance.

Therefore all I need/can do is to supress the filter processing in the area of 'invalid' vectors. Only problem is - how to tell what vectors are invalid?

[Manao]

Only problem is - how to tell what vectors are invalid?If that answer was simple...

I have implemented two measurement of validity for a vector : the SAD, and what I call 'DifferenceFromNeighbours', which is basically by how much the motion vector is different from motion vectors of the surrounding blocks.

So you have to use both of them, but it won't be easy.

[violao]

Originally posted by Manao
'DifferenceFromNeighbours'...
"Sum of quadratic differences between the motion vector of the block and all its surrounding blocks' motion vectors."

8 surrounding blocks? Is this a difference between MV length?

[Manao]

No, not exactly. If the value is high, it means that the motion vector is a singularity ( it doesn't belong here ), because it isn't homogenous with its neighbours. If it was a mere difference of MVLengths, vectors could be very different and the value still be low.

To compute it, you compute the difference between the motion vector and one of its neighbours ( hence you obtain a vector ), you consider its length, and you sum the square of the lengths of difference obtained for all the surrounding blocks.

For the moment, I detect scenechanges using only that value, and it works ( not perfectly, I will later use a combination of SAD and this value ).

[violao]

Originally posted by Manao
To compute it, you compute the difference between the motion vector and one of its neighbours ( hence you obtain a vector ), you consider its length, and you sum the square of the lengths of difference obtained for all the surrounding blocks.
Are you assuming here that all neighbours are 'valid'? Wouldn't a 'distance from median' be more appropriate here? It would protect from possible outlier neighbours.