Motion vector mask

[poptones]

I desperately need a filter that will produce a mask based on motion vectors. But I know nothing about producing them, and thus far haven't found an algorithm. Rather than redesign the wheel, can someone point me to either a) motion vector estimation code or b) a filter that can be readily modified to provide this information in a mask? I don't even care about the vectors themselves - I just need an ordinal mask. Surely some of you have experience with this sort of thing?

thanks heaps...

[sh0dan]

XviD contains block based motion estimation.

Using ME for filters (especial temporal) would be a great gain, and would probably help quality a lot. AFAIK the Dust filters uses ME.

[mf]

http://www.xvid.org if you live outside the US. If you live in the US, I think they have ME over at http://www.theora.org.


lol shodan, you beat me to it cause mozilla closed itself :p. Yeah, I've also been thinking about motion compensated 3d smoothers :). And when does anyone make a proper MoComp ? *hides* :D

[MfA]

The ME routines for xvid dont tend to produce true motion, probably true for Theora too, which is a problem if you use them for filters. The use of spatial correlation also makes the ME sometimes miss the best match in high motion scenes, not statistically relevant to codec quality ... but can cause some problems for filtering again.

It would be nice to have a ME routine as an avisynth filter which produced a dense "true motion" motion field. A reasonable approach for fast "true motion" motion estimation would be to use feature matching for an initial coarse motion field, can use a very large search area and does not get caught in local minima so has no problems with large motion, followed by a predictive zonal search with only a smoothness constraint to fill in the blanks (not a R-D constraint).

[poptones]

Actually, the ME part of ffdshow would work quite well. I say this only because the "lines" it draws on screen represent pretty much exactly the info I would like. The 8x8 part is no problem at all - in fact, it would be fine if a filter returned a decimated (1/8 size) bitmap as it would be trivially easy to then resize it using a bilinear transform.

The problem now with virtually every temporal filter I have looked at is they solve the problem pixel by pixel. For many problems (including the one I want to solve) that is not just overkill - it simply don't work.

[MfA]

If you just want an approximate MV magnitude for thresholding the inaccuracies dont matter much I guess, which I assume is what you want, but if you want to do motion compensated denoising/deinterlacing you want to know the true motion as accurately as possible.

[WarpEnterprises]

Hi again, poptones!

I don't get exactly what you need but another filter that does ME is Gunnar Thalins deshaker (for VD). Maybe this could be useful, if you didn't look at it already.

[sh0dan]

@poptones: There is no Motion Estimation in ffdshow. It is capable of showing the embedded motion vectors of an Xvid/DivX file. It does contain Motion Compensation in order to decode the MPEG4 streams.

@MfA: Yes - and the XviD ME is optimized to deliver the motion vectors that result in the best compression - not necessarily the most correct motion vectors. In fades for example you often see many long motion vectors, because this helps compression. Unfortunately there is very seldom any actual motion in the image.

For the best test of that, create a gradient - fade it down and up to black, and watch the large amount of motion vectors.
Two dissolving pieces of film often has the same effect - massive "false" motion vectors.

[mf]

Originally posted by sh0dan
@MfA: Yes - and the XviD ME is optimized to deliver the motion vectors that result in the best compression - not necessarily the most correct motion vectors. In fades for example you often see many long motion vectors, because this helps compression. Unfortunately there is very seldom any actual motion in the image.

For the best test of that, create a gradient - fade it down and up to black, and watch the large amount of motion vectors.
Two dissolving pieces of film often has the same effect - massive "false" motion vectors.
Which generates massive ME artifacts, which is why I've begun testing with actually tuning down XviD ME search effort in anime encodes. VHQ is all nice and reducing bitrate, but I really don't like the artifacts of image elements bouncing up and down when a wall pans. But I'm going offtopic now.

[MfA]

The bit based search should do ok on fades.

BTW how exactly does a wall pan? :) Do you mean when the camera pans past a wall?

[Acaila]

Does it have to be block based though? Because Wavelet based Motion Estimation is said to perform much better on fades, and should be faster in general as well.

[sh0dan]

@MfA: Could you describe "bit-based search"?

The only way I can think of to avoid "false" hits is to somehow "normalize" the sections before they are compared. Perhaps "autoleveling" a block and the area to which it has to be compared to could be a good idea. Then reapplying

Also some kind of "confidence" could be used for weighing the results - that could be achieved by a simple SAD (if SAD is Sum of Absolute Differences). That would of course make the possible denoising weaker, but also lessen the impact of mis-predictions.

A further prossible improvement could be to try to create a Global Motion Vector for the entire frame.
Often quite evenly colored surfaces are not set as motion, since the cost of a motion vector often outweighs what it actually helps. But trying to compensate in these areas could often help to preserve finer details, like wall grains, if the motion vectors from the rest of the image was also applied to this area.

But if the ME from XviD can be set to "find the optimal" motion vector (within a given space), that would be very interesting.

Another usage with ME is of course frame interpolation, where the motion vectors are used to create intermediate frames between two existing.

[sh0dan]

@Acaila: Do you have any links?

[Acaila]

Digital Image Processing by Kenneth Castleman

I read this book a few months ago, and he claimed that by performing the motion estimation in two parts could speed things up considerably. The first part was a wavelet phase correlation to find all the motion vectors (i.e. only their size, not direction) in a frame. Then once you have these you could use a standard block based search to find the direction of these vectors.
That was the general idea though, I don't remember the specfics exactly because my linear algebra isn't quite up to an adequate level yet.

Also this (http://biron.usc.edu/~sagetong/ee596proj_report.doc) document mentions a wavelet based algorithm that is also faster (although not always better) than the standard way.

Anyway I've just recently begun researching Wavelets so I can't give hard evidence yet. But the general idea I get from most papers is that wavelets are better on almost anything (i.e. noise reduction, edge detection, ME, etc.), although actual implementations are very scarce unfortunately.

[MfA]

Aceila, I think it very unlikely the correlation based technique used wavelets ... the standard way of doing this is via the FFT.

Wavelets are often used in multiscale searches, the problem is that the correlation across scales is less reliable than the simple spatio-temporal correlation between neighbouring MVs ... so it comes at a higher computational cost than predictive zonal searches, the search at the finest level has to be done anyway (full search is nearly never done in software codecs). The only advantage of the multiscale search is that for large fast moving objects it is less likely to miss the true motion (gradient descent search at the finest level might get stuck in a local optimum before finding the true MV). But it can still miss true motion for small fast moving objects.

Sh0dan : it simply computes the real number of bits it take to code texture and MV, and finds the MV with the minimum number of bits through gradient descent. The problem with level shifts is that they cause a high SAD, so SAD based searches are confused ... level shift has a far smaller effect on the number of texture bits though, so bit based search shouldnt be too much affected. It is slow though :)

Personally Im a fan of feature based searches, I should add that I have no experience to back this up though :) The motion of features is the only kind of motion which can be determined with a reasonable degree of certainty in isolation. The error for a given block on a untextured featureless piece of a surface will always be pretty uniform over a great range of MVs, so to determine the true motion the only good way of doing it is to assume motion smoothness and look at nearby points on the same moving surface which are featured (xvid ME can only accomplish this if the motion is truly totally uniform across the surface, and propogation of this true motion across the surface will only work if the motion is detected correctly at the edges in the first place and there are no skipped MBs in between).

To be honest multiscale searches also help a little here, because at the larger scale the block used for searching is more likely to have some features. That will make it more likely that an unambigious match for the global motion of a surface can be found, when propogated to the finer scales this will ensure a good match with true motion even if there is no error gradient to speak of for a gradient descent search to work with (this lack of gradient is what makes life difficult for xvid ME). Again though, this is fine for large objects ... but still doesnt help for small fast moving stuff.

Maybe a feature based search followed by a multiscale search to fill in the gaps would be a good idea.

[poptones]

Actually, the problem I have been trying to solve is pretty specific and much simpler than these solutions y'all are coming up with. (See attachment)

I think I have it solved - it's solved at least enough to eliminate the trouble shown in the example while still keeping all the detail of the picture. It's pretty simple, really:


smooth=clip.ReduceBy2()

masksmooth=smooth.MSmooth(threshold=2,strength=1,mask=true).ResetMask()
#make a mask using msmooth
masksmooth=masksmooth.layer(masksmooth.trim(1,0),"mul",255,0,0).levels(0,1,255,255,0)

#MUL=AND... remember?
#masksmooth now will blur any large temporal shifts
#apply mask to resized image, blur a little more for good measure, and yer done!

smooth=smooth.mask(masksmooth).Blur(1,1).BilinearResize(688,512)

#finally, apply it to the clip
return clip.layer(smooth,"add",192,0,0).levels(6,.97,255,3,250)


Because 0 x anything always = 0, any areas not present in both clips (ie areas that are moving quickly) will be removed from the mask. Thus, Butch's hands and sticks will get the blur treatment while the kit - and his face and anything else moving at a reasonable speed - keep pristine detail. This also leaves the "alligator skin" in areas not moving quickly, but that's not nearly so annoying as the large areas like shown above.

Here's a "short" (5MB - only about an HOUR upload for me!) sample of what I have so far. This is from an MPEG2 stream ripped from (I think) SKY in Germany.

http://www.maltman.home.dixie-net.com/palast-mofiltered-Q95.avi

Applying the filters above allowed me to go from a 640x480 xvid of Q=91 to a 688x512 xvid of Q=93 while still keeping roughly the same file size. With some tweaking on the transitions I think I can get it to Q=95 while still keeping the hour of full size video to two CDs.

[MfA]

Glad you found out a solution on your own, next time if you want usefull answers you might try asking the question differently though.

If you read up you will actually notice you never said what you wanted to do, so after your question was answered to the degree possible given the lack of detail we moved on to a different topic :)