MVClean - new filter based on MVTools

[violao]

After some playing around with mvtools I have finally produced what seems to be the working version of dirt cleaner based on motion vectors from mvtools. It can detect and remove most of the bigger very disturbing blotches, but it's somewhat less effective with smaller dirt and fine scratches. It's able to detect distrbances that are present in single frame only.

Detection is performed first by SAD for moved blocks and then by optional rank order median (ROM) detector. It is possible to exclude faster moving pixels from detection (to avoid false detections).

Replacement is performed by median filter that takes ROM found median, motion compansated previous and next frames as input. Filtering is performed only on 'dirty' areas of the frame, so all details on clean parts are preserved.

Since we can't trust motion vectors found for pixles that are covered by dirt, MC is perfomed using estimated motion vectors for those areas. Currently I'm calculating average for 100 surrounding 'healthy' pixels. This is the area for improvement. First things that come to mind are image segmentation and watershed algorithm. This is my current priority. Also better motion vector field smoothing and interpolation is planned.

Nevertheless this filter seems to work on my dirty test samples and I would like to encourage anyone that would like to see some improvement in this area to try it with their clips and let me know about possible problems and their ideas.

Download (http://www.bossanovaguitar.com/video/)

[kassandro]

I made a first test of your interesting filter. Of course I wanted the filter to clean a moving object. It produced severe artifacts. You can download the three frame clip here (http://home.pages.at/kassandro/test2.avi). It is compressed with lossless jpeg (unfortunately avisource fails with uncompressed avi). You need ffdshow to decode it. I have used mvclean with the default values. It is also quite slow. AvsTimer reported 1 fps on my 1.3 GHZ Celeron. That means a frame rate between 0.5 and 1.5. I will run much more extensive tests next weekend.
In my last contribution to the RemoveDirt thread I was more optimistic about motion compensated cleaning than I am now. My main objection is that if neighbouring blocks have different motion vectors, which is quite normal, artifacts are already quite likely. Thus I now think that only global motion compensation, where all blocks have the same motion vector, is feasible.

[Fizick]

Wow! My concurrent was born! :D

I will try your filter. But do you know about mine? ;)

I try to use my DeSpot with the compensated clips given by external MVTOOLS compensate function (about 4 fps on my AthlonXP2400).
It is also produce some artifactes near spots (blotch) positions.
For good motion vectors estimation we must have clean video, and for good cleaning of video we must have good motion vectors. :(

But in professional restoration systems (AURORA, FRAME, etc) these problem is solved (partially).
By the way, I use Despot (and sometimes Removedirt) with global motion compensation (by Depan) with good results.
We must search and try. May be, combinatation with global compensation.

[scharfis_brain]

what about overriding motion vectors?

example: when linear motion is present, the length of the individual vectors in a field of vectors has always the same length each frame (don't care about border effects on that example)

now, an error occurs at frame D, meaning, the vectors CD and ED are not clean due to the spot/dirt/error in that frame.
but because motion is (should be) not that different in Frames A B C D E F, it could be possible to dischard vectors (vector fields, maybe the field of vectors has to be moved, too) of CD & ED and regenerate them using the surrounding Frames.

is something like that possible, or was that one beer to much this evening?

[kassandro]

Originally posted by scharfis_brain
what about overriding motion vectors?

example: when linear motion is present, the length of the individual vectors in a field of vectors has always the same length each frame (don't care about border effects on that example)

now, an error occurs at frame D, meaning, the vectors CD and ED are not clean due to the spot/dirt/error in that frame.
but because motion is (should be) not that different in Frames A B C D E F, it could be possible to dischard vectors (vector fields, maybe the field of vectors has to be moved, too) of CD & ED and regenerate them using the surrounding Frames.

is something like that possible, or was that one beer to much this evening?

I think this is a very good conservative idea - and one has to be conservative here. It is easier, however, to do this spatially rather than temporally. The quality of the motion vector (usually measured with SAD) is more important than the motion vector itself. If a block has a bad quality motion vector and all the surrounding blocks have good quality motion which are all similar (but not necessarily similar to the block in the middle) then the block in the middle is very likely dirty and one should clean compensated with the motion vector of the surrounding blocks.

[violao]

Originally posted by kassandro
If a block has a bad quality motion vector and all the surrounding blocks have good quality motion which are all similar (but not necessarily similar to the block in the middle) then the block in the middle is very likely dirty and one should clean compensated with the motion vector of the surrounding blocks.
That is exactly what mvclean tries to do. The problem is - which surrounding blocks? One direction might be image segmentation to objects and then picking vectors from the same object that the dirty spot covers. However, what if dirty spot covers the object's border?

@Fizick - Catch 22 :confused: Of course I know your filters. In fact I had a problem with DeScratch. I can post short clip if you want. As for vectors, there is simply no way we can calculate good motion vectors from dirty spots. But we may try to interpolate with known good vectors, that's why I'm speaking about segmentation. Also I need to bring mvclean to 0.9.3 version of mvtools. We'll see what happens after this. For now I suggest supressing with sthresh.

Thank you all for trying to help.

[Fizick]

May be not DeScratch but DeSpot? Recently I released new version.

I tried MVClean shortly. It detects middle size spots well if "points"=1000 or above.
But cleaning artefactes is too visible.
I will try more later.

[violao]

Originally posted by Fizick
It detects middle size spots well if "points"=1000 or above.
But cleaning artefactes is too visible.
I will try more later.
Dou you mean "pixels"=1000? Be careful with that parameter. It's just a hack that will hopefully dissapear once I manage to find good segmentation solution. This parameter doesn't affect detection at all, it just tries to find good vector replacements for unreliable vectors in dirty spots. If you set it too high it is likely that you'll pick some vectors that don't belong to the object under the spot. Therefore 'wrong' vectors might be used in MC which will compromise cleaning stage. This could be the reason for cleaning artifacts you mention. Initially I used 100, but there is no real ground for this value. One article I found even suggest only 8 'good' vectors for interpolation. I proposed more since I expect that some of the blotch borders might still passed undetected and will bring their bad vectors into interpolation.

In my tests I had no problem with cleaning. The problems I found are mainly related to 'incorrect' MV search. Under certain circumstances, one of them being very fast motion, mvtools might fail and produce very 'wrong' vectors on clean spots, with bad SAD too, so mvclean is fooled to believe this should be cleaned. Wrong vectors produce wrong replacement decision, therefore producing artifacts. That's what I try to supress with sthresh.

Detection is controled primarily with mthresh (green area), then with t1, t2 and t3 (orange area).

As for Descratch/Despot, I found that Descratch detection works wery good. Is Despot also able to detect scratches? I know for certain that most of the scratches will never be detected with mvclean, due to spatio/temporal detection limitations. If the dirt is present in more than 1 frame, and that is mostly the case with scratches, then motion vectors will be 'good' so I cannot detect these. IMO only spatial approach is effective with scratches.

@kassandro - I cannot open your clip, neither with ffdshow, not with Lead Tools mjpeg. What's the catch?

[kassandro]

Originally posted by violao
In my tests I had no problem with cleaning. The problems I found are mainly related to 'incorrect' MV search. Under certain circumstances, one of them being very fast motion, mvtools might fail and produce very 'wrong' vectors on clean spots, with bad SAD too, so mvclean is fooled to believe this should be cleaned. Wrong vectors produce wrong replacement decision, therefore producing artifacts. That's what I try to supress with sthresh.

I think we all agree that you need high quality (low SAD) motion vectors near a dirt spot, which then should generate a bad quality motion vector (that's why dirt has such a negative impact on compression). On the other hand, I would guess that more than 95% of all high quality motion vectors are in fact zero motion vectors (i.e. the area is temporally static). Thus is it really worthwhile to go for this small cleaning gain with more artifact risk and a huge speed penalty. I have to say, though, that in my second test there were considerably less problems with artifacts and I agree with Fizick that mvclean does especially well with medium size dirt.


@kassandro - I cannot open your clip, neither with ffdshow, not with Lead Tools mjpeg. What's the catch?
If your ffdshow doesn't work, you should get the most recent one (http://athos.leffe.dnsalias.com/). Lossless jpeg has the advantage that unlike huffyuv it preserves yv12. I rather would have used an uncompressed three frame avi compressed with winrar, but avisource cannot load uncompressed avi.

[violao]

Originally posted by kassandro
If your ffdshow doesn't work, you should get the most recent
I've got it. Do you think it's possible that 2nd frame is blended or is it just blurred? In any case, this is very blurred moving object, rather than just moving object and in either case (blended or blurred) this just illustrates the problems with motion estimation that I talked about. It is highly probable that motion vectors for blurred objects, as well as for moving objects that change illumination conditions will be very wrong. Perhaps we should move to statistical approach, hoping to eliminate outlier vectors, but first object detection is needed.

The hair at the bottom of the frame appears in both frames 2 and 3 and is impossible to detect by motion vectors.

Nevertheless I think that some moving objects can be cleaned by this approach and I am not going to abandon this idea as long as there are some things that can be done to improve detection and as long as mvtools are being developed. In the final stage I believe a combination of 2 separate filters, one MC and the other spatial will be ideal combination to remove as much dirt as possible.

[kassandro]

Originally posted by violao
I've got it. Do you think it's possible that 2nd frame is blended or is it just blurred? In any case, this is very blurred moving object, rather than just moving object and in either case (blended or blurred) this just illustrates the problems with motion estimation that I talked about.

No there is no blending going on. The hand is thrown down so fastly - too fast for the film camera - that it looks very blurry. Such things are not unusual in old films, though I have to admitt that this a very extreme example. But some blurring of fast moving objects is normal.


The hair at the bottom of the frame appears in both frames 2 and 3 and is impossible to detect by motion vectors.

Yes, the hair is on the film projector and thus on all the three frames. On the first frame it is partially obscured by the hat.


Nevertheless I think that some moving objects can be cleaned by this approach and I am not going to abandon this idea as long as there are some things that can be done to improve detection and as long as mvtools are being developed. In the final stage I believe a combination of 2 separate filters, one MC and the other spatial will be ideal combination to remove as much dirt as possible.
Certainly your approach is a valid one. However, the question is, whether automatic cleaning should be restricted to static areas or not. If you do not exclude motion, then you have to be very conservative, probably too conservative for the static areas. Thus for the gain you make in the moving areas you loose to much in the static areas.
Another question: computation of motion vectors is done for blocks of pixels. It cannot be done for single pixels. Nevertheless, watching your filter with show=true it looks like a pixel rather than a block based technique. Probably this is one reason for the slowness.

[violao]

Originally posted by kassandro
Certainly your approach is a valid one. However, the question is, whether automatic cleaning should be restricted to static areas or not. If you do not exclude motion, then you have to be very conservative, probably too conservative for the static areas. Thus for the gain you make in the moving areas you loose to much in the static areas.
What do you mean by conservative? If the pixel has asigned zero motion vectors then MC previous and next will likely be very similar, therefore replacement pixels should be very similar to both previous and next frames.
Another question: computation of motion vectors is done for blocks of pixels. It cannot be done for single pixels. Nevertheless, watching your filter with show=true it looks like a pixel rather than a block based technique. Probably this is one reason for the slowness.
That's right, one of the reasons is vector interpolation. I'll try to optimize it in future versions. The other is the fact that I use 2 calls to SearchMVs (previous->current and current->next).