Observations on TomsMoComp deinterlacer

[neuron2]

I'd like to report on some interesting results I have obtained with trbarry's deinterlacer TomsMoComp.

Consider first the following script:a=avisource("nonfilm1.avi")
b=TomsMoComp(a,0,10,0)
Subtract(b,a)This script applies TomsMoComp to my test clip and then shows the difference between the input and processed clips. The differences show what has been touched by the deinterlacer. Untouched areas appear as grey. The test clip has some static figures standing in the foreground and a moving hand in the background (available on request). The result of the differencing is shown here (I used BMPs for fidelity):

http://shelob.mordor.net/dgraft/TMC/Result.bmp

It can be seen that the deinterlacer is apparently doing a good job, in that it is deinterlacing the hand nicely while avoiding touching the static figures very much.

But closer examination reveals a serious problem, in my view. Zooming in on any area of the result frame, but especially in the static areas, discloses noise. The entire frame has been overlayed with noise! Here is a small sample:

http://shelob.mordor.net/dgraft/TMC/Noise.bmp

This should not be happening. Non-interlaced parts of the frame should be totally untouched. You can see using the same technique that (e.g.) FieldDeinterlace does not create this noise.

A second problem is disclosed by this script:a=avisource("nonfilm1.avi")
b=TomsMoComp(a,0,1,0)
c=TomsMoComp(a,0,30,0)
Subtract(b,c)This script should show the differences between the effects of the search parameter. But it is clear that there is no difference between 1 and 30! In fact, I could not make the parameter have any effect at all.

If it were not for this noise effect, TomsMoComp would outperform FieldDeinterlace because it seems to have less effect on static areas. But this noise issue should be resolved, IMHO. No point in having fancy noise smoothers if our deinterlacer is going to inject noise. :) I assume this is just a small bug that Tom can easily correct. I hope he does so because if fixed it will be the deinterlacer of choice and I would happily add it to Decomb, as Tom suggested in another thread.

[TheUnforgiven]

i think the search effort does not affect speed as well !
but great filter though.

[neuron2]

@TheUnforgiven

You don't mind that the filter is peppering your clips with noise?

[trbarry]

Maybe the noise should be optional. ;)

- Tom

PS - I've never tried TomsMoComp with 4 parms like you used. I don't know if that makes a difference specifying the clip as one of them. But the fact that your SearchEffort parm seems to have no effect makes me suspicious.

[trbarry]

Donald -

What release of Avisynth are you using? And what processor?

- Tom

[neuron2]

@trbarry

Here's a new script that avoids the extra clip parameter (although it is harmless and perfectly valid). The results are the same.
AVISource("nonfilm1.avi")
b=TomsMoComp(0,5,0)
a=AVISource("nonfilm1.avi")
Subtract(b,a)I used both Avisynth 1.0b5 as well as 2.05. I used both Pentium and Palamino processors. Have you noted the other poster pointing out that the search parameter has no effect on CPU time?

Have you tried to duplicate this?

[trbarry]

Have you tried to duplicate this?

neuron2 -

No, but I will.

I don't suppose you could post about 5 frames from that clip so we could be looking at the same thing? or email it (trbarry#trbarry. com #=@)

- Tom

[neuron2]

@Bach

What are the two values that you think act differently? Thank you.

@trbarry

Yes, I will email you a clip.

[WarpEnterprises]

I can confirm the noise neuron2 is speaking off.
The searcheffort has no noticeable effect on CPU but if you Subtract different values, there is a (VERY) small effect visible e.g. between 1 and 3.

(Win98, celeron850)

[neuron2]

OK, I see the difference between different search parameters now. :) My mistake was not stepping off frame 0. Apparently this filter does not act on each frame by itself, but rather uses intra-frame computation as well. This is also suggested by the fact that the noise on the static figures is higher on frame 0, where there is no previous frame available. Tom, is it so? If so, it complicates using it in Decomb.

[neuron2]

why not simply

a=AVISource("nonfilm1.avi")
subtract(a,a.tomsmocomp(0,5,0))
Simply because I did not think of it. :) Thank you, Bach, for pointing out that concise formulation.

[High Speed Dubb]

@neuron

Did this source have any noise in it? It's possible the noise pattern is from the noise which is being removed by Tom's filter.

[neuron2]

@High Speed Dubb

I had the vertical filter turned off. I want deinterlacing from a deinterlacing filter, not smoothing!

I will have to run some new tests to see if your theory is the case, but it could well be. Can you tell us what filtering is being done when the vertical filter is off? A filter shouldn't silently do things that the user may not want.

I will try to post new test results later tonight, but right now I am in the last stages of preparing the beta of my new motion-adaptive temporal smoother. :)

[High Speed Dubb]

@neuron

I don't think there's any explicit noise reduction in Tom's filter -- I think it could just be implicit in the method. I'll need to check the source for it (or wait for Tom to show up ;) ) to see if that's on the right track.

[trbarry]

A few random points to help understand all this:

1) TomsMoComp measures pixel motion as a change in value from the previous frame. Since there is no previous frame for frame zero it will not search on that frame.

2) Because of 1) the SearchEffort parm will have no effect on the output of frame zero.

3) In Greedy/HM I found the vertical filter useful, not only for inherent noise but also to paper over any deinterlacing artfacts I created myself. And it was cheap to add there for just a couple of assembler instructions so it seemed worth it. I carried this over to TomsMoComp.

4) Assume TopFirst. In this case when the vertical filter is OFF the top field pixels will be copied over unchanged (hopefully).

5) If the motion search turns up a decent match for a pixel value (prev & next fields) then the average of those 2 will be used for the bottom field pixel. The search effort parm roughly determines how many pairs of pixels will be looked at.

6) For unmatched (uncompensated) motion the pixel value for the bottom field pixel is calculated from the closest 10 top field pixels, 5 from each adjacent line. See the huge block of comments on the WierdBob.inc member for this.

7) Not all possible values of search effort are significant. I think currently supported are 0,1,3,5,9,11,13,15 and maybe a couple over 20 that are too high and cause artifacts. For other numbers the next highest on is used.

8) Vertical motion search starts at 9 and seems to cause more artifacts for some reason. I've been recommending 5 for Avisynth but maybe 11 for DScaler if the machine can keep up with it.

But I still haven't had a chance to take a look at neuron2's case.

- Tom

[neuron2]

Given Tom's notes, it looks like the filter is doing smoothing of non-interlaced areas as High Speed Dubb postulated.

Is it possible for this filter to not filter non-interlaced areas (perhaps optionally)? For best picture fidelity, the user may very well not want any unnecessary smoothing.

[High Speed Dubb]

Yeah, Tom's point #5 -- If I understand it right -- could reduce noise, since you'll be choosing a different two pixels each field.

This suggests a potential problem to me. With a large search region, would motion in a lightly textured region lead to incorrect inference of a different motion direction at each pixel?

[Didée]

I like Tom´s MoComp alot, (but) my personal observations so far, shortly:

- Sometimes SearchEffort >5 gives some artefacts.

- The speed difference between low and high SE values is minimal. (Is there something wrong inside, or is it the genious coder Tom?) I definetly cannot confirm to the readme: SE of 30 is too CPU intensive for everybody. Nope.

- In some scenes with fine, horizontal constrained details (like stairs far away, or - ah, "TheMetalThingBetweenACarsHeadlights" [German: "Kühlergrill"]) and either the object moving or the camera panning, it can produce *nasty* moirée effects.

Nevertheless, very good work, Tom. It´s only that it´s 99% perfect for now ... hopefully you´ll get into the last 1% ;)

And I really love the (x,-1,x) option! (Is the horizontal counterpart still in the pipe?)

[TheUnforgiven]

@donald
i prefer the minimum filtering too.

[trbarry]

Given Tom's notes, it looks like the filter is doing smoothing of non-interlaced areas as High Speed Dubb postulated.

Is it possible for this filter to not filter non-interlaced areas (perhaps optionally)? For best picture fidelity, the user may very well not want any unnecessary smoothing.

neuron2 -

This is not intended to be filtering, just an interpolation. Say my best guess was that a pixel moved from the left to the right by 2 pixels in the time between the prev and next frames. But though it is the best guess, the value of that pixel between left and right varied by 2 or 3. I'm hoping that at the time it was in the middle (current time) that it has the median of the 2 values. The alternative is to use it only if it is completely unchanged, in which case the median would be unfiltered. But I think there is more bang for the buck to allow a little slack.

This suggests a potential problem to me. With a large search region, would motion in a lightly textured region lead to incorrect inference of a different motion direction at each pixel.

Lindsey -

You are of course completely correct. :( The probability of false hits goes up on a single pixel (non-block) search as the search area increases. I wrote the first version of this as a heirarchical block search with a small single pixel search as the last step. But I decided to release only the single pixel search because basically I managed to get it working and it could run real time in DScaler. ;)

But indeed somewhere above the range of 5-15 or so the number of artifacts created from false hits may (I'm not sure) outweigh the benifits of a greater number of properly compensated moving pixels. Eventually someday (I'm unreliable in these things ;) ) I intend to put the heirarchical search back in and get it working right.

I look at deinterlacing as a percentage game, optimizing the average quality at the expense of occasionally doing a very limited bit of damage. If there was any definitive "best" answer I don't think anyone would really like the results.

Sometimes SearchEffort >5 gives some artefacts

Didée -

Yep. See above. But also I'm looking for a bug there because I specifically notice the artifacts go up as SE goes from 5 to 9 (1 step). I can't find one though. Also I don't notice this in DScaler. ???

- In some scenes with fine, horizontal constrained details (like stairs far away, or - ah, "TheMetalThingBetweenACarsHeadlights" [German: "Kühlergrill"]) and either the object moving or the camera panning, it can produce *nasty* moirée effects.

It's a grill, and those are aliasing artifacts, common to most interlaced video and deinterlacers. But I am open to suggestions from anyone on this issue. I would love to solve it but am not optimistic except for filtering.

And I really love the (x,-1,x) option! (Is the horizontal counterpart still in the pipe?)

The Horizontal Expand has been complete for a couple weeks. I haven't released it because I ran out of energy, got side tracked with the similar "Checkered Interlace" idea, and it was both slower and not as nice looking as I'd hoped. Sometimes I code like a madman and sometimes I just drop back and think for awhile.

Everybody -

I'm fighting a silly cold and still haven't looked at the data that neuron2 so nicely sent me last night. But I'll try to get to it this afternoon. But please don't use frame 0 of a clip to evaluate TomsMoComp because it can't mocomp that one.

- Tom

[Didée]

Hope you feel better soon, Tom.
A summer cold is a really awful thing!

[neuron2]

@trbarry

Tom, there's really no need any longer to look at my data. High Speed Dubb has correctly diagnosed what is happening. The algorithm's interpolation is causing the difference that we see on the subtraction. I am assessing the implications of that.

Thank you for your interesting explanatory detail on the filter. I am still curious about some aspects of it and will be seeing you in email!

[trbarry]

Okay, I guess I fibbed a little bit. ;)

Even where there is compensated motion TomsMoComp will still clip the calculated value of a weave pixel to stay in the range of its 2 vertical neighbors. Say, for instance, we were looking at 3 pixels in a vertical col somewhere on the screen that had the values:

10 <-- on line 1, in the curr frame
30 <-- on line 2, calculated by motion comp
20 <-- on line 3, in the curr frame

... the mocomp algorithm would clip the value of 30 to be only 20, avoiding obvious weave artifacts in the case where the mocomp was just flat out wrong (see discussion of false hits above). So I guess I do filter a little bit in all case, but I don't see a way around it for the moment. But this filtering will cause a small loss of vertical detail, and will certainly show up on compares. But I stll don't completely understand why it produced a regular pattern of differences, like macro blocks, in the posted picture.

Donald - how did you upscale this to see those results? And by how much?

As far as TomsMoComp SearchEffort parm having no effect, I think this was only for folks looking only at frame 0, where of course that is true. Otherwise my own compares will even show small differences between 5 and 9, on that same clip. Someone correct me if I'm wrong here.

- Tom

[neuron2]

@Tom

I used an app called FATBITS.EXE. It's a screen lens program. It uses nearest neighbour so there is no resampling. I think it was around 8x magnified. You can also scale up the VirtualDub window, but you're stuck with the upper left corner then.

I have a suspicion that you are getting LOTS and LOTS of false hits and this is causing a lot of damage to the uninterlaced parts of the picture. Is there a way for you to quantify the performance in that regard?

[High Speed Dubb]

[This refers to a bunch of stuff from the last few posts, sort of intermixed.]

Okay, I checked over the code, and... There sure is a lot of preprocessor stuff in there -- the search code was pretty hard to follow. But WeirdBob.inc makes sense to me, and looks like a really cool way to handle "bob"ing without jagged looking diagonal lines.

It seems reasonable to me that a deinterlacer would be doing some noise reduction, as well. Both are trying to infer the "true" image in the presence of noise. Deinterlacing just gets to deal with the added fun of missing data.

Certainly it's worth a few subtle artifacts to be able to improve the picture most of the time. TomsMoComp looks really good, despite any small problems we're coming up with. If the false inference of motion can't be detected by eye, then it's not too serious a problem.

Tom -- Do you use the vertical median even when no motion at all is detected? Maybe you could run that only when the matched pixels are relatively far from one another? On the other hand, that might produce a little "fizzling" when motion is incorrectly inferred.

Regarding aliasing of the car's grill -- I think the grill aliasing problem may be worse with TomsMoComp than with other algorithms. Since the algorithm is explicitly looking for motion, it's possible that it could be positively mislead by aliasing, and make the effect even clearer.

[trbarry]

Tom -- Do you use the vertical median even when no motion at all is detected? Maybe you could run that only when the matched pixels are relatively far from one another? On the other hand, that might produce a little "fizzling" when motion is incorrectly inferred.

No. The WierdBob results are just one of two calculated values, the other is the "weave" value calculated from the motion search. One or the other will be used depending on the motion search results. If a moving (or still) pixel is found with a value that doesn't change by more than 5 (IIRC ??) then I use (possibly clipped) motion-weave, else WierdBob.

- Tom

[trbarry]

I used an app called FATBITS.EXE. It's a screen lens program. It uses nearest neighbour so there is no resampling. I think it was around 8x magnified. You can also scale up the VirtualDub window, but you're stuck with the upper left corner then.

I have a suspicion that you are getting LOTS and LOTS of false hits and this is causing a lot of damage to the uninterlaced parts of the picture. Is there a way for you to quantify the performance in that regard?

I don't know how to quantify it, but maybe we can compare it more fairly. For instance, try running the following script against your clip and then magnifying it 8x in vdub using nearest neighbor. This shows the frame changes (noise) of the supposedly still scene in the lower left corner. Compare to your pic above.


a=avisource("d:\vcr\tmc.avi")
b=selectevery(a,5,4,0,1,2,3)
Subtract(a,b)
crop(0,400,64,32)


For frame 4, for instance, you would see this image:

- Tom

[neuron2]

So, if I read you correctly, you are saying that the changes that your filter makes in the non-interlaced areas are no worse than the noise that already exists there?

[neuron2]

I've attached an image that simply compares areas without TomsMoComp (above) to those processed with TomsMoComp.

It is clear that the slight smoothing done by the filter does not damage detail and is in fact quite a welcome effect. My conclusion: this *is* a great filter.

[High Speed Dubb]

@Tom,

I'm a little confused. Isn't the (motion compensated) weave -- with its vertical median -- used when the scene is completely still? That would make more sense to me than using the bob result in that case.

[trbarry]

Donald -

Thank you. :)

Lindsey -

I misunderstood which "vertical median" you were talking about. There are many, but let's consider the (wierd) bob and (mocomp) weave cases separately.

First, the weird bob actualy takes the best of 5 vertical or diagonal medians passing through the pixel in question. But it then always clips to within the range of the vertical 2 neighbors. This is always needed to avoid artifacts such as disappearing flag poles, etc. But this bob value still may not even be used.

Second, is the mocomp (weave) step. One of the possibilities of mocomp is to discover that the pixel location has zero motion. Other possibilities are also tried depending upon the search effort parm. But say our best guess is that a pixel moved from the left to the right and because of intensity changes or noise also brightend from 5 to 9. The calculated value is then 7 and the difference of 9-5 is close enough we will use this value, or one derived from it.

But a couple other things also happen that I've brushed over previously in this thread, just to keep it simpler. We first see how much both vertical neighbors have changed from THEIR previous values. If either of them has changed more than some threshhold then we will assume their was motion (compensated or not) in the area and we will clip the calculated value 7 to be within the range of the vertical neighbors, avoiding obvious weave artifacts on things like the CNBC stock ticker.

But if there is no apparent change in the values of the vertical neighbors we go for broke and try to get the best detail, using the value 7 without clipping. This gives pristine results on (noise free) still scenes, at least on better days. ;)

- Tom

[High Speed Dubb]

Ah -- I get it, now. I was a little confused, and thinking that the median was being applied to the weave pixels. (Though apparently there really is a vertical median being used on the weave pixels, making my question doubly ambiguous. :) )

[High Speed Dubb]

@Tom and Neuron

This is something of a tangent, but... That's an odd looking picture from the difference of two still frames. It looks like there's some vertical correlation in the noise, which isn't something I've normally seen. Neuron -- Was the source from a DVD, or otherwise compressed? It looks like there are some hints of blocks in there.

[neuron2]

It was a capture from an MJPEG hardware compression capture card (at highest data rate).

[High Speed Dubb]

Thanks -- That explains the odd pattern. I hadn't though about differences in noise patterns before -- This probably means that horizontal noise correlation stuff should be optional, so people with compressed sources can turn it off.

[trbarry]

Lindsey -

Also remember that 8x upscaling with a nearest neighbor algorithm will also create 8x8 apparent blocks. Notice my own confused comments somewhere above about this.

- Tom

[High Speed Dubb]

So that picture was really just ~32 (upscaled) pixels high? That's what I'd been figuring.

I'm looking at a larger scale effect -- For example, the fifth(?) column from the left looks darker than the surrounding columns.

[trbarry]

This thread has served a useful purpose. While I don't really believe TomsMoComp usually adds much noise to a picture a couple of people have commented on the artifacts that sometimes appear after increasing the SearchEffort parm from 5 to 9, a single step since 6-8 don't yet exist. This is problably also a problem in DScaler though I haven't seen it yet due to the more dynamic nature of real time display.

But SE=9 is where I start doing vertical motion search. This seems more vulnerable to noise for some reason and can tend to amplify small changes. The worst effects seem to be around stationary horizontal lines or edges.

I have a couple ideas but am open to suggestions here. Part of the problem is that, in any given single field, an interlaced frame has only half the vertical resolution of horizontal. This means the smallest increment of vertical motion I can measure is really only 2 pixels / field. Add to this the fact that capture card noise due to electrical stuff tends to be more pronounced vertically because vertical pixels are separated by a larger increment of time than horizontal, allowing more drift in the A/D components. (if I understand this at all correctly)

It is not really clear what the solution is. I have considered things like half pel search or smooth pixel blending but would really like to understand the problem better. So if anyone thinks they have any understanding of this, or can even narrow down the problem better, please speak up.

- Tom

[High Speed Dubb]

> ...This is problably also a problem in DScaler though I haven't seen it
> yet due to the more dynamic nature of real time display.

I’ve seen a little vertical fizzling in DScaler. It has more trouble there than GreedyHM. Looking a bit at motion over textures, I _think_ there were some minor problems — but it was subtle, and there are usually subtle motion glitches even in unfiltered over-the-air TV.

[edited out a nonsensical comment about vertical motion ;) ]

> Add to this the fact that capture card noise due to electrical stuff
> tends to be more pronounced vertically because vertical pixels are
> separated by a larger increment of time than horizontal, allowing more
> drift in the A/D components. (if I understand this at all correctly)

I think that’s actually an advantage, since it means the noise between vertically nearby pixels won’t be correlated, possibly leading to incorrect inference of motion.

> ...So if anyone thinks they have any understanding of this, or can even
> narrow down the problem better, please speak up.

I don’t think I’ve got a great understanding of it, but that’s never stopped me from coming up with suggestions. ;) Here are a few hare-brained ideas.

- Reduce the threshold for color differences when the search region is large, so the error rate doesn’t go up.
- Use a different (smaller) threshold for vertical than horizontal differences, since vertical pixels are further away, and therefore are less likely to have the moved pixel.
- When near the bob/weave decision threshold, don’t choose one, but rather weight the bob and weave results based on the closeness of the motion match. This won’t improve your inference, but it will hide mistakes.
- Save the direction of motion (vertical and horizontal difference) from the previous pixel. You could then reject motion which differs too much from that direction. The reason for this is that motion in a particular area of the screen will tend to be in one direction.
- You could swipe the correlated motion stuff from the adaptive noise filter, using your motion detecting method rather than the simple color differences used in the noise filter. This would help because motion tends to occur in a bunch of nearby pixels.

But I think the hierarchical method you’re working on is probably the best way to go. That should really drop any sensitivity to noise or textures.