Marching Ants on Telecined Capture

[Benihana]

Hey.
The problem I'm having is that I've recently captured some shows off of Cartoon Network and unlike all other captures I've done before, this is not a pure interlaced video. I examine it frame by frame, and I get a pattern of iipppiipppiippp and ippppippppipppp which leads me to believe that it is of telecined content, and I use telecide and decimate in AVISynth, which doesn't produce any noticible jerks or unfluid motions. The problem I am getting, however, is that in static areas of the scenes on thin lines, I get the "marching ants" effect, and it is quite distracting to the eye. I have seen about 4 or 5 files (of the same cartoon) from other people, and they are all running at 23.976 fps (ntsc) which leads me to believe that the IVTC is not the fault, but somewhere else lies the error in the marching ants problem. My AVS script is:

LoadPlugin("D:\GORDIA~1\mpeg2dec3.dll")
LoadPlugin("D:\GORDIA~1\undot.dll")
LoadPlugin("D:\GORDIA~1\decomb521.dll")
LoadPlugin("D:\GORDIA~1\VSFilter.dll")
LoadPlugin("D:\GORDIA~1\SimpleResize.dll")

avisource("I:\Captures\Capture_20040918-192506.avi")
Telecide(order=1,guide=1).Decimate()
crop(6,58,632,364)
Undot()
Temporalsoften(4,8,8,mode=2,scenechange=10)
LanczosResize(640,368)

Like I said, lines have a flickering/marching ants type effect, and it is even worse on colored lines, as they flicker between light and dark colors. It would be most appreciated if a solution could be found, thanks.

[manono]

Hi-

The deinterlacer is on by default, and it may be deinterlacing good frames and creating that effect. Try turning off postprocessing by setting Post=0 in Telecide. Then load the .avs and go to some of the problem places to see if it looks better. If so, you then have several choices. You can leave it off, you can tweak the GThresh to make it less sensitive, or you can turn on hints and make either KernelDeint or TDeint the deinterlacer, with appropriate settings. Read the manual.

I don't capture, so I don't know if there's something about capturing that makes the problem more likely to occur. If Post=0 doesn't do any good, then you might consider one of the filters that help to control that problem. And there are better people than I that can help you with that.

[Benihana]

Thank you, that indeed helped fix the problem some. Now, the colors don't flicker, but there are still marching ants on the thin lines, particularly the lines that are at an angle as opposed to perfectly vertical or horizontal lines. I've tried messing around with the settings of both Telecide and KernelDeInt, and as of right now, my AVS is:

LoadPlugin("D:\GORDIA~1\mpeg2dec3.dll")
LoadPlugin("D:\GORDIA~1\undot.dll")
LoadPlugin("D:\GORDIA~1\decomb521.dll")
LoadPlugin("D:\GORDIA~1\VSFilter.dll")
LoadPlugin("D:\GORDIA~1\SimpleResize.dll")

avisource("I:\Captures\Capture_20040918-192506.avi")
Telecide(order=1,guide=1,post=0,chroma=false,show=false,hints=true)
kerneldeint(order=1)
Decimate()
crop(6,58,632,364)
Undot()
Temporalsoften(4,8,8,mode=2,scenechange=10)
LanczosResize(640,368)

Your help was greatly appreciated, and now onto the marching ants problem. Smoothing filters don't seem to do anything to the ants, does anybody else have any suggestions for the ant problem? I've taken the settings of Telecide to the extreme, like combinations of gthresh=255, vthresh=255, bthresh=255, and with 0's in place of each as well in all possible combinations, but it seems to have no effect on the marching ants, as the ants are still there. I've even messed with the post=0,1,2,3, but still no luck. Any additional insight would be helpful. Thanks.

[manono]

First, I was wrong above when I said to adjust GThresh to make the deinterlacer less sensitive. I should have said to adjust the VThresh.

From the Decomb FAQ:
I get "marching ants" on some hard horizontal edges when using FieldDeinterlace.
This artifact is a consequence of the field-differencing algorithm employed by FieldDeinterlace. All deinterlacers have characteristic faults like this. There is no perfect solution to deinterlacing. This is usually seen at the edges of letterboxing. You can crop off the letterboxes first and restore them afterwards (if you need them) as a workaround. Alternatively, you can use a frame-differencing deinterlacer, such as TomsMoComp, if you can accept the characteristic artifacts of frame differencing.

I am exploring ways to reduce this artifact and hope for better performance in the future.
So, I don't know. Is it in the source? I only work with DVDs, and sometimes see that with widescreen 4:3 non-anamorphic DVDs. There's not enough resolution to resolve fine lines and hard edges properly. I don't know how or what you're capturing, but I would expect that regular SD TV captures could produce them. I understand that Guava Comb is designed to combat your artifacts:

[Guava Comb(by Lindsey Dubb)]: This is a comb filter, meant to get rid of rainbows, dot crawl, and shimmering in stationary parts of an image.

That's from here:

http://www.avisynth.org/Section+3%3A+Filters%2C+plugins+and+colorspaces#q3.5

And you can find Guava Comb and a bunch of other filters here:

http://www.avisynth.org/warpenterprises/

I also understand that it's slow. Good Luck.

[Benihana]

I thought perhaps it was the capture, so I recaptured many test files to see if the combing effect is present there, indicating interlaced video. Sure enough, it's present. I tried TomsMoComp, Telecide.Decomb KernelDeInt, TDeint, and much more, but I don't think that a deinterlacer will get rid of the buzzing lines. What's happening is that if the line is solid enough, then it appears normal and good. If the line is in the background, or if it's a thin line, then it gets that buzzing effect, which I assume is the interlaced lines. I seperated the fields and watched the thin lines, and they still look as if it is interlaced on the frame.

Even though it's telecined content (I'm pretty sure about that), I decided to run just a plain old FieldDeinterlace() on it, just to see if the lines really are just interlaced and if they can be brought back down to progressive. Still it doesn't work, and they remain ever so shimmering. I'm guessing that there is no way to get rid of the interlace-effect/shimmering/marching-ants/buzzing of the thinner lines in the capture, so I guess I'll have to try another approach. Yea manono, I usually work with DVD's, and they are easy as pie to work with, and most of my capturing works flawlessly well. Just I've never worked with telecined captured content, and that's giving me quite some problems. Hehe, it's almost perfect, save a few messed up lines here and there. Anyways, thanks for your help, hopefully somebody else has a few ideas/suggestions. :)

EDIT: I had screenshots, but they were removed as the problem was solved, and webspace was temporary...

[Benihana]

Thanks a million manono!
I kept calling it "marching ants" and the like, and GuavaComb called it "dot crawl" which was my problem exactly. Thanks a million man, I spent perhaps 8 hours yesterday just reading and trying to figure out my problem. Boy is feels good to have it all suddenly "click" into place. :) Anyways, thanks for the prompt help, I take leave now.