cleaning cartoon outlines

[HyperSpad]

I'm looking for a good filter to "solidify" the black outlines. In other words, I want to get rid of the little white dots.

http://www.ic.sunysb.edu/Stu/cspadanu/doom9/venture.jpg

This was not caused by interlacing or anything like that. It came off of a DVD and has not been filtered yet. What's strange is that only certain scenes seem to have this problem. Most of the time, the outlines are very solid.


Thanks
-HYPERSPAD

[dbzgundam]

Nice to see a Venture Bros encode! :D

But I'd suggest some kind of antialiasing perhaps? TIsophote?

[danpos]

FastLineDarken (http://forum.doom9.org/showthread.php?t=82125&highlight=FastLineDarken)

[Eeknay]

Fastlinedarken makes it worse.

Try aaa() on defaults. Seemed to do a decent job.

[HyperSpad]

Could I have links to TIsophote and aaa()?

I'll give these a shot tonight, and post again in the next few hours.

[Backwoods]

# AAA - AnimeAntiAliasing
#
# Thanks @ Didée, mf, Akirasuto, SpikeSpiegel & ScharfisBrain...
#________________________________________________________________________________________
#
# Usage: AAA(Xres, Yres, Xshrp, Yshrp, Us, Ds, chroma)
#________________________________________________________________________________________
#
# Xres/Yres = The final resolution... InputSize = OutputSize is the default
#
# Xshrp/Yshrp = Unfilter strength... Settings of 15,15 are the defaults
#
# Us = Resizer for upsampling... 0 = PointResize (default) / 1 = Lanczos
#
# Ds = Resizer for downsampling... 0 = Bilinear (default) / 1 = Bicubic / 2 = Lanczos
#
# Chroma = Enable/disable chroma antialiasing... Disable = false (default) / enable = true
#________________________________________________________________________________________
#
# Example...
#
# Import("C:\Programme\AviSynth 2.5\plugins\AAA.avs")
#
# AAA(720,576,20,20,1,2,chroma=false)
#________________________________________________________________________________________

function AAA(clip clp, int "Xres", int "Yres", int "Xshrp", int "Yshrp",
\ int "US", int "DS", bool "chroma")
{
clp = clp.isYV12() ? clp : clp.ConvertToYV12()
ox = clp.width
oy = clp.height
Xres = default(Xres, ox)
Yres = default(Yres, oy)
us = default(us, 1)
ds = default(ds, 2)
Xshrp = default(Xshrp, 15)
Yshrp = default(Yshrp, 15)
chroma = default(chroma, false)

us==0 ? clp.PointResize(ox*2,oy*2) : clp.LanczosResize(ox*2,oy*2)

TurnLeft()
SangNom()

TurnRight()
SangNom()

ds==0 ? BilinearResize(Xres,Yres) :
\ ds==1 ? BicubicResize(Xres,Yres) :
\ LanczosResize(Xres,Yres)

Unfilter(Xshrp,Yshrp)

chroma ? MergeChroma(clp.Lanczosresize(Xres,Yres)) : last

}

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\sangnom.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\unfilter.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\AAA.avs")
AVISource("CSS3.avi")
#AAA(720,540,50,50,1,2,chroma=false)

AAA()

function AAAFaster(clip clp, bool "Scale2X")
{
Scale2X = default( Scale2X, false )
clp.Lanczos4Resize(clp.Width * 2, clp.Height * 2)
TurnLeft().SangNom().TurnRight().SangNom()
return Scale2X ? last : ReduceBy2()
}

[foxyshadis]

EEDI2().xResize(width,height)

is the same but faster and better. :p You don't even need to double-up EEDI2 for antialiasing. Then you can sharpen, but if you want it as sharp as it is now, it's a shame to undo and redo it. A better solution would be:


ed=EEDI2().xResize(width,height)
mt_merge(last,ed,combmask(),chroma="process")

This will process only the badly combed areas, unless you prefer the slight softening eedi gives the whole frame.

(x is Lanczos, Bicubic, etc.)

You may actually want to do this during IVTC, which is where it's normally caught. If it's combing related to dot crawl, there are tools for that, but it doesn't really look like it, just vhs transfer artifacts on a dvd somehow. I don't know what your IVTC is like now, but for TIVTC:

assumetff()
interp = separatefields().EEDI2(field=-2)
deint=tdeint(mode=0,edeint=interp)
tfm(clip2=deint)
tdecimate()

If it's currently just force film, don't bother with this though.

mt_merge is masktools2, combmask is masktools1.

[HyperSpad]

TIsophote seems to be doing the trick...

http://www.ic.sunysb.edu/Stu/cspadanu/doom9/tisophote.jpg

I'm only using 3 iterations instead of the default 4 because I'm concerned with some of the detail being lost. It's not very noticeable in the screenshot above, but in shots that are more "zoomed out" (where outlines are thinner) using too many iterations causes a little too much "blurring".

I'm going to encode a couple of episodes tonight with these settings and analyze them tomorrow. I really just want to make sure TIsophote isn't causing any bad side-effects.

danpos, I tried FastLineDarken, but everything seemed to be coming out green. I've tried sangnom in the past and I never figured out why everything was turning green. In any case, I think TIsophote seems to be more of what I needed.

I'll give the rest of the suggestions a try tomorrow.

Thanks everyone,
-HYPERSPAD

[foxyshadis]

filtered=Filter()
mt_merge(last,filtered,combmask(),chroma="process")

is generally applicable for this, you could replace it with TIsophote just as well as eedi2 or anything else you find effective at eliminating the problem. You can hit it a lot harder too, since it'll only affect the combed areas. (And to a lesser extent random noise and very fine patterns.)

[SirCanealot]

This is edge enhancement, isn't it? Try Dehalo_alpha? :/

[MrTroy]

This is edge enhancement, isn't it? Try Dehalo_alpha? :/As its name implies, DeHalo removes halos. White dots aren't halos.

[SirCanealot]

Oh, duh. I posted that just after waking up and missed the white dots. Sorry~
However, there are halos, but they're pretty mild ^^

[HyperSpad]

Here's the comparison between AAA and TIsophote. All wishing to compare/criticize the filters pick out which image you think is better and scroll down for the answers...

http://www.ic.sunysb.edu/Stu/cspadanu/doom9/comparefilters.jpg







The left one is AAA(640,480,50,50,1,2,chroma=false)
The right one is TIsophote(iterations=3,type=0)

AAA seems to have come out a bit sharper, but I wouldn't exactly say that TIsophote came out "soft". AAA left behind something that looks like ringing (near Hank's blonde hair and forehead and I'm guessing in other places). For some areas it actually looks like TIsophote came out better because of the slight blurring. By the way, AAA was much slower, so TIsophote wins the award for efficiency.

But overall, not much of a difference. Those with an eye for detail can point out some of the things I missed. I actually had to layer them in photoshop to see any differences at all.



foxyshadis, I couldn't seem to get ed=EEDI2().xResize(width,height)
mt_merge(last,ed,combmask(),chroma="process")
working. I'm not sure how EEDI2 can be tweaked to increase the effect it has on the outlines. I tried reading the documentation, but changing the inputs didn't seem to have any affect. It seemed to be finding the outlines as I was able to see with the map mode, but it didn't seem to be changing anything, as far as i could tell, when compared to the original clip. Maybe you can give me some hints on what parameters to change to make it work.


Thanks