Removing ringing artifacts for animes

[Manao]

Here is a part of the script I used to clean "The Beauty and the Beast". The version I got ( PAL, french ) presented a lot of ringing artifacts around edges, and was way too sharpened. I wanted to eliminate those artifacts without blurring the picture, and I may have succeeded.source = MPEG2Source("E:\Ripp\beauty.d2v",cpu2="xxxxoo").crop(16,16,-16,-16)

#make a filtered version of the source

blurclip = source.deen("a2d",2,10,12)

#make a edge's mask of the clip, only for luma

edgemask = source.edgemask(8,8,0,0,"sobel",Y=3,U=1,V=1) #(3)

#enlarge the radius of that mask seven times (1)

mask7 = edgemask.blur(0.5).blur(0.5).blur(0.5).blur(0.5).blur(0.5) /
.blur(0.5).blur(0.5).binarize(1,false,Y=3,U=-0,V=-0)

#create a clip with luma and chroma at zero (2)

greenclip = source.blankclip().coloryuv(cont_u = 0, off_u = -255,cont_v = 0, off_v = -255)

#create the final mask

mask = mask7.maskedmerge(greenclip,edgemask,Y=3,V=1,U=1)

#apply the filtering

result = source.maskedmerge(blurclip,edgemask,Y=3,V=2,U=2)

return resultSeveral notes have to be made :

- I used the MaskTools (http://cultact-server.novi.dk/kpo/avisynth/masktools-1.4.1.zip), which are a great tool, but are unfortunately buggy.
- This bugginess explains the (1) and (2) :

(1) : I wanted to enlarge the mask, which could have be done by using inflate() ( from the MaskTools package ), but it didn't work. So I had to use a mean filter (blur), several times, and binarize afterwards.
(2) : I wanted to substract the two mask, there is also a function in the masktools ( YV12Subtract ), but this one also didn't work, so I had to create a blankclip ( in YV12 space, so it isn't black, but darkgreen ), and use maskedmerge to make the difference between the two mask.

- My script may have some similarities with mf's one, but since I don't understand his script very well, I can't be sure
- I only filter the luma plane
- It works only in YV12
- The version of the MaskTools I use is different from the one used by mf in mfToon ( I least I think so, from what I read )
- To adapt my script to a previous version of the MaskTools ( the one used by mf, for example ), remplace line (3) by :edgemask = source.edgemask("sobel",Y=3,U=1,V=1).binarize(12,false,Y=3,U=-0,V=-0)It should work ( not tested though )

- You can see results here (http://jourdan.madism.org/~manao/index.html) ( The screenshots were made with a previous version of the script, but the quality is improved with this new script )
- The script is largely 'tweakable' ( different softening filters, different radius / threshold for the mask, and even the possibility to make different filtering with different radius )
- It's not too slow ( ~14 fps with a null test speed, on an Athlon XP 2000+ )
- It might work with movies but it isn't tested ( yet )

[Mug Funky]

have you tried unfilter(-100,0) or fixVHSoversharp()? these are very nice for un-sharpening pictures like that.

but it looks nice. well done.

[Manao]

I tried Unfilter, but didn't like the results ( in that case, it blurs without removing all artifacts, because it isn't enough wide ). I could not find fixvhsoversharp ( a search points me to three threads, none of them having a link to the filter, which is also not on warpenterprise's page )

Now, a little more on my script. Its idea ( which basically is blurring only the area around the edges, not the edges nor the background ) is interesting, but lacks :

- A good edge's finder. I wasn't happy with the result of msharpen, maskedge ( sobel, special or cartoon ), they either miss some edges, or make them too thick, or find some unwanted edges. I know it's not the fault of the filters but of the problem itself, but it is still very frustrating.

- A way to expand the edges only if the area around them is "uniform" ( in order not to harm detail in the background )

- The treatment of the chroma

My attempt was luckyly efficient only because of the nature of Deen("a2d"), which is in fact a _very_ good filter ( for anime at least ) that I long misjudged. It has the ability to preserve ( totally ) the edges, while eradicating all kind of ringing / oversharpening. In fact, my script gave the same results whether I substracted the edges from the areas where I applyed Deen or not ( which is kind of desappointing :rolleyes: )

Hence, my script becomes much more simple :source = MPEG2Source("E:\Ripp\beauty.d2v",cpu2="xxxxoo").crop(16,16,-16,-16)

blurclip = source.deen("a2d",2,10,12)

edgemask = source.edgemask(8,8,0,0,"sobel",Y=3,U=3,V=3)

mask = edgemask.blur(0.5).blur(0.5).blur(0.5).blur(0.5).blur(0.5) /
.blur(0.5).blur(0.5).binarize(1,false,Y=3,U=3,V=3)

result = source.maskedmerge(blurclip,edgemask,Y=3,V=3,U=3)

return result The difference in the result will only be in the using of chroma and luma, not only chroma.

To catch as much edges as I can, I can make several edge's masks, with different methods, and blend them before the enlarging.

And to finish the prefiltering, I can also filtered the almost uniform area, with a strong spatiotemporal filter. Finally, the script looks like :source = MPEG2Source("R:\beauty.d2v",cpu2="xxxxoo").crop(16,16,-16,-16)

blurredversion = source.deen("a2d",2,10,12)

sobel = blurredversion.edgemask(8,8,8,8,"sobel",Y=3,U=3,V=3)
special = blurredversion.edgemask(8,8,0,0,"special",Y=3,U=3,V=3)
edges = MergeChroma(MergeLuma(sobel,special,0.5),special,0.5) #(1)

edges7 = edges.blur(0.5).blur(0.5).blur(0.5).blur(0.5).blur(0.5).blur(0.5).binarize(1,false,Y=3,U=3,V=3)

backgroundmask = blurredversion.msharpen(1,10,true,true,false) / #(2)
.blur(0.5).binarize(240,false,Y=3,U=3,V=3).blur(0.5).binarize(1,false,Y=3,U=3,V=3) #(3)

result = source.maskedmerge(blurredversion,edges7,Y=3,V=3,U=3)

stabilizedbackground = blurredversion.convolution3d(preset = "moviehq").temporalsoften(2,3,3,6,2) #(4)

final = stabilizedbackground.maskedmerge(result,backgroundmask,Y=3,V=3,U=3)

return final.lanczosresize(640,352)Then again, some comments :

(1) : It's the merging of the two masks, there may be faster ways of doing it. I do not binarize at that point because it's almost useless with what follows

(2) : I make a mask with only truly flat / uniform areas, with the lowest threshold I can use. That doesn't let me a lot of areas in fact, but (3) ( which is basically a closing ( or opening, depending of the point of view )), adds other areas ( which were too thin to be worth preserving )

(4) : that is the strong spatiotemporal filtering : deen + convolution3d + temporalsoften, all at default parameters. Since I select where to apply this filter, it doesn't give me oversoftening. But it do stabilize the picture a little.

The previous script gave me a compressibility gain of 10 %, while improving the looking of the picture. This one should improve a little further the compressibility - at least I hope.

[Mug Funky]

interesting concept.

as far as fattening out the edges... i don't know how to do that without compromising background details (most anime isn't strictly flat colours, and edge detection often picks up things that aren't edges).

whenever i've needed line darkening, i subtract a blurred clip, levels it to about 0,124 (to reveal only the dark parts of the "highpassed" clip), then simply layer the result on top as black.

this looks alright sometimes, but basically just darkens edges, and hence doesn't "conserve luma" the way i'd prefer it to (physics background... what can i say?)

lately i've not had the time to enhance animes to the point of insanity (look to Mf for that...). i just code them as they are, save for a couple of small corrections and a blanket temporalsoften(5,2,2) after everything (for killing transient blocking, and nothing else).

[OBcecado]

This has already been discussed here (http://forum.doom9.org/showthread.php?s=&threadid=61851).




Greetz.

[aaar9800]

Try this:

http://neuron2.net/msmooth/msmooth.html

It gave me some good results

[mf]

Originally posted by OBcecado
This has already been discussed here (http://forum.doom9.org/showthread.php?s=&threadid=61851).




Greetz.
Not really. This is the fullscreen version, and the thread you're linking to is the widescreen version. The widescreen version is alot less icky.

[Manao]

@aaar9800 : Msmooth doesn't act exactly as my script : I only filter around the edges while Msmooth filters everything else. Moreover, even at his lowest strength's setting, it is too strong for a Disney's cartoon.

But, there is something very interesting about this filter : it doesn't make its edge's detection the same way Msharpen does ( which is very astonishing ), and it does it better - from my point of view : edges are thiner, and more numerous.

So it's a better edge's finder than msharpen.

@OBcecado : I knew this thread, but it had fastly gone OT, and the filtering proposed was far too strong ( IMHO : deringing in MPEG2DEC3 blurs to much, and smoothing for sharpening afterwards kills too much details in Disney's cartoons )

@all : I did a visual comparison between two encodes, one with my script, the other without ( only the lanczos resize ). Codec was XviD ( ME 6, VHQ 4, MPEG-Custon - HVSGood, CM, B-frames : 2,150,75,0 ), final size aimed at 604 175 KB.

And I must confess that even if the first pass size without filtering is high ( 1 447 000 KB), the compressed result was surprisingly detailed, and not blocky ( except scenes with the snow storm, and some scenes with a kind of fog, but these are codec killers even at quant 2 ). Of course, there are more ringing artefacts ( those already here and those added by the compression ), but not that much.

In fact, since I overfiltered the background in my script, I lost some details with the first encode, and the gain in compressibility ( 1 336 000 KB for the first pass ), didn't bring me less artifacts ( except ringing ).

So I'll do another encode, filtering only the edges this time ( and using msmooth to find them ).

[Yuri]

Great script! Thank you!