Simpsons: it just doesn't work (a.k.a I need help...again)

[Chainmax]

So most of you guys will know about the problems I have making a good rip out of my Simpsons 1st Season R4 NTSC discs. Everytime I think I nailed it, closer observation makes me realize that I didn't. So, without further ado, here's (http://www.31012.com/~azulftp/TestClip.vob) a sample vob and here's (http://rapidshare.de/files/11371621/OhWell.mp4.html) my latest encode, made with the following script:

DeDot()
TFM(d2v="X:\wherever\SimpTest.d2v",mode=6,PP=7,slow=2,mChroma=true,chroma=true)
TDecimate(mode=1)
FixChromaBleeding()
ffdshow("default") <-- SPP Deblocking at 75% strength
DeGrainMedian(limitY=5,limitUV=7,mode=0)
Crop(14,0,696,476,align=true)
BicubicResize(640,480,0,0.5)
DeBlock_QED()
DeHalo_Alpha(rx=2.5,ry=2.5)
aWarpSharp(depth=16,cm=1)

Have at it, and hopefully you'll accomplish what I couldn't :(.

[Revgen]

You're simpson's vob isn't that bad. At least not as bad as my Garfield and Friends Season 1 DVD.

Are you trying to make it look like as if you were looking at a real animation cell?

The big problem with cleaning up cartoons is that settings that you use to clean up one scene are too strong or weak in another scene. The way I deal with this is to apply weak and strong settings in different areas of my clip using Stickboys JDL_ApplyRange function. You can get this from www.avisynth.org/stickboy. Make sure to download and install both the jdl-util.avsi and the jdl-range.avsi in your avisynth plugin directory.

Look at my previous post (http://forum.doom9.org/showthread.php?p=761244#post761244) to see how I did it.

EDIT:

Also I'd recommend using EEDI2 as a postprocessor instead of PP=7.

EEDI2 is a better ELA deinterlacer for cartoons than PP=7 as illustrated in this post by tritical. (http://forum.doom9.org/showthread.php?p=744308#post744308)

It shouldn't be that slow since it's only being used to deinterlace leftover combed frames that TFM couldn't handle.

[Chainmax]

It's an awful vob: the blocking can't be eliminated even by blurring the hell out of the picture with SPP at a strength of 200%. I saw the recommendation to use EEDI2 with TDeint's eDeint switch, but never saw a recommendation to use it instead of TFM's PP=7, do I have to do this (http://forum.doom9.org/showpost.php?p=767504&postcount=2)?

[Revgen]

Exactly. However the TFM settings can be changed as long as clip2=deinted is kept in place.

PP=7 is about the same as the other ELA modes in TDeint.

Do you have a DX9 GPU processor? If so I'd also recommend trying TSP's FFT3DGPU plugin. This plugin along with Msmooth worked miracles in really ugly sections of my Garfield clip.


I'll try to see what I can do for your clip later on today or tonight.

[Chainmax]

I don't have a DX9 GPU, but I tried much more aggresive filters than FFT3DFilter (like LRemoveDust_YV12, the aforementioned ffdshow's SPP, DeGrainMedian) without complete success. Those blocks just won't go away, and using sharpening to enhance the picture so that it doesn't look like a muddy mess of course worsens things a lot. By the way, ApplyRange is not needed here, as the whole sample is uniformly awful.

[Revgen]

I've just remembered awhile ago that mg262 (AKA Clouded) released a denoiser for cartoons called Cel Foreground. It basically tries to reconstruct a cartoon or anime's original cel by separating the cel from noise in the video.

It's kind of complicated and I'm trying to read mg262's documentation and figure it out. You can take a take a look at it here. (http://forum.doom9.org/showthread.php?t=99778)

[Chainmax]

I know about that filter, but it never ocurred to me to use it because as you say the instructions seem way too complicated. Besides, I don't know if it's appropriate for this kind of artifacting. Hopefully mg262 stops by here and gives us an opinion.

[Revgen]

I've just tried it.

The good news is that it gets rid of the noise. The bad news is that I haven't figured out how to make it properly recognize the color regions yet.

This filter would be better served if it had it's own GUI.

Here is the script I've tried so far.

LoadPlugin("E:\dgmpgdec146b3\DGDecode.dll")
LoadPlugin("E:\TIVTC.dll")
LoadPlugin("E:\TDeint.dll")
LoadPlugin("E:\EEDI2.dll")
LoadPlugin("E:\De.dll")
LoadPlugin("E:\CelForeground_08Sep05.dll")
mpeg2source("E:\TestClip.d2v")
assumetff()
DeDot()
interp=separatefields().selecteven().eedi2(field=1)
deinted=tdeint(edeint=interp, order=1, field=1)
source=tfm(clip2=deinted, mode=1, order=1, field=1, PP=4)
white = blankclip(source, color=$ffffff).converttoYV12
black = blankclip(source, color=$000000).converttoYV12
red = blankclip(source, color=$ff0000).converttoYV12
seeds = source.Blur(1).Blur(1).LocalMaxima
graph = ConstructFloodGraph(source, seeds, 11, 30)
sizemask = black.ColourGraphBySize(graph, 0, 40)
graph = graph.MaskGraph(sizemask)
absorbedgraph = graph.AbsorbByColour(source, 6, 25, 400000)
source.ColourGraphByAverage(absorbedgraph,source)

[Didée]

First of:
[...]
Crop(14,0,696,476,align=true)
BicubicResize(640,480,0,0.5)
DeBlock_QED()
[...]
Ahuuh, the pain! ;)

As for any deblocker, and for QED **in particular**, it would be wise to NOT resize before, and to KEEP block alignment:
[...]
DeBlock_QED()
Crop(14,0,696,476,align=true)
BicubicResize(640,480,0,0.5)
[...]

But that's only BTW.

Now: try a plain, unprotected "Clense()" and see if it does something useful for your source. If it does in general, we'll see how to avoid the artefacts on motion. (The usual spatial approach "repair(clense,last,repmode)" is counterindicated for fluctuations that blotchy.)

[Chainmax]

I already use SPP deblocking before resizing or cropping, DeBlock_QED is there as a postprocessor of sorts. I'll try using just DeBlock_QED instead of SPP and add Clense() (I have to load Removegrain, right?) after it.

[mg262]

I am looking at this... but in the meantime, if you want a simpler solution, you could try this semi-manual method:

http://forum.doom9.org/showthread.php?t=99890

Edit:
This script looks generally sensible...
seeds = source.Blur(1).Blur(1).LocalMaxima
graph = ConstructFloodGraph(source, seeds, 11, 30)
sizemask = black.ColourGraphBySize(graph, 0, 40)
graph = graph.MaskGraph(sizemask)
absorbedgraph = graph.AbsorbByColour(source, 6, 25, 400000)
source.ColourGraphByAverage(absorbedgraph,source)

The easiest place to tweak it is this: (source, 6, 25, 400000)... you can increase the luma threshold (6) and chroma threshold (25) to merge regions more aggressively until you find that it's causing problems -- i.e. things of dissimilar colour are being merged into single regions.

Let me annotate the script as well to provide something more compact than the main documentation...

Cut the picture up into small regions of similar colour
seeds = source.Blur(1).Blur(1).LocalMaxima
graph = ConstructFloodGraph(source, seeds, 11, 30)

Throw away very small regions to speed the filter up
sizemask = black.ColourGraphBySize(graph, 0, 40)
graph = graph.MaskGraph(sizemask)

Combine the small regions into bigger regions (by merging regions that touch and have similar colours)
absorbedgraph = graph.AbsorbByColour(source, 6, 25, 400000)

Paint over each region in the source with a uniform colour
source.ColourGraphByAverage(absorbedgraph,source)[/QUOTE]

[Didée]

It's way beyond me what you guys are doing in those anime-specific scripts ;) - but this should be noted:

Cut the picture up into small regions of similar colour
seeds = source.Blur(1).Blur(1).LocalMaxima
graph = ConstructFloodGraph(source, seeds, 11, 30)
Don't do this like that. Assumingly some "exact" calculations are done later on, and "blur()" is pretty much inaccurate (as is "sharpen()" ).

Use "RemoveGrain(mode=11)" instead. (Or mt_convolution() or VariableBlur() for 5x5 kernel blurs.)

[Revgen]

@mg262 and Didee

I tried both your recommendations and the "campfire effect" has been significantly reduced, but unfortunately it's still not acceptable.

I'm going to continue to try other denoisers (for the "seeds" parameter) and settings to see if this can be improved.

In the meantime here is my latest script that I tried.

LoadPlugin("E:\dgmpgdec146b3\DGDecode.dll")
LoadPlugin("E:\TIVTC.dll")
LoadPlugin("E:\CelForeground_08Sep05.dll")
LoadPlugin("E:\RemoveGrainPre\RemoveGrainSSE3.dll")
SetMTmode(2,2)
mpeg2source("E:\TestClip.d2v")
assumetff()
tfm(mode=1, order=1, field=1, PP=4)
source=tdecimate(mode=1)
white = blankclip(source, color=$ffffff).converttoYV12
black = blankclip(source, color=$000000).converttoYV12
red = blankclip(source, color=$ff0000).converttoYV12
seeds = source.RemoveGrain(mode=11).RemoveGrain(mode=11).LocalMaxima
graph = ConstructFloodGraph(source, seeds, 11, 30)
sizemask = black.ColourGraphBySize(graph, 0, 40)
graph = graph.MaskGraph(sizemask)
absorbedgraph = graph.AbsorbByColour(source, 10, 30, 400000)
source.ColourGraphByAverage(absorbedgraph,source)

[Chainmax]

I made another encode, this time with the following script:

DeDot()
AssumeTFF()
Interp = SeparateFields().SelectEven().EEDI2(field=1)
Deinted=TDeint(order=1,field=1,EDeint=Interp)
TFM(d2v="X:\wherever\SimpTest.d2v",mode=6,order=1,PP=7,slow=2,mChroma=true,chroma=true,Clip2=Deinted)
TDecimate(mode=1)
FixChromaBleeding()
DeBlock_QED(quant1=25,quant2=50)
Clense()
Crop(14,0,696,476,align=true)
BicubicResize(640,480,0,0.5)
DeHalo_Alpha(rx=2.5,ry=2.5)
aWarpSharp(depth=16,cm=1)

Clense() doesn't help at all, and there are nasty trailing artifacts on motion. You can download the encode here (http://rapidshare.de/files/11535051/OhWell2.mp4.html).

Revgen, what do you mean by "campfire effect"?

[Revgen]

Revgen, what do you mean by "campfire effect"?

It's a flickering effect that the video has when the color regions aren't filled properly by Cel Foreground.



You'll see what I mean when you look at this video.

http://rapidshare.de/files/11540011/simpsons-campfire_effect.avi.html

[mg262]

Let me quote the relevant bit of my reply in the other thread...
That [Campfire] effect happens when areas are unstable from frame to frame... and actually you can notice this on your [processed] video: look for dark areas just above people's heads. When these areas are assimilated into the main background, the average colour of that background is dark; when they are not, it's bright. They belong in the background, i.e. they should always be assimilated... Choosing parameters to do this will reduce the flicker substantially or eliminate it... give me a bit to play with the settings. I might also look at how hard it would be to add a 'median' option which would be immune to flicker.I'm a bit tied up with real-life things (and my CPU with not-quite-so-real-life things!), so I'm afraid it may be a bit before I have something useful -- but I am on it.

Didée,

That blur bit is an insane hack... it's not doing anything precise at all (but thanks for the thoughts!). It basically finds the dark lines outlining objects; it should be clearer with a picture:

http://people.pwf.cam.ac.uk/mg262/posts/Foreground/minima_edgemask.jpg

That was produced by the script

seeds = source.Blur(1).Blur(1).LocalMaxima().expand.greyscale

It's a completely mad way of doing things, but it seems to be working, so I think it's okay to use blur here?

Let me explain the point of this thing as well (and then maybe extract portions of this discussion into documentation)...

We have a picture...

http://people.pwf.cam.ac.uk/mg262/posts/Foreground/still%20foreground%201431%20mini.jpg (http://people.pwf.cam.ac.uk/mg262/posts/Foreground/still%20foreground%201431.jpg)
(click on it for larger version)

... in which we want to
a) find the regions that are originally of a single colour, before all the noise and messiness [HARD] and
b) colour each region with (say) its average colour [EASY].

One way to do the hard step is to use the 'magic wand' tool in Photoshop, which is very similar to the 'paint bucket' tool in many paint programs. If I select this tool and click in the middle of the man's hair, I get this:

http://people.pwf.cam.ac.uk/mg262/posts/Foreground/still%20foreground%201431%20selection%20mini.jpg (http://people.pwf.cam.ac.uk/mg262/posts/Foreground/still%20foreground%201431%20selection.jpg)http://people.pwf.cam.ac.uk/mg262/posts/Foreground/still%20foreground%201431%20selected%20area%20mini.jpg (http://people.pwf.cam.ac.uk/mg262/posts/Foreground/still%20foreground%201431%20selected%20area.jpg)
(Click on it for larger version; I do recommend trying this kind of thing for yourself in a paint package, is it will really help you understand what's going on. What the paint package has done is to select all the nearby pixels whose colour is similar to the one you clicked on.)

Fine, right... you just click in the middle of each block and you're done? Unfortunately, it isn't that simple... the software has no good way of knowing what the 'middle of a block' is. It more or less has to try clicking in random places, and that doesn't always work. In particular: if you click near one of the black lines, it really messes up... and this close-up of the top of the man's head shows you why:

http://people.pwf.cam.ac.uk/mg262/posts/Foreground/still%20foreground%201431%20close-up.jpg

The black line is not a line with crisp edges, but something that slowly gradates from black to (on the hair side) light brown. If you don't click on the deep black, but instead on the dark brown area, you'll end up selecting a bit of hair and a bit of edge... total mess. So the black lines need to be ripped out of the picture to deal with the problem... like this:

http://people.pwf.cam.ac.uk/mg262/posts/Foreground/still%20foreground%201431%20edges%20removed%20mini.jpg (http://people.pwf.cam.ac.uk/mg262/posts/Foreground/still%20foreground%201431%20edges%20removed.jpg)
(Click on it for larger version.)

*That's* where the mask is useful. More to say, but not right now...

[Chainmax]

Revgen: about the EEDI2/TDeint/TIVTC combo, why differentiate between TFF and BFF instead of just using TFM's and TDeint's order=-1 switch?

mg262: I really appreciate all the work you're putting into this, thanks :).

[Revgen]

Revgen: about the EEDI2/TDeint/TIVTC combo, why differentiate between TFF and BFF instead of just using TFM's and TDeint's order=-1 switch?

Avisynth always assumes that the field order is BFF. This can be problematic for TFF material.

You should ask Boulder about it. I think I learned it from him.

[mg262]

So... here's a preliminary version, to show that the campfire effect can be reduced:

http://rapidshare.de/files/11634572/Simpsons_preliminary.avi.html

it still needs work. (I will post and explain a script when everything is finished... but not for reach version because I don't want to drag you all my dead ends; it will be hard enough work with one version to understand!) I think I may implement a median too, to deal with residual flickering... though that will take a little while as I haven't touched the code since September. NB... it's hard to see because of the general noise levels, but I think there is a little luma flickering in the background in the source?

[Chainmax]

Yeah, there is some flickering (not sure if it's luma, chroma or both) on Skinner's suit. By the way, would it be wise to use something like hqdn3d(0,0,3,3).Deblock() to calm down the source before using your filters?

[Revgen]

@mg262

Your vid looks a lot better than what I could do. I'd really like to learn how to use this filter properly since I also have Simpsons and Garfield DVD's as well. So when you have your script done hopefully I can learn what I didn't do right.

Thnx for your continued efforts.

[Chainmax]

Any news on your front, mg262?

[mg262]

Chainmax,

I'm sorry things are taking so long... a combination of real life being very busy and PC instability (I've uninstalled both AVISynth and my compiler). I will definitely come up with something for you, but it will take a bit for assorted work to calm down and for me to sort the PC out (probably reinstall the OS).

[Chainmax]

Oh, I wasn't trying to rush you or anything, I was just curious. Take as long as you want :).