Dehaloing advice needed

[Chainmax]

I'm trying to clean up some X-Men DVDs and have come to a crossroads. The source has some weird, nasty haloing that DeHalo_Alpha can't completely remove. Here are some before and after shots:

Before:
http://img210.imageshack.us/img210/751/source2gj.png (http://imageshack.us)

After:
http://img50.imageshack.us/img50/3429/filtered5sh.png (http://imageshack.us)


Those two were done with Lanczos4Resize, the following were made with BilinearResize.


Before:
http://img160.imageshack.us/img160/7310/before008401202gm.png (http://imageshack.us)

After:
http://img160.imageshack.us/img160/8195/after008401206es.png (http://imageshack.us)


Before:
http://img464.imageshack.us/img464/5201/before009200267il.png (http://imageshack.us)

After:
http://img159.imageshack.us/img159/38/after009200262nl.png (http://imageshack.us)


Before:
http://img232.imageshack.us/img232/4985/before009494897bb.png (http://imageshack.us)

After:
http://img456.imageshack.us/img456/7864/after009494895bs.png (http://imageshack.us)


Before:
http://img158.imageshack.us/img158/4950/before001179113th.png (http://imageshack.us)

After:
http://img474.imageshack.us/img474/6518/after001179112oo.png (http://imageshack.us)


Before:
http://img58.imageshack.us/img58/7682/before7lq.png (http://imageshack.us)

After:
http://img58.imageshack.us/img58/9766/after8qh.png (http://imageshack.us)


As you can see, most of the haloing is removed, but in many cases the halos are instead replaced by a translucent dark contour. Also, as you can see in the text on the last couple of pictures, some detail is lost which is fine, but pretty much forbids adding another halo remover like MPEG2Source's cpu2="ooooxx" parameter. What would you recommend me to do?

[Didée]

It's the old problem of "Halos with sharp borders", caused by multistep-enhancement.

As of now, you'll need an additional filter for that, applied after Halo removal. Somehow, HQDering() and BlindDeHalo(PPmode=-1) come to my mind ...

[Chainmax]

HQDering causes aliasing with default settings (don't know what other smoother/settings to use) and BlindDeHalo blurs too much and kills detail.

[Revgen]

Couldn't you just color it in with a magic marker?:D

[tedkunich]

At the risk of stirring up the hornets nest again... it is just cartoons... encode those puppies and call it done already! ;)

T

[foxyshadis]

Try following up whatever you do with vmtoon, unless you're already using it. It's excellent at recreating sharp edges and smooth color fills. Good for killing ringing and smoothing. Slow, but not quite Didée slow. :D

[Audionut]

BlindDeHalo blurs too much and kills detail.

Turning down the strength parameter helps.;)

[Didée]

BlindDeHalo blurs too much and kills detail.
Just to make sure we're talking about the same thing -

did you call BlindDeHalo with a *negative* value for PPmode?
Remember that BlindDehalo falls apart in two operations:
a) the main dehaloing, and
b) a postprocessing step somewhat similar to HQDering.

PPmode = 0 ----> only dehaloing
PPmode = 1|2 ---> dehaloing + postprocessing
PPmode = -1|-2 --> *only* postprocessing

When using a negative PPmode, there should not be much blurring or detail removal. Sure, there still might be some spots where the masking goes a 'lil overboard, but generally it shouldn't be an issue.
Additionally, you can limit the changes postprocessing makes, by specifying a value for "PPlimit".

[Chainmax]

Could BlindDeHalo's postprocessing be included in DeHalo_Alpha? I'd rather have only one halo remover, beides BHD3 doesn't worj with MaskTools v2.x

By the way: given the amount of rainbowing, there's no way around the color change in Storm's (3rd pair of pictures) and Wolverine's (5th pair of pictures eyes) due to the derainbowers (BiFrost + SmartSSIQ), right?

[Chainmax]

Come to think about it, BlindDeHalo3 doesn't only has PP values of 1 or 2, not -1 and -2

[Didée]

COme to think about it, BlindDeHalo3 doesn't only has PP values of 1 or 2, not -1 and -2
Sorry, but how's that sentence to be interpreted? Does it have, or does it not? :)

Looking at the script, I'm pretty sure it has ...
# From BlindDeHalo3 script
[...]
postclean = (PPmode== 1) ? maskedmerge(clean,small.bicubicresize(oxi,oyi,1.0,.0),hull)
\ : (PPmode== 2) ? maskedmerge(clean,clean.DEdgeMask(0,255,0,255,"2 3 2 3 0 3 2 3 2"),hull)
\ : (PPmode==-1) ? maskedmerge(i,small.bicubicresize(oxi,oyi,1.0,.0),hull)
\ : (PPmode==-2) ? maskedmerge(i,i.DEdgeMask(0,255,0,255,"2 3 2 3 0 3 2 3 2"),hull)
\ : clean
[...]

[Chainmax]

Indeed it does, I didn't read the BHD thread correctly :o. ANyway, could BDH3's PP modes -1 and -2 be included in DeHalo_Alpha or does BDH3 operate too differently than DeHalo_Alpha to do so?