MaskTools2 newbie trying to dehalo

[JEEB]

Hello there.

I have been backing up my Sola DVDs, and so far have been using the very basic command of "dehalo_alpha(darkstr=0)" for its sometimes stronger, sometimes slighter haloing (picture 1 (http://img43.imageshack.us/img43/3696/solapicture1halos.png)) - which has worked, but also made some not-so nice remainder stuff on the areas where the halos used to be (picture 2 (http://img43.imageshack.us/img43/4700/solapicture2dehaloalpha.png)). Then I tried adding some awarpsharp, which seemed to make the halo remainders go away nicely, but with the expense of... well, the original picture (picture 3 (http://img43.imageshack.us/img43/1200/solapicture3awarpsharp.png)).

Then I remembered that there was this wonderful plugin, MaskTools2 - and before long I found the mt_edge command and some others... Ending up at:


MPEG2Source("sola10.d2v", cpu=0, info=3)
TFM(mode=5,d2v="sola10.d2v", slow=2,pp=5,clip2=TDeint(type=3,edeint=nnedi2(),emask=TMM()))
tdecimate()

crop(0,2,0,0)
Spline36Resize(720,480)

MCTemporalDenoise(settings="low")

currentclip = last
warpsharpclip = currentclip.dehalo_alpha(darkstr=0, brightstr=0.85)
warpsharpclip = warpsharpclip.awarpsharp(4,2,0.75,0)

mask = last.mt_edge("cartoon",4,32,4,32)
mask2 = mask.mt_expand().mt_expand().mt_expand()
linemask = mask2.mt_convolution()
mt_merge(currentclip, warpsharpclip, linemask)

gradfun2dbmod()


(Picture 04 (http://img43.imageshack.us/img43/4539/solapicture4mt2filterin.png))

Now what problems do I still have:

The mask isn't completely "perfect", albeit quite good it seems.
dehalo_alpha still modifies colors, leading to darker shades.


So, dear ladies and gentlemen - any ideas on how to make this dirty script of mine better?

13.58MB source clip: clicky (http://www.mediafire.com/?mxutdyh0zay)

[thewebchat]

I don't know what you're doing, but DHA(rx=1.5,ry=1.5,dark=0) kills all the halos that I can see. I also don't know what these "remainders" you're talking about are but dfttest(sigma=0.5) kills some stray white splotches on the edges.

Edit: throwing some removegrains works better than using dft.
Edit2: Or I'm just making stuff up.

[JEEB]

Thank you for your input. The settings you gave give results very close to the defaults (which are 2.0/2.0), but really do not make the leftovers any smaller as far as I can see (picture 5 (http://img22.imageshack.us/img22/8555/solapicture5dharx15ry15.png)). As for what those leftovers I keep babbling about are - you can see them more clearly in this example: dehalo'd (http://img20.imageshack.us/img20/8241/onewol.png) versus some awarpsharp applied (http://img20.imageshack.us/img20/4488/twop.png). Do excuse me for using GIMP for this comparison. Of course awarpsharp overall distorts the image, which is a bad thing, as is the linethinning - but the leftovers of the halos actually go away. As for how visible the 'leftovers' are, I actually do see them in some of the scenes that I've encoded and watched of the previous episodes. Mostly alright, but jumps onto my eyes from time to time when watching video full-screened on my 15.4" 1440x900 laptop screen.

Now, to what I'm actually trying to achieve. Yes, dehalo_alpha by itself uses MaskTools, and I do love it. I just want to limit it to lines as well as do some post-processing, which at the moment consists of awarpsharp, which is one of the most badly overused filters that should be kept away as far as possible from actual use, yes - I do understand that and that is why I actually decided to post what I did at four in the morning.

Let's go over it one thing after another:
1) We make two clips; one original (basically we can use "last" for this as well, but I just wanted to save it under a name) and the filtered (dehalo and awarpsharp).
2) We make an edge mask with mt_edge (pic (http://img16.imageshack.us/img16/4586/13198mask.png)), and make it bigger so that it actually fills the area around the lines where the halos are (best would probably be to only select the area close to the lines instead of the lines themselves, but I have no idea how to implement that algorithmically).
3) We apply the filtered clip onto the original clip only on the area that gets onto the positive (white'ish) area of the clip.

Far from anything perfect and nothing special, but I was just interested if there was anything better I could use for a similar purpose that wouldn't break the overall picture as much. I wonder if Repair or something used correctly might help with it... Also now that I'm not completely tired it seems like those "darker shades" could as well be an optical illusion, hurf.

[*.mp4 guy]

Are you trying to get something along thses lines (http://www.imagebam.com/image/98aab142451708)? Filtering like that requires some very slow operations.

[skystrife]

It's not exactly fancy, nor is it using any masking (outside of the two scripts being used, anyway), but I find this to work fairly well on the source and it doesn't introduce the awarpsharp deforming.

dehalo_alpha(darkstr=0,rx=1.5,ry=1.5)
addborders(4,0,0,0)
edgecleaner(12)
crop(4,0,0,0)

(The addborders and crop are there because edgecleaner, at least the one I have, likes to make a little mess on the left side of the frame--doing this keeps the frame size the same while cropping out the little mess, keeping the frame clean.)

Comparisons:

Source (http://img115.imageshack.us/img115/9118/originall.png) - JEEB's script (http://img41.imageshack.us/img41/505/jeebscript.png) - DHA+edgeclean (http://img43.imageshack.us/img43/65/dhaedgeclean.png)

[JEEB]

That looks quite nice. I was recommended edgecleaner earlier, but I didn't look that far into it, which is my bad I guess. Running DHA completely loose does seem to introduce some slight blurriness on some parts of the background, though.

So I just tried limiting the DHA, but then letting edgecleaner loose. The result was pretty good (pic (http://img40.imageshack.us/img40/5809/solapicture6edgecleaner.png)). It might leave some halos there, but edgecleaner takes care of it quite well as I scroll around the video - I think I've found what to use for this encode of mine.

I had discussed mp4_guy's results as well on IRC, so I guess I'll also keep an eye if anything nice pops out from that side as well. So far, thank you very much - this has been very helpful in finding a way to get rid of halos and what they leave behind.

[Firebird]

Hqdering()
Dehalo_alpha(darkstr=0)

That works quite well too.

Image (http://img208.imageshack.us/img208/3696/solapicture1halos.png)