Masked DeHalo_alpha

[`Orum]

EDIT: I strongly recommend you consider using FineDehalo (http://avisynth.nl/index.php/FineDehalo) over this filter as I feel it does a much better job. That said I'll leave the rest of this post intact for posterity.

I hacked some masking into DeHalo_alpha. Why? Because as much as I like DeHalo_alpha's dehaloing, I hate what it does to actual detail. Looking at BlindDeHalo3 (which I don't think is as good at dehaloing as DeHalo_alpha), and its great masking, I got an idea. Why not take the masking from BlindDeHalo3, and combine it with the dehaloing of DeHalo_alpha? Thus, I made Masked_DHA(), which is basically a combination of the best of each filter IMHO, plus a few minor tweaks to the masking.

First, a screen shot so you can see what I'm talking about - DeHalo comparison (http://img.photobucket.com/albums/v16/yoitsmeremember/video/masked_dha.png)

And finally, the filter itself: (updated 7/29 - improved quality, speed, and default values)
function Masked_DHA(clip clp, float "rx", float "ry", float "darkstr", float "brightstr", float "lowsens", float "highsens", int "msk_pull", int "msk_push", float "ss", bool "show_msk")
{
rx = default( rx, 2.0 )
ry = default( ry, 2.0 )
darkstr = default( darkstr, 1.0 )
brightstr = default( brightstr, 1.0 )
lowsens = default( lowsens, 50 )
highsens = default( highsens, 50 )
msk_pull = default( msk_pull, 48 )
msk_push = default( msk_push, 192 )
ss = default( ss, 1.5 )
show_msk = default( show_msk, false )

rx = max(rx, 1.0)
ry = max(ry, 1.0)
darkstr = max(min(darkstr, 1.0), 0.0)
lowsens = max(min(lowsens, 100), 0)
highsens = max(min(highsens, 100), 0)
msk_pull = max(min(msk_pull, 254), 0)
msk_push = max(min(msk_push, 255), msk_pull + 1)
ss = max(ss, 1.0)

LOS = string(lowsens)
HIS = string(highsens/100.0)
DRK = string(darkstr)
BRT = string(brightstr)
MPL = string(msk_pull)
MPS = string(msk_push)
ox = clp.width()
oy = clp.height()

# Parameters that can only be adjusted from within the script
uv = 1
uv2 = (uv==3) ? 3 : 2
mbl = 1.58

sm = clp.bicubicresize(m4(ox/rx),m4(oy/ry))
lg = sm.bicubicresize(ox,oy,1,0)
chl = mt_lutxy(clp.mt_expand(U=uv,V=uv),clp.mt_inpand(U=uv,V=uv),"x y -","x y -","x y -",U=uv,V=uv)
lhl = mt_lutxy(lg.mt_expand(U=uv,V=uv),lg.mt_inpand(U=uv,V=uv),"x y -","x y -","x y -",U=uv,V=uv)
mask_i = mt_lutxy(lhl,chl,"y x - y 0.001 + / 255 * "+LOS+" - y 256 + 512 / "+HIS+" + *")

mask_f = mt_lutxy(sm.mt_expand,sm.mt_inpand,"x y - 4 *").blur(mbl).bicubicresize(ox,oy,1.0,0)
\ .mt_lut(yExpr="255 255 "+MPL+" - 255 "+MPS+" - - / x "+MPL+" - *",U=-128,V=-128)

mmg = mt_merge(lg,clp,mask_i,U=uv,V=uv)
ssc = (ss==1.0) ? clp.repair(mmg,1,0)
\ : clp.spline64resize(m4(ox*ss),m4(oy*ss))
\ .mt_logic(mmg.mt_expand(U=uv,V=uv).bicubicresize(m4(ox*ss),m4(oy*ss)),"min",U=uv2,V=uv2)
\ .mt_logic(mmg.mt_inpand(U=uv,V=uv).bicubicresize(m4(ox*ss),m4(oy*ss)),"max",U=uv2,V=uv2)
\ .spline64resize(ox,oy)
umfc = mt_lutxy(clp,ssc,"x y < x x y - "+DRK+" * - x x y - "+BRT+" * - ?",U=2,V=2)
mfc = mt_merge(clp,umfc,mask_f)

return(show_msk ? mask_f : mfc)
}

function m4(float x) {return(x<16?16:int(round(x/4.0)*4))}

The two new parameters are to adjust the masking. Use show_msk=true to see what it looks like. You can even make it a binary mask (caveot lector; this is not optimized for such use, mt_binarize would be faster) by setting msk_push to msk_pull + 1.

Let me know what you guys think.

[thewebchat]

Link. (http://www.imagebam.com/image/840cab42907877) The only differences I see are at edges. Further, no additional detail has been retained, and from what I can see, some halos are no longer being removed.

Certainly this does not bode well for your changes, as cartoons like this should be particularly easy to dehalo.

[`Orum]

Link. (http://www.imagebam.com/image/840cab42907877) The only differences I see are at edges. Further, no additional detail has been retained, and from what I can see, some halos are no longer being removed.
This cannot, in any way, make DeHalo_alpha be *more* powerful, as it's masking off some of the changes. Unless you consider that you might be able to up the strength thanks to masking off some detailed areas that would previously have been destroyed.

Certainly this does not bode well for your changes, as cartoons like this should be particularly easy to dehalo.
Certainly, you misunderstood what I was trying to do. You're using a wrench when you need a hammer. The right tool applied in the wrong context will never win any prizes. There's no detail in most anime/cartoon that *needs* to be masked off, thus if it's masking anything at all, it would be the dehaloing. "Particularly easy to dehalo", sure, if I was designing a filter for animated content, but that's not the purpose here.

[thewebchat]

Since you don't like cartoons, I made another comparison (http://img269.imageshack.us/img269/3227/dehaloalpha.png). As you can see, there is still no significant difference between your "advanced masked version" and the original DHA. Further, in your own screenshot that you provided of the "pro qualities" of your function, there is not even an appreciable difference except the artificially exaggerated difference frame.

[`Orum]

Since you don't like cartoons, I made another comparison (http://img269.imageshack.us/img269/3227/dehaloalpha.png). As you can see, there is still no significant difference between your "advanced masked version" and the original DHA.
I like cartoons, just not for this filter. I do notice a difference in my source, but it really depends on the detail in your source, which I see less of in your comparison.

Further, in your own screenshot that you provided of the "pro qualities" of your function, there is not even an appreciable difference except the artificially exaggerated difference frame.
I see a very noticeable difference, especially on the texture of the wall. Faces, too, retain more more detail in the speckles of the skin. Even without a luma histogram, I could pick out my filter from DHA in almost every scene of the movie.

[xuguang_he]

I can't use this Masked_DHA in Avisynth MT latest version.

It reported, "there is no function named 'max', 'Masked_DHA.avs', line 14".

How could I solve this problem?

:helpful:

[Evil_Burrito]

xuguang_he, I think that means you are missing a plugin. Maybe mt_masktools_26?

[AGKnotUser]

Just rediscovered your function. Just what I was looking for. Worked like a charm.

[StainlessS]

It reported, "there is no function named 'max', 'Masked_DHA.avs', line 14".


It is not the first time I've seen that error message mentioned in posts, Max() is a built-in function:

Max(int, int[, ...]) / Max(float, float[, ...]): Maximum value of a set of numbers.


This is line 14 of the script:

rx = max(rx, 1.0)

both rx and 1.0 are float, I can see no reason for that error message (I dont imagine that mt_masktools has a max function to replace built-in Max).

Anybody got a satisfactory explanation ?

EDIT: Masktools does use eg "max" as a string arg in places eg

mt_logic : clip clip1, clip clip2, string mode("and"), int th1(0), int th2(0)

Applies the function defined by mode to clip1 and clip2.
Possible values for mode are :
"and" : does a binary "and" on each pairs of pixels ( 11 & 5 is computed by converting them to binary, and to and all the bits : 11 = 1011, 5 = 101, 11 & 5 = 1 ).
"or" : does a binary "or" on each pairs of pixels ( 11 | 5 = 1011 | 101 = 1111 = 15 ).
"xor" : does a binary "xor" on each pairs of pixels ( 11 ^ 5 = 1011 ^ 101 = 1110 = 14 ).
"andn" : does a binary "and not" on each pairs of pixels ( 11 & ~5 = 1011 & ~101 = 1011 & 11111010 = 1010 = 10 ).
"min" : equivalent to mt_lutxy("x th1 + y th2 + min").
"max" : equivalent to mt_lutxy("x th1 + y th2 + max").

but cant see why that should be confused with built-in max func.
EDIT: And why should loading masktools fix that problem ? there is something a little strange here.

[lisztfr9]

I'm unsure about how Dehalo is working... it's smoothing a little bit the whole image according to my tests; You can see that the result image is a lot lighter. I can't really imagine a solution to exclude every thing but halo from processing...