High-depth FineDehalo

[`Orum]

I was looking for a high bit-depth version of this, and didn't find one, so I made some edits myself. I haven't yet adjusted the contra sharpening or the FineDehalo2 filters (I hardly, if ever, use them), but maybe I'll have some time later to play with them as well. Or, if someone else wants to take the reigns, feel free.

The results are still slightly different from the 8-bit version, so if anyone spots where I've screwed up, please let me know. It doesn't appear to be in the mt_lut/Expr though, as the masks look identical to me, so I suspect it's related to the other filters used (e.g. RemoveGrain()). Here's what I currently have (the function has been renamed in order to easily compare the results to the original):

# http://avisynth.nl/index.php/FineDehalo
#
# This program is free software. It comes without any warranty, to
# the extent permitted by applicable law. You can redistribute it
# and/or modify it under the terms of the Do What The Fuck You Want
# To Public License, Version 2, as published by Sam Hocevar. See
# http://sam.zoy.org/wtfpl/COPYING for more details.

Function FineDehaloHBD (clip src, float "rx", float "ry", int "thmi", int "thma", int "thlimi", int "thlima", float "darkstr", float "brightstr", int "showmask", float "contra", bool "excl", float "edgeproc")
{
	rx        = Default (rx,          2)
	ry        = Default (ry,         rx)
	thmi      = Default (thmi,       80)
	thma      = Default (thma,      128)
	thlimi    = Default (thlimi,     50)
	thlima    = Default (thlima,    100)
	darkstr   = Default (darkstr,   1.0)
	brightstr = Default (brightstr, 1.0)
	showmask  = Default (showmask,    0)
	contra    = Default (contra,    0.0)
	excl      = Default (excl,     true)
	edgeproc  = Default (edgeproc,  0.0)

	rx_i = Round (rx)
	ry_i = Round (ry)

	src


	### Dehaloing ###

	dehaloed = DeHalo_alpha (rx=rx, ry=ry, darkstr=darkstr, brightstr=brightstr)

	# Contrasharpening
	dehaloed =   (contra > 0)
\	           ? dehaloed.FineDehalo_contrasharp (src, contra)
\	           : dehaloed


	### Main edges ###

	# Basic edge detection, thresholding will be applied later.
	edges = mt_edge (mode="prewitt", thY1=0, thY2=255)

	# Keeps only the sharpest edges (line edges)
	strong = edges.Expr("x "+String(thmi)+" scaleb - "+String(thma-thmi)+" scaleb / range_max *")

	# Extends them to include the potential halos
	large = strong.mt_expand_multi (sw=rx_i, sh=ry_i)


	### Exclusion zones ###

	# When two edges are close from each other (both edges of a single
	# line or multiple parallel color bands), the halo removal
	# oversmoothes them or makes seriously bleed the bands, producing
	# annoying artifacts. Therefore we have to produce a mask to exclude
	# these zones from the halo removal.

	# Includes more edges than previously, but ignores simple details
	light = edges.Expr("x "+String(thlimi)+" scaleb - "+String(thlima-thlimi)+" scaleb / range_max *")

	# To build the exclusion zone, we make grow the edge mask, then shrink
	# it to its original shape. During the growing stage, close adjacent
	# edge masks will join and merge, forming a solid area, which will
	# remain solid even after the shrinking stage.

	# Mask growing
	shrink = light.mt_expand_multi (sw=rx_i, sh=ry_i, mode="ellipse")

	# At this point, because the mask was made of a shades of grey, we may
	# end up with large areas of dark grey after shrinking. To avoid this,
	# we amplify and saturate the mask here (actually we could even
	# binarize it).
	shrink = shrink.Expr ("x 4 *")

	# Mask shrinking
	shrink = shrink.mt_inpand_multi (sw=rx_i, sh=ry_i, mode="ellipse")

	# This mask is almost binary, which will produce distinct
	# discontinuities once applied. Then we have to smooth it.
	shrink = shrink.RemoveGrain (20, -1)
	shrink = shrink.RemoveGrain (20, -1)


	### Final mask building ###

	# Previous mask may be a bit weak on the pure edge side, so we ensure
	# that the main edges are really excluded. We do not want them to be
	# smoothed by the halo removal.
	shr_med = (excl) ? mt_logic (strong, shrink, mode="max") : strong

	# Substracts masks and amplifies the difference to be sure we get 255
	# on the areas to be processed.
	outside = Expr (large, shr_med, "x y - 2 *")

	# If edge processing is required, adds the edgemask
	ep_str  = "x y "+String(edgeproc * 0.66)+" * +"
	outside = (edgeproc > 0) ? Expr (outside, strong, ep_str) : outside

	# Smooth again and amplify to grow the mask a bit, otherwise the halo
	# parts sticking to the edges could be missed.
	outside = outside.RemoveGrain (20, -1).Expr ("x 2 *")

	### Masking ###

	mt_merge (last, dehaloed, outside, y=3, u=2, v=2)

	  (showmask == 1) ? outside.GreyScale ()
\	: (showmask == 2) ? shrink.GreyScale ()
\	: (showmask == 3) ? edges.GreyScale ()
\	: (showmask == 4) ? strong.GreyScale ()
\	:                   last
}

Currently the effect is a bit stronger in the higher bit depth modes than the standard 8-bit. Any ideas?

[real.finder]

I already did port it (and others) https://forum.doom9.org/showthread.php?t=174121

[`Orum]

Ah, thanks! We need to keep the wiki more in sync with what happens here--though I'm sure I share just as much blame for that. I'll work on updating it with what you've adapted.