Khameleon: bidirectional masking cleaner

[GMJCZP]

For some time I have had frustrations trying to prevent certain denoisers from getting more details of an image, especially with animé/cartoon.
I came to use masks, but I noticed that even mt_edge(mode="sobel") did not fulfill for me what he wanted to achieve.
One day it came to my hands DeHaloH, and I loved the way it takes the edges of the images, thanks mainly to the Camembert function, and there was no turning back.

Recently I thought if you could not reverse the process, that is, only clean the inside of what delimits the edges and in this way simplify the annoying configuration
of cleaning and try to retain as much detail.
So I came up with a two-way (bidirectional) function, Khameleon, for its chameleon effect.

# Khameleon: bidirectional masking cleaner
# version 1.11, 27/05/2021
# Based in DeHaloH, thanks to the authors!
# Developed by GMJCZP
#
# Requirements: TCannyMod, RGTools, MaskTools2, TUnsharp

# SUGGESTIONS FOR CLEANING/RESTORING

# - Suggestions for restoring edges:
# denoised = input.neo_MiniDeen(radius=2, thrY = 10 ,thrUV = 0,u=2,v=2) (used by default by DeHaloH)
# denoised = input.frfun7(1.01,3.0,0).frfun3b(s=8,T=3,tuv=7).neo_MiniDeen(radius=2, thrY = 5 ,thrUV = 0,u=2,v=2)
# denoised = input.frfun7(1.01,14,0).fft3dfilter(sigma=3,sigma2=3.25,sigma3=3.5,sigma4=6).neo_MiniDeen(radius=2, thrY = 10 ,thrUV = 0,u=2,v=2)
# denoised = input.fft3dfilter(sigma=3,sigma2=3.25,sigma3=3.5,sigma4=6).neo_MiniDeen(radius=2, thrY = 15 ,thrUV = 0,u=2,v=2)

# - Suggestions for inner cleaning:
# denoised = input.Temporalsoften(2,3,3,mode=2,scenechange=6)
# denoised = input.dfttest(sigma=2, tbsize=1)

Function Khameleon(clip denoised, clip input, int "DR_Radius", bool "Maska", int "mode",
\ bool "edge", bool "focus", int "str_focus", float "gamma", float "cont", bool "sobel")
{
 edge       = Default(edge, true)
 mode       = Default(mode, edge ? 0 : 1)
 Maska      = Default(Maska, false)
 DR_Radius  = Default(DR_Radius, 0)
 focus      = Default(focus, false)
 str_focus  = Default(str_focus, 40)
 gamma      = Default(gamma, 1.0) # 2.0 for inner cleaning is interesting
 cont       = Default(cont, 1.0)
 sobel      = Default(sobel, false)

 Assert(!(edge == true && gamma <> 1.0), "Gamma values can only be varied for inner cleaning")
 Assert((mode == 0 || mode == 1), "Wrong value for mode")
 Assert(!(focus == true && edge == false), "Focus = true: only valid for edge=true")

 EM = mode == 0 ? input.Camembert2() : input.tweak(cont=cont).TCannyMod(sobel=sobel)

 input_low    = 1
 gamma        = gamma
 input_high   = 128
 output_low   = 255
 output_high  = 0

 lmask = input.mt_lut(Yexpr = "x " +string(input_low)+ " - " +string(input_high)+ " " +string(input_low)+ " - / 1 " +string(gamma)+
    \        " / ^ " +string(output_high)+ " " +string(output_low)+ " - * " +string(output_low)+ " +", U=2,V=2).invert().RemoveGrain(12)#.blur(0.5)

 RM = DR_Radius(EM,DR_Radius,0)
 full = mt_logic(RM, lmask, "and")
 full = edge ? full :  full.Invert().mt_deflate().mt_binarize(threshold=192)
 
 # For blurring edges
 denoised = (edge == true && focus == true) ? denoised.spline16Resize(int(width(input)*1.5),int(height(input)*1.5)).tunsharp(str_focus,type=5).spline16Resize(width(input),height(input)) 
    \ : denoised

 denoised = Maska ? denoised.Invert() : denoised
return mt_merge(input, denoised, full) 
}

Function DR_Radius(clip i, int dr_rad, int count)
{
return count > dr_rad ? i : DR_Radius(i.mt_expand(), dr_rad, count+1)
}

function Camembert2(clip input)
{
input
a=RemoveGrain(12,-1).RemoveGrain(12,-1)
mt_makediff(a,input)
mt_lut("x 128 - abs 2 *")
greyscale()
mt_lut(Yexpr = "X 0 - 115 0 - / 1 1.0 / ^ 255 0 - * 0 +", U=2,V=2)
RemoveGrain(12)
mt_lut(Yexpr = "X 0 - 64  0 - / 1 0.4 / ^ 255 0 - * 0 +", U=2,V=2)
RemoveGrain(12)
RemoveGrain(12)
mt_lut(Yexpr = "X 0 - 30  0 - / 1 1.0 / ^ 255 0 - * 0 +", U=2,V=2)
RemoveGrain(12)
return (last)
}

* Requirements

TCannyMod, RGTools, MaskTools2, TUnsharp


* Syntax and Parameters


- clip denoised

Clip to which the filtering is going to apply, for example:

denoised = neo_MiniDeen(radius=2, thrY = 10 ,thrUV = 0,u=2,v=2) (used by default by DeHaloH)

denoised = dfttest (sigma = 2, tbsize = 1) (for inner cleaning)


- clip input

Original clip


- int "DR_Radius" = 0

Analogous to DeHaloH, it represents the radius or detection range.


- bool "Maska" = false

If true, the detection mask will be displayed.


- int "mode" = 0

The detection mode. By default (mode = 0) the Camembert function is used. Mode = 1 uses TcannyMod. I recommend mode = 0 for cleaning edges and mode = 1 for inner cleaning, but the decision is at the taste of each person.


- bool "edge" = true

Defines whether to clean the edge or the inside of the images.


Now let's look at other additional parameters:


a) For edge = "true"

- bool "focus" = false

If true, it will try to reverse the blur caused by edge cleaning.


- int "str_focus" = 40

Defines the focusing force.


b) For mode = 1

- float "gamma" = 1.00

By increasing this value, it can help improve edge detection to protect.


- float "cont" = 1.0

As with "gamma", increasing this value may improve edge detection.


- bool "sobel" = false

If true, it improves edge detection.


* Examples

For cleaning edges:

orig = last
denoised = neo_MiniDeen(radius=2, thrY = 10 ,thrUV = 0,u=2,v=2)
Khameleon(denoised, orig, DR_Radius=0, edge=true, focus=true, str_focus=40)


For inner cleaning:

orig = last
denoised = dfttest(sigma=2, tbsize=1)
Khameleon(denoised, orig, DR_Radius=1, mode=1, edge=false, focus=false, gamma=2.0, cont=2.0, sobel=true)


* Tips for cleaning edges (for edge=true, use your favourite combo):

neo_MiniDeen: is good for halos and mosquito noise.
frfun7: good for general cleaning and for light aliasing.
fft3dfilter: it's great for rebuilding borders and general cleaning.
frfun3b: for general cleaning. (only for 8 bits!)
dfttest: for general cleaning.
vsDegrainMedian: may be good for random dots and mosquito noise.