Problem to mask chroma

[anton_foy]

Hi.
I want to mask out the edges in both luma and chroma on clip using the (bmask) to add onto den.
But I cannot seem to get the chroma to be added correctly, why? Either it seems to apply all the chroma from den
or all the chroma from clip(depending on how I set the Y, U, V -parameters in mt_merge). So the bmask seems to ignore the chroma?

Here is the test script:
function test_LC(clip Input)
{
###this should be added on top of "den" using "bmask"###
clip = Input.hqmc(lt=9,ct=10,ls=0,cs=1,restart=10)

###this should be in the bottom###
den = clip.hqdn3d(lt=0,ls=0,ct=10,cs=6,restart=1).fft3dfilter(sigma=1.7,sigma3=4,bt=3,plane=4,ncpu=4)

###this is masking out edges/detail both the luma and chroma to be taken from "clip" to be added on top of "den"###
bmask = Input.coloryuv(autogain=true).fft3dfilter(sigma=10,sharpen=0,bt=1,plane=0,ncpu=4).
\ ex_edge(mode="prewitt").tweak(cont=10.0).mt_inpand(paramscale="i16").mt_expand(paramscale="i16").
\ mt_expand(paramscale="i16").mt_expand(paramscale="i16").mt_expand(paramscale="i16").fastgaussblur(12)

###here I want to add the masked "clip" (luma and chroma) on top of "den"###
mt_merge(den,clip,bmask, Y=3,U=3,V=3 ,chroma = "process", paramscale="i16").neo_f3kdb()
return last
}

[anton_foy]

So now I have found out that random frames gets weird when I use these scripts (see the images below) and I think it is mt_merge doing it.

I used: clip = ColorYUV(off_u=+255, off_v=-255)
den = fastgaussblur(40)
bmask = coloryuv(autogain=true).fft3dfilter(sigma=10,sharpen=0,bt=1,plane=0,ncpu=4).
\ ex_edge(mode="prewitt").tweak(cont=10.0).mt_inpand(paramscale="i16").mt_expand(paramscale="i16").
\ mt_expand(paramscale="i16").mt_expand(paramscale="i16").mt_expand(paramscale="i16").fastgaussblur(12)

mt_merge(den,clip,bmask, Y=3,U=3,V=3 ,chroma = "process", paramscale="i16") just to see if the mask would leave blue only in masked areas but it did not. Instead I get an unchanged image in the chroma:
Blue mask, untouched chroma (https://ibb.co/xhTsq4n)

yet random frames gets completely blue and some purple:
Blue mask, frame completely blue (https://ibb.co/kxDB9wh)
Blue mask, frame completely purple (https://ibb.co/gr7GYfn)

Here I used: clip = Levels(0, 1, 65280, 0, 0, coring=false, dither=false)
den = fastgaussblur(40)
bmask = coloryuv(autogain=true).fft3dfilter(sigma=10,sharpen=0,bt=1,plane=0,ncpu=4).
\ ex_edge(mode="prewitt").tweak(cont=10.0).mt_inpand(paramscale="i16").mt_expand(paramscale="i16").
\ mt_expand(paramscale="i16").mt_expand(paramscale="i16").mt_expand(paramscale="i16").fastgaussblur(12)

mt_merge(den,clip,bmask, Y=3,U=3,V=3 ,chroma = "process", paramscale="i16") just to see if the mask would leave black only in masked areas and it works at least on luma:
Black mask, frame normal (https://ibb.co/CJDfSMS)

but random frames gets desaturated:
Black mask, random frame desturated (https://ibb.co/hR4r6sK)

I also tried to export the same sequence again and the blue/purple/desaturated frames occur on different frame numbers. Inconsistent, a bug in Masktools2?
I use Masktools2 2.2.26.

[VoodooFX]

I also tried to export the same sequence again and the blue/purple/desaturated frames occur on different frame numbers. Inconsistent, a bug in Masktools2?
I use Masktools2 2.2.26.
If you think that there is bug then try to isolate it. Something similar I had with Overlay ( https://forum.doom9.org/showthread.php?p=1927812 )

[pinterf]

http://avisynth.nl/index.php/MaskTools2
For masktools2 filters default is Y=3, U=1 V=1. So chroma is unprocessed, not even copied by default. Probably all mt_inpand/expand call returns with whatever chroma content is found is memory. Previous frame, something which is found in cache. Paramscale is unnecessary there as well.

[pinterf]

In wiki see "common parameters" section

[anton_foy]

http://avisynth.nl/index.php/MaskTools2
For masktools2 filters default is Y=3, U=1 V=1. So chroma is unprocessed, not even copied by default. Probably all mt_inpand/expand call returns with whatever chroma content is found is memory. Previous frame, something which is found in cache. Paramscale is unnecessary there as well.

Thank you both, yes I have used: x = 3 : the plane will be processed with the processing the filter is designed to do. Is this not correct to process the chroma when U and V are set to 3? I mean the mask itself does not have any chroma information in itself right? So the inpand()/expand() is just how the mask should shrink/grow?
I have now successfully used: overlay(den,clip,mask=bmask) it is processing chroma correctly although it is much slower than mt_merge.

[Reel.Deel]

Is this not correct to process the chroma when U and V are set to 3? I mean the mask itself does not have any chroma information in itself right? So the inpand()/expand() is just how the mask should shrink/grow?


You should set luma=true in mt_merge, when set to true it uses the luma plane as a mask for the chroma and forces chroma processing. mt_inpand/mt_inpand/mt_expand and other masktools filters all default to u/v=1 so the chroma contains nondeterministic data and if you feed that mask to mt_merge it will process those channels with a bad mask.

edit:
overlay(den,clip,mask=bmask) it is processing chroma correctly although it is much slower than mt_merge.

That is because Overlay has a parameter named greymask that defaults to true and uses the first plane as a mask, similar to the luma parameter in mt_merge. If you were to set greymask=false you would get the same results that you're getting with mt_merge.

[anton_foy]

You should set luma=true in mt_merge, when set to true it uses the luma plane as a mask for the chroma and forces chroma processing. mt_inpand/mt_inpand/mt_expand and other masktools filters all default to u/v=1 so the chroma contains nondeterministic data and if you feed that mask to mt_merge it will process those channels with a bad mask.

edit:


That is because Overlay has a parameter named greymask that defaults to true and uses the first plane as a mask, similar to the luma parameter in mt_merge. If you were to set greymask=false you would get the same results that you're getting with mt_merge.

Thank you so much! Now it is working and so much faster than using "overlay". luma=true did the trick.