Converting mask from Gray16 to Gray 32

[Reclusive Eagle]

I'm trying to mask merge with BM3D Cuda.

Only 32 bit float is supported so I need a 32 bit mask
Doing this works:

clip=core.fmtc.bitdepth (clip,bits=32)
m1 = mv.AnimeMask(clip, shift= -0.5, mode=-1)

However this converts my mask from Gray 16 to GrayS which completely messes up the mask.
Increasing its Opacity and creating Jagged edges and completely changing what is edge detected.
How can I convert it to Gray32 instead of GrayS?

[poisondeathray]

I'm trying to mask merge with BM3D Cuda.

Only 32 bit float is supported so I need a 32 bit mask
Doing this works:

clip=core.fmtc.bitdepth (clip,bits=32)
m1 = mv.AnimeMask(clip, shift= -0.5, mode=-1)

However this converts my mask from Gray 16 to GrayS which completely messes up the mask.
Increasing its Opacity and creating Jagged edges and completely changing what is edge detected.
How can I convert it to Gray32 instead of GrayS?


"GrayS" is Gray32

probably you want

clip=core.fmtc.bitdepth (clip,bits=32, fulls=True, fulld=True)

[Reclusive Eagle]

"GrayS" is Gray32

probably you want

clip=core.fmtc.bitdepth (clip,bits=32, fulls=True, fulld=True)


No, here's (https://imgsli.com/NzkxNDU)my problem.
Even with clip=core.fmtc.bitdepth (clip,bits=32, fulls=True, fulld=True) this is the result. See how the mask completely changes?

[Reclusive Eagle]

No, here's (https://imgsli.com/NzkxNDU)my problem.
Even with clip=core.fmtc.bitdepth (clip,bits=32, fulls=True, fulld=True) this is the result. See how the mask completely changes?

Nvm I solved it by doing this:

clip=core.fmtc.bitdepth (clip,bits=32)

ref = core.bm3dcuda.BM3D(d,sigma=[20,20,20],chroma=True, block_step=3)
d = core.bm3dcuda.BM3D(d, ref=ref, sigma=[10,10,10],chroma=True, block_step=3)

d=core.fmtc.bitdepth (d,bits=16)
clip=core.fmtc.bitdepth (clip,bits=16)

Then your mask doesn't need to change bit depth

[MysteryX]

but... why does the mask change like that in 32-bit?

[Reclusive Eagle]

but... why does the mask change like that in 32-bit?

Please ask. All masks do this. From AnimeMask, retinex_edgemask, everything from Kagefunc and all other implementations of masking.

Idk what's going on with GrayS

[Julek]

You can also do pip install vsutil on your console and then:

from vsutil import depth

clip32 = depth(clip, 32)

[MysteryX]

Please ask. All masks do this. From AnimeMask, retinex_edgemask, everything from Kagefunc and all other implementations of masking.

Idk what's going on with GrayS
If it distorts masks like that, what effect does it have on other video content? What types of clips are affected?

[Reclusive Eagle]

You can also do pip install vsutil on your console and then:

from vsutil import depth

clip32 = depth(clip, 32)
Results in the same problem but the mask looks slightly shifted

[Reclusive Eagle]

If it distorts masks like that, what effect does it have on other video content? What types of clips are affected?

Anything with a Mask I guess? This based on a YUV420P8 source.
I've also tried upsampling to 444. Everything results in the same issue.

It just basically destroys masks. What it looks like is happening is a mask gets traced, then a trace of the trace gets traced. Or masks shift from edges to inverted edges. Either way a double mask is created

[MysteryX]

Is it a pixel-shift issue between Luma and Chroma planes?

Keep in mind the _ChromaPosition frame property

[Reclusive Eagle]

Is it a pixel-shift issue between Luma and Chroma planes?

Keep in mind the _ChromaPosition frame property

No planes are aligned. Keep in mind I've also tried with Luma only so it makes even less sense

[Greenhorn]

If you're calling the mask function on a 32bit clip, you're making the mask in 32bit depth, not converting it from 16 to 32 bits. Converting would imply something more like:

clip16 = (your 16bit clip here)
mask = mv.AnimeMask(clip16, shift= -0.5, mode=-1)
mask32 = mask.fmtc.bitdepth(bits=32)
clip32 = clip.fmtc.bitdepth(bits=32)

which shouldn't really alter the appearance of the mask at all.

I have noticed that some filters (both built-in and external) have substantially different results in float, though I'm not really qualified to speculate as to why. Masking functions and possible other stuff using convolutions? (Also AverageFrames, which just kinda breaks entirely.)

[MysteryX]

ah if it's generating mask in 32-bit, then that's a whole different story. The code can have difference between processing INT and FLOAT values; and that's a filter-specific issue that should be reported to the plugin author.

[Reclusive Eagle]

If you're calling the mask function on a 32bit clip, you're making the mask in 32bit depth, not converting it from 16 to 32 bits. Converting would imply something more like:

clip16 = (your 16bit clip here)
mask = mv.AnimeMask(clip16, shift= -0.5, mode=-1)
mask32 = mask.fmtc.bitdepth(bits=32)
clip32 = clip.fmtc.bitdepth(bits=32)

which shouldn't really alter the appearance of the mask at all.

I have noticed that some filters (both built-in and external) have substantially different results in float, though I'm not really qualified to speculate as to why. Masking functions and possible other stuff using convolutions? (Also AverageFrames, which just kinda breaks entirely.)

Makes sense, though I don't see why it is an issue since edge detection remains edge detection regardless of float or not.
That's really weird. Especially since programs like photoshop have no issues with bit depth changes and you can change 8bit to 32 bit color or cmyk etc with 0 effects to masks.

Idk what Photoshop is written in but it does have Python support and those plugins don't have issues either where as this is a consistent thing with VapourSynth Plugins. There has to be more to it