Edge mask help

[Reclusive Eagle]

So currently I am not able to generate a decent edge mask that is sensitive enough to pick up all the lines while not picking up noise.

The noise is not that much of an issue. I can just denoise the mask but that feathers the mask edges creating issues like broken lines and reintroducing the masked out noise.
Here (https://imgur.com/a/F9j8NWa)are some images the original image so you can see what I am working with. The edge mask and end result.

Basically what I am doing is denoising the clip but also selectively sharpening only edges and applying the mask to keep the edges from blurring due to BM3D's denoise.
The problem is, as you can tell in the Result. The mask is not sensitive enough to pick up all lines. Which leads to warped edges like the one circled in blue.

This is prevalent throughout the entire clip and I am losing details.
Currently I am using Animemask to make the edgemask. I have tried Sobel etc but they all are even less accurate creating horrible lines.
Is there someway I can increase sensitivity of masks? Or does anyone have any functions to create better masks? I've been trying these (https://blog.kageru.moe/legacy/edgemasks.html) at Kageru. However, half of them I can't get to work and the other half aren't sensitive enough to pick up all the lines.

Also the Convolution matrix's create green masks. Idk why. Not an issue, just makes things harder to see.
I just want to be able to make a sensitive mask and then manipulate it to get what I want in white. I have tons of experience with Photoshop masks-
I just don't know how to use them through commands. Small things I would just brush out and end of story but you can't do that here.

Any advice? My project can't move forward without solving these basic issues.

[Myrsloik]

Post your pictures to an external image host if you ever want comments.

Green masks are due to the UV planes being mostly 0 after processing. Only process the Y plane and it'll look better.

[Reclusive Eagle]

Post your pictures to an external image host if you ever want comments.

Green masks are due to the UV planes being mostly 0 after processing. Only process the Y plane and it'll look better.

Yup, posted them to imgur (https://imgur.com/a/F9j8NWa). That's the thing though I am only processing the Y plane. Both 1 and 2 values give me the blue and red chroma layers so It's like? and if I convert it to RGB I get color noise in the gray mask. Either way, that's the least of my issues

[Greenhorn]

I'm aware that green masks are the smallest of your problems, but they can also very easily be solved by extracting luma from the clip before processing, something like:

clip = core.ffms2.Source(r"path/goes/here") ## or whatever your script looks like before masking
luma = core.std.ShufflePlanes(clip, 0, vs.GRAY)
## run your masking functions on "luma"
mask = (your masking function of choice here)
## denoise "clip"
denoised_clip = (your denoising function here)
core.std.MaskedMerge(denoised_clip, clip, mask)

of course, you might already be doing things this way, or a better way.

Anyway, I think there are still a lot of tools at your disposal. Just to throw a few things out there:

-- There are still more masking functions to try. Here's a few:
-- https://github.com/HomeOfVapourSynthEvolution/VapourSynth-TCanny
-- https://github.com/Ichunjo/vardefunc/blob/master/vardefunc/ (The "mask" submodule)
-- https://github.com/Irrational-Encoding-Wizardry/kagefunc/blob/master/kagefunc.py (The "retinex_edgemask" function)

And there are definitely more, especially among ported avisynth filters I don't have much experience with as a relative newbie.

-- Prefiltering the clip before creating the mask; ie, creating a copy of it, and running a weak denoise or something like core.std.Median() to try and remove only the most extreme grain, and making the mask from the filtered clip.

-- Binarizing the mask (core.std.Binarize or core.std.BinarizeMask) to exclude low values and force everything else to full white.

-- Expanding the created mask with functions like core.std.Maximum(), core.std.Inflate(), or mt_expand_multi from havsfunc (https://github.com/HomeOfVapourSynthEvolution/havsfunc/blob/master/havsfunc.py)

Generally I've found that there's no free lunch though. Some masking functions are more accurate than others, but more sensitivity = more noise in the mask.

Also, regarding the denoising itself: I'm not an expert on such subjects, but I've found that BM3D can start deleting details at surprisingly low strengths, so I'd do as much experimentation with its parameters as I could before proceeding further. You might want to tweak it per-scene; while that's laborious, so is fiddling with masking functions.

I hope something in all of that is helpful.

[Reclusive Eagle]

I'm aware that green masks are the smallest of your problems, but they can also very easily be solved by extracting luma from the clip before processing, something like:

clip = core.ffms2.Source(r"path/goes/here") ## or whatever your script looks like before masking
luma = core.std.ShufflePlanes(clip, 0, vs.GRAY)
## run your masking functions on "luma"
mask = (your masking function of choice here)
## denoise "clip"
denoised_clip = (your denoising function here)
core.std.MaskedMerge(denoised_clip, clip, mask)

of course, you might already be doing things this way, or a better way.

Anyway, I think there are still a lot of tools at your disposal. Just to throw a few things out there:

-- There are still more masking functions to try. Here's a few:
-- https://github.com/HomeOfVapourSynthEvolution/VapourSynth-TCanny
-- https://github.com/Ichunjo/vardefunc/blob/master/vardefunc/ (The "mask" submodule)
-- https://github.com/Irrational-Encoding-Wizardry/kagefunc/blob/master/kagefunc.py (The "retinex_edgemask" function)

And there are definitely more, especially among ported avisynth filters I don't have much experience with as a relative newbie.

-- Prefiltering the clip before creating the mask; ie, creating a copy of it, and running a weak denoise or something like core.std.Median() to try and remove only the most extreme grain, and making the mask from the filtered clip.

-- Binarizing the mask (core.std.Binarize or core.std.BinarizeMask) to exclude low values and force everything else to full white.

-- Expanding the created mask with functions like core.std.Maximum(), core.std.Inflate(), or mt_expand_multi from havsfunc (https://github.com/HomeOfVapourSynthEvolution/havsfunc/blob/master/havsfunc.py)

Generally I've found that there's no free lunch though. Some masking functions are more accurate than others, but more sensitivity = more noise in the mask.

Also, regarding the denoising itself: I'm not an expert on such subjects, but I've found that BM3D can start deleting details at surprisingly low strengths, so I'd do as much experimentation with its parameters as I could before proceeding further. You might want to tweak it per-scene; while that's laborious, so is fiddling with masking functions.

I hope something in all of that is helpful.


That's actually extremely helpful. I am new to Vapour and AVI synth (only discovered them last week and basically have been pouring every hour of the day into learning Python, reading technical documentation etc) so any advice is amazing.

But yes BM3D does alter details which is why I need to use a mask with it. Dfftest denoises the scene perfectly however, it only denoises about 45% of the noise. And even less if you upscale before denoise. Which is where BM3D comes in because it can eliminate all noise at the cost of sharpness and accuracy which is what I am trying to find a way around.

It doesn't help that I am working with a PAL video with the best source coming from 5k bit rate dvd's from 2007. Do you have any advice beyond denoising and sharpening for this siuation?

I find upscaling before sharpening does eliminate a lot of the sharpened noise. But would I benefit from "oversampling" the video by denoising at higher resolution? E.g I scale it to 4k, apply all filters and then downscale to 1080p? I assume this would eliminate Alising at least.

[Reclusive Eagle]

I'm aware that green masks are the smallest of your problems, but they can also very easily be solved by extracting luma from the clip before processing, something like:

clip = core.ffms2.Source(r"path/goes/here") ## or whatever your script looks like before masking
luma = core.std.ShufflePlanes(clip, 0, vs.GRAY)
## run your masking functions on "luma"
mask = (your masking function of choice here)
## denoise "clip"
denoised_clip = (your denoising function here)
core.std.MaskedMerge(denoised_clip, clip, mask)

of course, you might already be doing things this way, or a better way.

Anyway, I think there are still a lot of tools at your disposal. Just to throw a few things out there:

-- There are still more masking functions to try. Here's a few:
-- https://github.com/HomeOfVapourSynthEvolution/VapourSynth-TCanny
-- https://github.com/Ichunjo/vardefunc/blob/master/vardefunc/ (The "mask" submodule)
-- https://github.com/Irrational-Encoding-Wizardry/kagefunc/blob/master/kagefunc.py (The "retinex_edgemask" function)

And there are definitely more, especially among ported avisynth filters I don't have much experience with as a relative newbie.

-- Prefiltering the clip before creating the mask; ie, creating a copy of it, and running a weak denoise or something like core.std.Median() to try and remove only the most extreme grain, and making the mask from the filtered clip.

-- Binarizing the mask (core.std.Binarize or core.std.BinarizeMask) to exclude low values and force everything else to full white.

-- Expanding the created mask with functions like core.std.Maximum(), core.std.Inflate(), or mt_expand_multi from havsfunc (https://github.com/HomeOfVapourSynthEvolution/havsfunc/blob/master/havsfunc.py)

Generally I've found that there's no free lunch though. Some masking functions are more accurate than others, but more sensitivity = more noise in the mask.

Also, regarding the denoising itself: I'm not an expert on such subjects, but I've found that BM3D can start deleting details at surprisingly low strengths, so I'd do as much experimentation with its parameters as I could before proceeding further. You might want to tweak it per-scene; while that's laborious, so is fiddling with masking functions.

I hope something in all of that is helpful.

Side note how do you actually use retinex edgmask? It can't just be
kagefunc.retinex_edgemask(Clip, sigma=1)
As the only functions?


Also am I doing something wrong here? I'm trying out vardefunc mask. Installed it and here is my code

import vardefunc as vf
clip= vf.get_mask(clip, lthr = 0.0, multi= 1.0)

AttributeError: module 'vardefunc.mask' has no attribute 'get_mask'

[Greenhorn]

Side note how do you actually use retinex edgmask? It can't just be
kagefunc.retinex_edgemask(Clip, sigma=1)
As the only functions?


Also am I doing something wrong here? I'm trying out vardefunc mask. Installed it and here is my code

import vardefunc as vf
clip= vf.get_mask(clip, lthr = 0.0, multi= 1.0)

AttributeError: module 'vardefunc.mask' has no attribute 'get_mask'

For the first, if I imported kagefunc as kgf it'd just be kgf.retinex_edgemask(clip), yeah

For script collections with submodules you have to specify the submodule. For that one it'd be

vf.mask.Kirsch().get_mask(clip)

where Kirsch() can be replaced with any of the other methods in the mask.py file. (The various classes defined as Class MaskName(EdgeDetect))

Also, apparently most of the masks in that actually came from G41Fun (https://github.com/Vapoursynth-Plugins-Gitify/G41Fun), so maybe give its EdgeDetect function a spin.

[Reclusive Eagle]

For the first, if I imported kagefunc as kgf it'd just be kgf.retinex_edgemask(clip), yeah

For script collections with submodules you have to specify the submodule. For that one it'd be

vf.mask.Kirsch().get_mask(clip)

where Kirsch() can be replaced with any of the other methods in the mask.py file. (The various classes defined as Class MaskName(EdgeDetect))

Also, apparently most of the masks in that actually came from G41Fun (https://github.com/Vapoursynth-Plugins-Gitify/G41Fun), so maybe give its EdgeDetect function a spin.

Thank you so much! Your help has been amazing!
There really is no info on edgemasks at all for Vapoursynth so I actually think you've accidently made the first guide.

Also, I figured out if you upscale before denoising with BM3D then downscale back to original resolution before maskmerge BM3D
does not modify any of the line work but still denoises effectively.

And since you can also use any clip as reference, denoising with Dfftest, upsacling and using that as a reference for BM3D final before downscaling again.
Should result in clean denoising no matter how aggressive the settings