mClean spatio/temporal denoiser v3.2 (01 March 2018)

[burfadel]

mClean by burfadel

Changelog: https://forum.doom9.org/showpost.php?p=1815046&postcount=3
Dependencies: https://forum.doom9.org/showpost.php?p=1834698&postcount=334

# mClean spatio/temporal denoiser
# Version: 3.2 (01 March 2018)
# By burfadel

# +++ Description +++
# Typical spatial filters work by removing large variations in the image on a small scale, reducing noise but also making the image less
# sharp or temporally stable. mClean removes noise whilst retaining as much detail as possible, as well as provide optional image enhancement

# mClean works primarily in the temporal domain, although there is some spatial limiting
# Chroma is processed a little differently to luma for optimal results
# Input must be 8-bit Planar type (YV12, YV16, YV24) or their equivalents in 10, 12, 14, or 16 bits
# Chroma processing can be disabled with chroma=false

# +++ Artifacts +++
# Spatial picture artifacts may remain as removing them is a fine balance between removing the unwanted artifact whilst not removing detail
# Additional dering/dehalo/deblock filters may be required, but should ONLY be uses if required due the detail loss/artifact removal balance

# +++ Sharpening +++
# Applies a modified unsharp mask to edges and major detected detail. Range of normal sharpening is 0-20, the default 10. There are 4 additional
# settings, 21-24 that provide 'overboost' sharpening. Overboost sharpening is only suitable typically for high definition, high quality sources.
# Actual sharpening calculation is scaled based on resolution.

# +++ ReNoise +++
# ReNoise adds back some of the removed luma noise. Re-adding original noise would be counterproductive, therefore ReNoise modifies this noise
# both spatially and temporally. The result of this modification is the noise becomes much nicer and it's impact on compressibility is greatly
# reduced. It is not applied on areas where the sharpening occurs as that would be counterproductive. Settings range from 1 to 20, default
# value is 14. The strength of renoise is affected by the the amount of original noise removed and how this noise varies between frames. It's
# main purpose is to reduce the 'flatness' that occurs with any form of effective denoising.

# +++ Deband +++
# This will perceptibly improve the quality of the image by reducing banding effect and adding a small amount of temporally stabilised grain to
# both luma and chroma. The settings are not adjustable as the default settings are suitable for most cases without having a large effect on
# compressibility. Auto balance uses Autoadjust, it calculates statistics of the clip, stabilises temporally and adjusts luminance gain & colour
# balance of the noise reduced clip.
# 0=disabled, 1=deband only, 2=auto balance only, 3=both deband and auto balance, 4=deband and veed, 5=all

# +++ Depth +++
# This applies a modified warp sharpening on the image that may be useful for certain things, and can improve the perception of image depth. Default
# is 0 (disabled), and ranges up to 5. This function will distort the image, for animation a setting of 1 or 2 can be beneficial to improve lines. The
# effect

# +++ Strength +++
# The strength of the denoising effect can be adjusted using this parameter. It ranges from 20 percent denoising effect with strength 1, up to the
# 100 percent of the denoising with strength 20 (default). This function works by blending a scaled percentage of the original image with the processed
# image.

# +++ Outbits +++
# Specifies the bits per component (bpc) for the output for processing by additional filters. It will also be the bpc that mClean will process.
# By default, mClean processes as 12 bits if the input is 8 bit, and converts back to 8 bit. If the input is 10 bits or higher no conversion is
# done unless outbits is specified and is different to the input bpc. If you output at a higher bpc keep in mind that there may be limitations
# to what subsequent filters and the encoder may support.

# +++ Required plugins +++
# Latest RGTools, MVTools2, Masktools2, f3kdb, Modplus, AutoAdjust
# Refer to https://forum.doom9.org/showpost.php?p=1834698&postcount=334

function mClean(clip c, int "thSAD", bool "chroma", int "sharp", int "rn", int "deband", int "depth", float "strength", int "outbits")
{

defH = Max (C.Height, C.Width/4*3) # Resolution calculation for auto blksize settings
thSAD = Default (thSAD, 400) # Denoising threshold
chroma = Default (chroma, true) # Process chroma
sharp = Default (sharp, 10) # Sharp multiplier
rn = Default (rn, 14) # Luma ReNoise strength from 0 (disabled) to 20
deband = Default (deband, 4) # Apply deband/veed and/or auto balance
depth = Default (depth, 0) # Depth enhancement
strength = Default (strength, 20) # Strength of denoising.
outbits = Default (outbits, BitsPerComponent(c)) # Output bits, default input depth
calcbits = BitsPerComponent(c) == 8 ? 12 : outbits

Assert(isYUV(c)==true, """mClean: Supports only YUV formats (YV12, YV16, YV24)""")
Assert(isYUY2(c)==false, """mClean: Supports only YUV formats (YV12, YV16, YV24)""")
Assert(isYV411(c)==false, """mClean: Supports only YUV formats (YV12, YV16, YV24)""")
Assert(sharp>=0 && sharp<=24, """mClean: "sharp" ranges from 0 to 24""")
Assert(rn>=0 && rn<=20, """mClean: "rn" ranges from 0 to 20""")
Assert(deband>=0 && deband<=5, """mClean: deband options 0 (disabled) to 5. Refer to description""")
Assert(depth>=0 && depth<=5, """mClean: depth ranges from 0 (disabled) to 5""")
Assert(strength>0 && depth<=20, """mClean: strength ranges from 1 (20%) to 20 (100%, default)""")
Assert(outbits>=8 && outbits<=16, """mClean: "outbits" ranges from 8 to 16""")

padX = c.width%8 == 0 ? 0 : (16 - c.width%8)
padY = c.height%8 == 0 ? 0 : (16 - c.height%8)
c = padX+padY<>0 ? c.addborders(0, 0, padX, padY) : c
cy = ExtractY(c)
sc = defH>2800 ? 8 : defH>1400 ? 4 : defH>720 ? 2 : 1
blksize = sc==8 ? 8 : ((defH/sc)/360)>1.5 ? 16 : ((defH/sc)/360)>0.8 ? 12 : 8
overlap = blksize>=12 ? 6 : 2
lambda = 775*(blksize*blksize)/64
sharp = sharp>20 ? sharp+30 : DefH<=2600 ? 16+round(defH*(34/2600)*sharp/20) : 50
depth = depth*2
depth2 = -(depth+(depth/2))


# Denoise preparation
c = chroma ? Median (c, yy=false, uu=true, vv=true) : c

# Temporal luma noise filter
fvec1 = bitspercomponent(c)>8 ? convertbits(c, 8) : undefined()
bvec1 = bitspercomponent(cy)>8 ? convertbits(cy, 8) : undefined()
super = MSuper (BicubicResize(chroma ? defined(fvec1) ? fvec1 : c : defined(bvec1) ? bvec1 : cy, c.Width/sc, c.Height/sc),
\ hpad=16/sc, vpad=16/sc, rfilter=4)
super2 = MSuper (chroma ? defined(fvec1) ? fvec1 : c : defined(bvec1) ? bvec1 : cy, hpad=16, vpad=16, levels=1)

# --> Analysis
bvec4 = MRecalculate(super2, MscaleVect (MAnalyse (super, isb = true, delta = 4, blksize=blksize, overlap=overlap), sc),
\ blksize=blksize, overlap=overlap, lambda=lambda, thSAD=180)
bvec3 = MRecalculate(super2, MscaleVect (MAnalyse (super, isb = true, delta = 3, blksize=blksize, overlap=overlap), sc),
\ blksize=blksize, overlap=overlap, lambda=lambda, thSAD=180)
bvec2 = MRecalculate(super2, MscaleVect (MAnalyse (super, isb = true, delta = 2, blksize=blksize, overlap=overlap,
\ badSAD=1100, lsad=1120), sc), searchparam=3, blksize=blksize, overlap=overlap, lambda=lambda, thSAD=180)
bvec1 = MRecalculate(super2, MscaleVect (MAnalyse (super, isb = true, delta = 1, blksize=blksize, overlap=overlap, badSAD=1500, badrange=27,
\ search=5, lsad=980), sc), blksize=blksize, overlap=overlap, search=5, searchparam=3, lambda=lambda, thSAD=180)
fvec1 = MRecalculate(super2, MscaleVect (MAnalyse (super, isb = false, delta = 1, blksize=blksize, overlap=overlap, badSAD=1500, badrange=27,
\ search=5, lsad=980), sc), blksize=blksize, overlap=overlap, search=5, searchparam=3, lambda=lambda, thSAD=180)
fvec2 = MRecalculate(super2, MscaleVect (MAnalyse (super, isb = false, delta = 2, blksize=blksize, overlap=overlap,
\ badSAD=1100, lsad=1120), sc), searchparam=3, blksize=blksize, overlap=overlap, lambda=lambda, thSAD=180)
fvec3 = MRecalculate(super2, MscaleVect (MAnalyse (super, isb = false, delta = 3, blksize=blksize, overlap=overlap), sc),
\ blksize=blksize, overlap=overlap, lambda=lambda, thSAD=180)
fvec4 = MRecalculate(super2, MscaleVect (MAnalyse (super, isb = false, delta = 4, blksize=blksize, overlap=overlap), sc),
\ blksize=blksize, overlap=overlap, lambda=lambda, thSAD=180)

# --> Bit depth conversion
c = chroma ? calcbits != BitsPerComponent(c) ? ConvertBits(c, calcbits) : c : c
super2 = calcbits != BitsPerComponent(super2) ? ConvertBits(super2, calcbits) : super2
cy = calcbits != BitsPerComponent(cy) ? ConvertBits(cy, calcbits) : cy

# --> Applying cleaning
clean = MDegrain4(chroma ? c : cy, super2, bvec1, fvec1, bvec2, fvec2, bvec3, fvec3, bvec4, fvec4, thSAD=thSAD)
u = chroma ? ExtractU(clean) : nop ()
v = chroma ? ExtractV(clean) : nop ()
filt_chroma = chroma ? CombinePlanes(c, mt_adddiff(u, clense(mt_makediff(ExtractU(c), u), reduceflicker=true)), mt_adddiff(v,
\ clense(mt_makediff(ExtractV(c), v), reduceflicker=true)), planes="yuv", source_planes="yyy", sample_clip=c) : c
clean = chroma ? ExtractY(clean) : clean

# Post clean, pre-process deband
filt_chroma_bits = BitsPerComponent(filt_chroma)
clean2 = deband==0 ? nop() : ConvertBits(clean, 8)
noise_diff = deband==0 ? nop() : BitsPerComponent(c)==8 ? nop() : mt_makediff(convertbits(clean2, calcbits), clean)
depth_calc = deband==0 ? nop() : CombinePlanes (clean2, filt_chroma_bits>8 ? ConvertBits(filt_chroma, 8) : filt_chroma, planes="YUV",
\ source_planes="YUV", pixel_type="YV12")
depth_calc = deband==0 ? nop() : deband>1 ? deband==4 ? depth_calc : AutoAdjust (depth_calc, auto_gain=true, bright_limit=1.09, dark_limit=1.11,
\ gamma_limit=1.045, auto_balance=true, chroma_limit=1.13, chroma_process=115, balance_str=0.85) : depth_calc
depth_calc = deband==0 ? undefined() : deband<>2 ? f3kdb (depth_calc, preset=chroma?"high":"luma", range=16, grainY=38*(defH/540),
\ grainC=chroma?37*(defH/540):0) :depth_calc
clean = deband==0 ? clean : BitsPerComponent(c)==8 ? ExtractY (depth_calc) : mt_adddiff(ConvertBits(ExtractY
\ (depth_calc), calcbits), noise_diff)
depth_calc = deband==0 ? nop() : BitsPerComponent(depth_calc)<>filt_chroma_bits ? ConvertBits(depth_calc, filt_chroma_bits) : depth_calc
filt_chroma = deband==0 ? filt_chroma : deband>4 ? veed(depth_calc) : depth_calc

# Spatial luma denoising
clean2 = removegrain(clean, 18)

# Unsharp filter for spatial detail enhancement
clsharp = sharp>0 ? sharp>=51<=54 ? mt_makediff(clean, gblur(clean2, (sharp-50), sd=3)) :
\ mt_makediff(clean, blur(clean2, 1.58*(0.03+(0.97/50)*sharp))) : nop()
clsharp = mt_adddiff(clean2, repair(clense(clsharp), clsharp, 12))

# If selected, combining ReNoise
noise_diff = mt_makediff (clean2, cy)
clean2 = rn>0<=20 ? mt_merge(clean2, mergeluma (clean2, mt_adddiff(clean2, tweak(clense(noise_diff, reduceflicker=true), cont=1.008+(0.0032*(rn/20)))),
\ 0.3+(rn*0.035)), mt_lut (overlay(clean, invert(clean), mode="darken"), "x 32 scaleb < 0 x 45 scaleb > range_max 0 x 35 scaleb - range_max 32
\ scaleb 65 scaleb - / * - ? ?")) : clean2

# Combining spatial detail enhancement with spatial noise reduction using prepared mask
noise_diff = mt_invert(mt_binarize(noise_diff))
clean2 = sharp>0 ? mt_merge (clean2, clsharp, overlay(noise_diff, mt_edge(clean, "prewitt"), mode="lighten")) :
\ mt_merge (clean2, clean, overlay(noise_diff, mt_edge(clean, "prewitt"), mode="lighten"))

# Converting bits per channel and luma format
filt_chroma = outbits < BitsPerComponent(filt_chroma) ? ConvertBits(filt_chroma, outbits, dither=1) : ConvertBits(filt_chroma, outbits)
clean2 = outbits < BitsPerComponent(clean2) ? ConvertBits(clean2, outbits, dither=1) : ConvertBits(clean2, outbits)
c = BitsPerComponent(c) <> BitsPerComponent(clean2) ? ConvertBits(c, BitsPerComponent(clean2)) : c

# Combining result of luma and chroma cleaning
output = CombinePlanes(clean2, filt_chroma, planes="YUV", source_planes="YUV", sample_clip=c)
output = strength<20 ? Merge(c, output, 0.2+(0.04*strength)) : output
depth_calc = depth>0 ? defh>640 ? bicubicresize(output, 720, 480) : output : nop()
output = depth>0 ? mt_adddiff(output, spline36resize(mt_makediff(awarpsharp2(depth_calc, depth=depth2, blur=3),
\ awarpsharp2(depth_calc, depth=depth, blur=2)), output.width, output.height)) : output
output = padX+padY<>0 ? output.crop(0, 0, -padX, -padY) : output

return output
}

[MysteryX]

Besides smart block size selection, is this different than the first version?

As I said, I'd be very interested in a script that includes optional correction of other types of defects. I don't know what order of execution gives best results.

Here's a question for you.

How would you define, in technical terms:
- noise
- ringing
- blocking
- banding

... and where do you draw the line between what is and isn't each of the above?

[burfadel]

Changes:

v3.2
- small update to to make use of changes in MvTools2 2.7.25 *** Please update dependencies ****
- analysis will always be done in 8 bit regardless of input depth. This will give a small speed bump with no quality loss for videos with input bitdepth greater than 8
- fixed deband=0 issue

v3.1
- minor tweaks to analysis
- minor tweaks to luma and chroma renoise

v3.0
- considerable amount changed:
- reverted to MDegrain chroma denoising with different handling of chroma
- heavily revised luma cleaning
- sharp scales to resolution based on the multiplier (now 0-24)
- renoise tweaked
- new masks
- remove cstr setting
- added 'depth' function (post-processing)
- added strength function; remixes a portion of the original image back with the processed imaged, scaled from 20 percent to 100 percent at strength 20 (default)
- no longer requires FFT3DFilter or fftw dependency
- requires modplus for chroma processing (in addition to existing features) http://www.avisynth.nl/users/vcmohan/modPlus/modPlus.html
- some other tweaks and changes

v2.3
- modified deband features so that luma bit depth difference is only applied when source material is greater than 8 bit
- resolved minor artifacts that were occasionally produced when there were high contrast differences and flat surfaces
- repurposed dctfilter

v2.2
- corrected an oversight regarding chroma processing of deband features, it was applied to luma but not to chroma
- resolved an associated script bug that didn't affect anything since chroma deband wasn't applied
- luma bit depth detail now retained even when using deband features

v2.1
- processing of denoising now undertaken in 12 bits (or whatever is specified for outbits if greater than 8), analysis still processed at source depth.
- added the use of veed by VCMohan. Deband options 4 is now default (deband+veed), deband option 5 is to use deband, veed, and level adjustment (autoadjust)
- modified renoise
- slight adjustment to the motion mask
- made changes to the temporal stabilisation of renoise and sharpening (and made a correction to the sharpening stabilisation)
- added non-8 bit workaround for modern high bit depth incompatible filters (deband from f3kdb and autoadjust)
- tweaked several parameters
- resolved bit depth issue when using different combinations of input depths and outbits

v2.0
- improved debanding feature, now ranges 0-3. 0=disabled, 1=deband (default), 2=levels/saturation auto balance adjustment only, 3=both
- levels/saturation is automatically adjusted using the Autolevels plugin, only required if manually enabled - https://forum.doom9.org/showthread.php?t=167573
- changed sharpening setting from 'enh' to 'sharp' to better distinguish what it is. 'enh' will be used later for another name appropriate feature
- refractored sharpening, it now increases a little less with higher resolutions
- added chroma renoise when chroma is enabled (default), non-adjustable
- fixed issue with block sizes on higher resolutions
- fixed issue with the passmask used as part of processing; appears Masktool2 may have a bug with the value scaling feature
- tweaked noise processing parameters

v1.9
- added option to disable chroma processing, default is to process chroma
- added an option to change the strength of chroma processing
- added debanding, default is enabled
- adjusted blocksize parameters
- tweaked some other settings

v1.8
- speed increase and reduced memory use for all but the lowest resolutions
- improved quality
- removed cpu option for FFT3DFilter, as any more than 4 threads proves no faster an for high thread counts, appears to run slower

v1.7c
- slight adjustments and slightly better speed

v1.7b
- modified analysis for MDegrain for performance and quality

[MysteryX]

mt_lut evaluates an expression on pixels. It basically allows implementing algorithms without having to write a DLL nor write assembly code. It uses a LUT table for optimization, but that doesn't work for 16-bit videos. Because noise reduction algorithms deal with subtleties and then affect the rest of the script, I'd recommend running it in 16-bit, and mt_lut then isn't a good option. If you need custom algorithms, creating a DLL is always a good option, like I did with FrameRateConverter to detect stripe patterns.

Banding is not related to blocking. Banding is due to rounding where each value appears as a distinct band. To avoid banding, we normally use dithering. No dithering leads to banding.

[burfadel]

Original third post:
The basic noise filtering is similar to the original script, although I did make a small mistake that limited part of the effectiveness of that original script. The difference is what this version of the script does with the outputs of the different filtering. It could also potentially allow for deringing and dehalo reutilising some of the calculations, and this to some extent can be done for deblocking as well, but I'd have to work out the best way of making it effective. Banding is the hard one though, there would be probably no benefit to include that in the script over running a separate filter.

I would describe temporal noise as small variations between each frame on the scale of a few pixels. Spatial noise is small variations on the scale of a few pixels compared to adjacent pixels, that doesn't change too much between frames. This is much harder to remove without affecting actual detail, because it's a math based solution, not an perceptual based where we look at it and determine that it shouldn't be there. Temporal denoising is therefore IMO potentially much more useful out of the two, historially though temporal denoising was considerably slower and not practical. A small amount of spatial denoising I think can be beneficial though, if you can work out how it should be applied.

Ringing is the small 'ring like' artifacts typically next to areas of large contrast difference, caused by resizing or compression. Halo's are a brightening of the edge, typically outside edge due to contrast changes of an object that may be present. This can be the result of oversharpening. Blocking is the visible edges of a block typically caused by not enough bandwidth or the use of an inefficient (by modern standard) codec. In effect it is large scale pixellation. Banding is related to blocking, it's the visualisation of the block boundaries on a flat area that has gradient, again caused by encoder or transfer inefficiencies.

No promises on a timeframe for the ringing, halo, deblocking filter, or whether it can be done related to a concept I have in mind. I think for best results I might have to use mt_lut functions, and to be honest I don't know how to use that. The documentation for masktools is a little lacking regarding most of its power features. There's a function called mt_gradient(), but not sure whether that actually does what it sounds like... and again, no idea how to use it!

-------------------

Ah ok. Makes sense about the banding, I thought that rounding occurs on a per block case causing the edges of the blocks to become pronounced over flat areas that have a gradient. Does the banding occur mid block? As for blocking, do you know if DCTFilter could be used for that, to ascertain block boundaries etc?

Updated version here:
https://github.com/chikuzen/DCTFilter/releases

I do have an idea for deringing and dehalo, however it would have to wait until the weekend to even contemplate sitting down and nutting it out :). I'd set it to enable it as an option, likewise with any deblocking.

[feisty2]

mt_lut evaluates an expression on pixels. It basically allows implementing algorithms without having to write a DLL nor write assembly code. It uses a LUT table for optimization, but that doesn't work for 16-bit videos. Because noise reduction algorithms deal with subtleties and then affect the rest of the script, I'd recommend running it in 16-bit, and mt_lut then isn't a good option. If you need custom algorithms, creating a DLL is always a good option, like I did with FrameRateConverter to detect stripe patterns.

Banding is not related to blocking. Banding is due to rounding where each value appears as a distinct band. To avoid banding, we normally use dithering. No dithering leads to banding.

no, pixel-wise evaluations are just literally, "pixel-wise", u got no access to the neighbor pixels and that renders it much less useful than a dynamic library

a Gaussian blur with a radius of 1 is simply like

dstp[y][x] = (srcp[y-1][x-1] + 2 * srcp[y-1][x] + srcp[y-1][x+1] + 2 * srcp[y][x-1] + 4 * srcp[y][x] + 2 * srcp[y][x+1] + srcp[y+1][x-1] + 2 * srcp[y+1][x] + srcp[y+1][x+1]) / (1 + 2 + 1 + 2 + 4 + 2 + 1 + 2 + 1);

for a c++ plugin

now how is that gonna work for ur fancy LUT or whatever?

well, another fun fact is that it's actually possible in vaporsynth with Expr

topleft = core.std.AddBorders(core.std.CropRel(clp, 0, 1, 0, 1), 1, 0, 1, 0)
topcenter = ...
topright = ...
adjacentleft = ...
center = clp
adjacentright = ...
bottomleft = ...
bottomcenter = ...
bottomright = ...

clp = core.std.Expr([topleft, topcenter, topright, adjacentleft, center, adjacentright, bottomleft, bottomcenter, bottomright],
"x y 2 * + z + a 2 * + b 4 * + c 2 * + d + e 2 * + f + 1 2 + 1 + 2 + 4 + 2 + 1 + 2 + 1 + /")

ain't that pretty, eh? that's why it's only possible but not practical

and I'm damn sure it's not even possible in avisynth
edit:
or maybe possible with y8rpn, but you see the point, it's nasty

[Mounir]

there is no function named veed that's what i get, any idea?
i can't find the plugin veed anywhere
nevermind, i found modplus(which contain veed i think)

now i get:
manalyse blocks must be 4x4, 8x4, 16x2 blabla

[feisty2]

why not use mt_convolution() / mt_luts() / core.std.Convolution()?

mt_lut is a pixel-wise evaluator, std.Convolution is NOT
the toy in vaporsynth corresponding to mt_lut(xyz) should be std.Expr (function-wise, they do things differently tho)

[feisty2]

Your code is the same as
core.std.Convolution(matrix=[1, 2, 1, 2, 4, 2, 1, 2, 1])
or simply
core.rgvs.RemoveGrain(11)
right?

So why you write such complicated code?

to show that a pixel-wise evaluator is far from enough to code any sophisticated algorithm

std.Convolution is NOT A PIXEL-WISE EVALUATOR, it is NOT CORRESPONDING TO mt_lut, stop distracting me

the point is there're limitations for pixel-wise evaluators, not how to perform a Gaussian blur quick and fast, that Gaussian blur thing is just a demonstration of the point

[feisty2]

I never said that. Maybe I should stop speaking because this is an avs thread and it seems that we both misunderstand each other.


the point is there're limitations for pixel-wise evaluators, not how to perform a Gaussian blur quick and fast, that Gaussian blur thing is just a demonstration of the point

did you even read my previous posts?

[feisty2]

How would you define, in technical terms:
- noise
- ringing
- blocking
- banding

... and where do you draw the line between what is and isn't each of the above?

noise: any unwanted components in the signal, concretely, noise is generally modeled as a random signal that follows Gaussian distribution in most denosing algorithms, this random signal could be canceled out by various approaches, bilateral assumes that pixel-wise weighted averaging could cancel out the signal, DFT assumes that if you extract a piece of pattern from the image, you will notice something fishy in that pattern and it's an intra-pattern based approach, pixel and block matching assume that noise could be canceled by averaging similar patterns, it's an inter-pattern based approach

ringing: https://en.wikipedia.org/wiki/Gibbs_phenomenon

blocking: has definitely nothing to do with low-bitrate, the real reason it happens with some obsolete codecs is that, macroblocks in those codecs do not share any overlap

banding: lack of quantizing precision, 8bit sucks, it won't happen if the entire process chain has a higher precision, say, 32bit float

[burfadel]

that's what i get, any idea?
i can't find the plugin veed anywhere
nevermind, i found modplus(which contain veed i think)

now i get:
manalyse blocks must be 4x4, 8x4, 16x2 blabla

What version of Avisynth and MVTools are you using? The script was written under AviSynth+ r2508, and the latest Pinterf's updated MVtools and Masktools. Are you using any custom options for blocksize? The auto blocksize calculation was from Mysteryx's Framerateconverter.

[MysteryX]

that's what i get, any idea?
i can't find the plugin veed anywhere
nevermind, i found modplus(which contain veed i think)

now i get:
manalyse blocks must be 4x4, 8x4, 16x2 blabla
Support for additional block sizes was added in one of Pinterf's latest version of MvTools2.

no, pixel-wise evaluations are just literally, "pixel-wise", u got no access to the neighbor pixels and that renders it much less useful than a dynamic library
I didn't say you could write C++ plugins with mt_lut. I said that anything you write with mt_lut can be written as a plugin.

Ah ok. Makes sense about the banding, I thought that rounding occurs on a per block case causing the edges of the blocks to become pronounced over flat areas that have a gradient. Does the banding occur mid block? As for blocking, do you know if DCTFilter could be used for that, to ascertain block boundaries etc?
I just realized banding mostly occurs for dark and bright scenes because of the 2.2 gamma curve. To preserve details, you would only work on Luma and only on value above/below a certain threshold where the difference between adjacent values is visible to the naked eye, leaving all mid-range values intact.

Banding happens only in specific scenarios:
- dark or bright scenes
- flat areas

If I was to implement a debander, I'd scan Luma horizontally line by line for flat areas that degrade by 1 or 2, and mark the division point between 2 flat areas of adjacent values, and mark the flat areas themselves. Repeat vertically.

Then I'd transform that patterns grid to detect significant zones, discarding detection on single lines. Similar to what I did with stripes detection.

Then, I could apply blurring/dithering/something to soften these edges. Since we're talking about flat Luma areas, there's not really any loss of data. In terms of order of execution, this should happen after denoising.

How does this compare to other debanding methods?


So what are the recommended plugins for each type of defect? In which order should they be run?
- Denoise: MClean is doing good so far
- Dering: I got best results with HQDeringmod.avsi (very complex script)
- Deblock: DCTFilter
- Deband: ?

[MysteryX]

I just tried your script. First version was good. Perhaps it was a lucky shot
https://s21.postimg.org/s2jsjbn5v/MClean1.png (http://postimg.org/image/s2jsjbn5v/)

This version is too sharp for me.
https://s21.postimg.org/hgzx7bgub/MClean2.png (http://postimg.org/image/hgzx7bgub/)

Other version you made me try was too blurry.

[johnmeyer]

I tried it, got the same "veed" error as everyone else, then downloaded ModPlus, but the script crashed right away with the error message: "An out-of-bounds memory access (access violation) occurred in module 'fftw3'...reading address FFFFFFFF."

So, no go here.

Even it I could get it to work, it looks to me like most of the noise reduction is simply using MDegrain in this line from the script:clean = c.MDegrain2 (super, bvec1, fvec1, bvec2, fvec2, thSAD=thSAD, plane = 0)There is also some selective (via mask) sharpening which may, or may not, be a good thing.

So, while I read what you said about your objectives, I am not sure you have created anything that is much different from what already existed.

[MysteryX]

Here's the original version that I got good results with. Less loss of details than with KNLMeansCL.

# MClean basic script
# Mask from bennynihon https://forum.doom9.org/showthread.php?p=1689444#post1689444
# Remaining script by burfadel altered from generic information
# Basics for this script is to remove grain whilst retaining as much information as possible
# The script should also be relatively fast, even without Masktools2 multithreading (disabled due to possible MT bug)
# Chroma is processed via a different method to luma for optimal results
# Requires RGTools, Modplus (Veed, for part of chroma filter), MVTools2, Masktools2, FFT3DFilter


function MClean(clip c, int "thSAD", int "blksize", int "blksizeV", int "overlap", int "overlapV", int "cblksize", int "cblksizeV", int "coverlap", int "coverlapV", int "cpu")
{
thSAD = Default(thSAD, 350) # Denoising threshold
blksize = Default(blksize, 16) # Horizontal block size for luma
blksizeV = Default(blksizeV, blksize) # Vertical block size for luma, default same as horizontal
overlap = Default(overlap, 4) # Block overlap
overlapV = Default(overlapV, overlap) # Overlap for vertical luma blocks, default same as horizontal

cblksize = Default(cblksize, 16) # Horizontal block size for chroma
cblksizeV = Default(cblksizeV, cblksize) # Vertical block size for chroma, default same as horizontal
coverlap = Default(coverlap, cblksize/4) # Overlap for horizontal chroma blocks, default quarter cblksize
coverlapV = Default(coverlapV, cblksizeV/4) # Overlap for vertical chroma blocks, default quarter cblksizeV
cpu = Default(cpu, 4) # Threads for FFT3DFilter


# Masks
LumaMask=mt_binarize(c, threshold=64, upper=true).greyscale().BilinearResize((c.width/16)*2, (c.height/16)*2).BilinearResize(c.width,c.height).mt_binarize(threshold=254)
EdgeMask=mt_edge(c, mode="prewitt",thy1=0,thy2=16).greyscale().mt_binarize(threshold=16, upper=true).BilinearResize((c.width/16)*2, (c.height/16)*2).BilinearResize(c.width,c.height).mt_binarize(threshold=254)
GrainMask=mt_logic(LumaMask,EdgeMask,mode="and")
DegrainMask=GrainMask.mt_invert()

# Chroma filter
filt_chroma=fft3dfilter(veed(c), plane=3, bw=cblksize, bh=cblksizeV, ow=coverlap, oh=coverlapV, bt=5, sharpen=0.5, ncpu=cpu, dehalo=0.2, sigma=2.35)

# Luma Filter
super = c.MSuper(rfilter=4, chroma=false,hpad=16, vpad=16)
bvec2 = MAnalyse(super, chroma=false, isb = true, delta = 2, blksize=blksize, blksizeV=blksizeV, overlap=overlap, overlapV=overlapV, search=5, searchparam=7)
bvec1 = MAnalyse(super, chroma=false, isb = true, delta = 1, blksize=blksize, blksizeV=blksizeV, overlap=overlap, overlapV=overlapV, search=5, searchparam=4)
fvec1 = MAnalyse(super, chroma=false, isb = false, delta = 1, blksize=blksize, blksizeV=blksizeV, overlap=overlap, overlapV=overlapV, search=5, searchparam=4)
fvec2 = MAnalyse(super, chroma=false, isb = false, delta = 2, blksize=blksize, blksizeV=blksizeV, overlap=overlap, overlapV=overlapV, search=5, searchparam=7)
Clean = c.MDegrain2(super, bvec1, fvec1, bvec2, fvec2, thSAD=thSAD, plane = 0)

#Luma mask merge
filt_luma = c.mt_merge(Clean, DegrainMask, U=1, V=1)

# Combining result of luma and chroma cleaning
output = mergechroma(filt_luma,filt_chroma)

return output
}

Now that I look at it though, it looks like SMDegrain, except that it instead uses FF3DFilter for chroma.

[burfadel]

Inevitably all scripts will look similar, the difference is how the results are treated afterwards. Mdegrain is purely temporal, I have detail independent spatial noise reduction added, as well as noise independent sharpening to recover detail, both temporally stabilised. A type of derainbow function can easily be implemented with very little performance cost as an option, once ideal preset settings are worked out. Deringing and dehalo could also be implemented utiising existing masks

@MysteryX, I'll add an adjustment function for the detail sharpening strength :).

Do people find the sharpening too strong for other sources? I'll reduce the defaults to make the sharpening more neutral.

I'll remove veed seeing as it's an Avisynth+ only filter, the Avisynth version is deveed. These can still be run separately. I'll also update the info regarding the need for fftw.

[MysteryX]

There's nothing I hate more than over-sharpened videos. Much better when it is sharp but neutral. It's often a fine line though. Also, sharpening amplifies noise and artifacts.

[burfadel]

There's nothing I hate more than over-sharpened videos. Much better when it is sharp but neutral. It's often a fine line though. Also, sharpening amplifies noise and artifacts.

That's true. That sharpening shouldn't really affect noise, however it could make some forms of artifacts stand out more for now until that part is sorted out :). At the moment the script almost entirely focuses on noise removal with the intention not to remove detail. I'll ease back the sharpening by half, and have it adjustable with a parameter scaled from probably 1/10 of what it is now to a bit more at 100, and have the default set at say, 40. I'll take a look at it shortly and update the first post script :).

[feisty2]

sharpening is not how u gonna magically resurrect the lost details, especially a cheap USM like that
I say you'd better off try some fancy new toys like denoising autoencoder and see how it goes

[feisty2]

also MDeGrain being one of the inter-patch (self-similarity) based approaches, is theoretically equivalent to a sparse coding unit, you can have one or many of those sparse coding layers in your denoising autoencoder and it would reasonably give you results similar to MDeGrain but better

[burfadel]

I've updated the script with a setting for that sharpening amount. This is optioned as 'enh' as it targets detail. A setting of 0 completely disables it. Default setting is 20, which is approximately half of what it was. I realise you can't resurrect lost details, that's not the intention. It's impossible to truly resurrect lost details regardless of how it's done, simply because if it's 'lost' then it's not there to recover. The best you can do is use algorithms to determine what is expected to be there, but that's still not recovering the true detail.

I started looking into other filters because I found KNLMeansCL, as well as other filters, had output that just wasn't as clear as it should be. I then thought of ways to circumvent this, the result being mClean. Another consiration is for it to be pretty fast, suitable for most scenarios, and also keeping things relatively simple. The future intention is to add dehalo, dering, derainbow etc as options, and hopefully deblock as well. Strong enough to be effective but not to reduce wanted detail, whilst still remaining relatively fast.

[feisty2]

quality and performance are on the opposite sides of the tradeoff, you simply can't have them both in general
some denosing filter kills a lot of details along with the noise, some kills less, if you're not happy with what you already got, the healthy choice would be making a new denoising filter that kills less details in the first place, not sharpening what's left of the crap

[burfadel]

That's why the sharpening is only targetting detail, the non-detail is actually being spatially cleaned :). Currently it's an 'adaptive' denoiser, but the intention is for it eventually to be an 'adaptive' noiser by default and an cleaner with the options set.

[GMJCZP]

burfadel, I'm noticing in this version of MClean no longer has this:

# Masks
LumaMask=mt_binarize(c, threshold=64, upper=true).greyscale().BilinearResize((c.width/16)*2, (c.height/16)*2).BilinearResize(c.width,c.height).mt_binarize(threshold=254)
EdgeMask=mt_edge(c, mode="prewitt",thy1=0,thy2=16).greyscale().mt_binarize(threshold=16, upper=true).BilinearResize((c.width/16)*2, (c.height/16)*2).BilinearResize(c.width,c.height).mt_binarize(threshold=254)
GrainMask=mt_logic(LumaMask,EdgeMask,mode="and")
DegrainMask=GrainMask.mt_invert()


Did you have this as planned?

[burfadel]

It meant the denoising was applied too weak on certain parts of the image.

[MysteryX]

I like this version

first MClean version / MClean 1.1 / KNLMeans(D=2, A=2, h=1.5, channels="YUV")

https://s22.postimg.org/gmi7lad59/MClean_Old.png (http://postimg.org/image/gmi7lad59/) https://s22.postimg.org/c20mk3itp/MClean11.png (http://postimg.org/image/c20mk3itp/) https://s22.postimg.org/4w4a3wkct/KNLMeans.png (http://postimg.org/image/4w4a3wkct/)

I like this one (middle) best, and it is notably better than your first version. Default Enh=20 looks good, eh=40 is too much.

[SaurusX]

I like this version

first MClean version / MClean 1.1 / KNLMeans(D=2, A=2, h=1.5, channels="YUV")

https://s22.postimg.org/gmi7lad59/MClean_Old.png (http://postimg.org/image/gmi7lad59/) https://s22.postimg.org/c20mk3itp/MClean11.png (http://postimg.org/image/c20mk3itp/) https://s22.postimg.org/4w4a3wkct/KNLMeans.png (http://postimg.org/image/4w4a3wkct/)

I like this one (middle) best, and it is notably better than your first version. Default Enh=20 looks good, eh=40 is too much.
The middle picture has more blocking, though less ringing around the letters compared the KNLMeansCL. The hand in the air for the woman on the right is better defined in the KNLMeansCL picture.

[MysteryX]

This doesn't solve blocking at all. I'll need to run a separate filter for that. You're right, hand in the air on the right is better defined for KNLMeans. The biggest difference, however, is in the curtains and roof, where KNLMeans discards the subtle details as noise giving a slight plastic or washed out effect.

Note than in my sample, I'm applying upscaling and interpolation after denoising which amplifies the denoising difference. Small variations can however cause large variations in interpolation, explaining why one hand would be clearer while other objects have less details. Sometimes it simply changes the motion estimation so that the hand appears clearer one frame earlier or later. So we really have to look at the whole picture.

[burfadel]

It's hard to compare spatial images when any filter has a temporal component. The temporal stability of motion and detail is something you can't judge in images but can have a huge impact on video quality. I've got another idea that I'll put in this week, many will say it's pointless but it's really simple and some may find it useful.

[MysteryX]

It always goes like this. First people say it's impossible. Then you achieve it anyway. Then some use it.

So don't worry what people say :) Just implement your ideas.

[burfadel]

Updated the script in the first post with a new feature, ReNoise. It's range is from 0 (default, disabled), to 20. This feature allows you to add back some of the luma noise that was removed, which may sound counterintuitive. However, the noise has been temporally cleaned and also spatially modified, so it's not the same as when removed and should have better compressibility. An advantage of doing the modifications to the noise is that it changes only the noise, not also the underlying picture which would be the case if it were applied to the whole picture. There is an option from 0 (disabled), 1 to 10, which adds back 10-100 percent of the modified noise, and 11-20 adds an additional 10 to 100 percent. The modified noise is much weaker, so I thought it would be good to give the option to apply it more strongly :). A setting of 5 is probably a good starting point if you wish to use it.

[MysteryX]

Interesting. As a denoiser before upscaling, renoising is useless.

As a prefilter on HD content... I've encoded with x264 to see the difference in encoding and taken screenshots afterwards. FrameRateConverter is with preset=Normal... preset=Slower just wasn't working with this at all performance-wise.

- RemoveGrain(21) [1.43MB)
- MClean() [1.46MB]
- MClean(rn=5) [1.46MB]
- MClean(rn=10) [1.46MB]
- MClean(rn=10), 16-bit processing [1.57MB]

https://s2.postimg.org/7d1kp9tn9/Enc_Remove_Grain.png (http://postimg.org/image/7d1kp9tn9/) https://s2.postimg.org/cjcq0b8et/Enc_MClean.png (http://postimg.org/image/cjcq0b8et/) https://s2.postimg.org/4rw01r49h/Enc_MClean5.png (http://postimg.org/image/4rw01r49h/) https://s2.postimg.org/800hesqj9/Enc_MClean10.png (http://postimg.org/image/800hesqj9/) https://s2.postimg.org/8qt7kkswl/Enc_MClean16.png (http://postimg.org/image/8qt7kkswl/)

MClean gives a plastic effect as a prefilter, or perhaps it could work with lower settings, but with rn=10 it looks better than RemoveGrain. Note that there are considerable rounding errors in 8-bit. If I do the whole processing in 16-bit, quality is much better. I think MClean should internally work in 16-bit if the source is 8-bit, otherwise there are rounding differences on subtle details applied several times in a row.

For some reason, I'm unable to convert to 16-bit and back to 8-bit for the prefilter.

Pref=last.ConvertBits(16).MClean(rn=10).ConvertBits(8, dither=1)
FrameRateConverter(NewNum=60, NewDen=1, Prefilter=Pref)

MRecalculate: wrong pixel type in FrameRateConverter line 145

In terms of performance, this is using MvTools2 which has MT performance issues. With FRC Preset=Normal which uses DCT=0, the encoding won't start at all, it jams, even though MT normally works with DCT=0. With FRC Preset=Slower (non-MT), it drags extremely slowly.

[MysteryX]

As for running MClean separately instead of as a prefilter, it considerably decrades the quality.

Prefilter / Denoise (both in 16-bit)
https://s2.postimg.org/x4w35yfdh/Clean1.png (http://postimg.org/image/x4w35yfdh/) https://s2.postimg.org/jchoabolx/Clean2.png (http://postimg.org/image/jchoabolx/)

I'm getting best results with MClean(rn=10) in 16-bit, but am unable to use it as a prefilter in 16-bit and can't figure out why. Additionally, I'm unable to get a decent encoding done with FRC Preset=slower combined with MClean. But ultimately, that would give the best.

[burfadel]

I've updated the script again, just note that rn strength may need to be adjusted :). You would think that it would work okay as a prefilter, but I suspect using a tempoeral filter for temporal analysis may be the issue? For the down dither, I'm using dither=0 as it would be better for compressibility and for running before framerateconverter.

[MysteryX]

This would cause frame requests to come in a weird order, and MT is known to have a limitation requiring frame to be requested in the right order.

With ST, however, it shouldn't be an issue.

[burfadel]

The whole script doesn't seem to run as a prefilter though, as the script instructs it to convert back to 8 bits. It's as if the MAnalyse of FrameRateConverter is pulling the MSuper/MAnalyse data directly from mClean, hence mixing 16 bit with 8 bit and the error.

[MysteryX]

What's concerning is that it looks like a bug in the core. Perhaps try AVS 2.6?

Duh, there's no ConvertBits in 2.6, never mind!

[MysteryX]

Interesting bug.

This freezes when opening in VirtualDub. Later on I ran this through the encoder and it went just fine...

file="Video.mp4"
LWLibavVideoSource(file, cache=False)
ConvertBits(16).MClean(rn=10).ConvertBits(8, dither=0)
Prefetch(8)


This works.

ColorBarsHD()
ConvertBits(16).MClean(rn=10).ConvertBits(8, dither=0)
Prefetch(8)


This also works.

file="Video.mp4"
LWLibavVideoSource(file, cache=False)
MClean(rn=10)
Prefetch(8)


Importing in this way still gives "wrong pixel type"

Pref=AviSource("PreviewPref.avs")
FrameRateConverter(NewNum=60, NewDen=1, Prefilter=Pref)


You're however able to run the prefilter as a first pass as an AVI file and then use that as a source for the 2nd pass. Not ideal but at least we can try and compare quality. Then you're also able to use Preset="slower".

[MysteryX]

OK I've done some comparison tests as a prefilter, running MClean in 16-bit and FRC in 8-bit. These tests take longer because I need to encode the prefilter output as an AVI file and then use that interim file. It gives the quality comparison though.

What did you change in this version? Renoise seems softer. Any settings you recommend for better results as a prefilter?

Here I didn't test H264 encoding which may show additional benefits.

RemoveGrain(21) / previous MClean(rn=10) / new MClean(rn=10) / new MClean(rn=12)

https://s2.postimg.org/wem8w0c0l/Remove_Grain.png (http://postimg.org/image/wem8w0c0l/) https://s2.postimg.org/iie0k4frp/MClean_B.png (http://postimg.org/image/iie0k4frp/) https://s2.postimg.org/3npf5y66t/MClean_C.png (http://postimg.org/image/3npf5y66t/) https://s2.postimg.org/lt2dk05p1/MClean_C12.png (https://postimg.org/image/lt2dk05p1/)

The prefilter AVI interm file is 392MB for previous MClean and 400MB for this version, which indicates there are more details.

Performance-wise, MClean in 16-bit encodes into AVI at 6fps on 1080p content. It's a bit slow for a prefilter unless it gives very clear benefits.

Burdafel, you don't want to convert back to 8-bit at the end if the source isn't 8-bit.

[burfadel]

I did adjust the noise alteration, do you prefer the new one or old one? I was thinking about the going back to 8 bits, how do you detect the source bit depth so you can go back to it?

[MysteryX]

how do you detect the source bit depth so you can go back to it?
BitPerComponent

[burfadel]

The non-detail spatial noise reduction has also changed, it's now removegrain (21), was 17. Any of 2, 12, 13, 14, 17, 21 could be suitable due to how it's applied, it's figuring which one is more suitable. I can continue to adjust the renoise feature as well before going on to the next feature.

[MysteryX]

I did adjust the noise alteration, do you prefer the new one or old one?
Honestly, the old one gives a more natural feel.

[MysteryX]

Something else I'm thinking about. Adjusting denoising strength with THSAD doesn't actually change the strength, but rather the width of areas being affected. It's a ON/OFF denoiser where you only select where to draw the line. Renoise allows for this subtle adjustment of strength, on top of allowing for stronger denoising than would normally be acceptable. It thus has at least 2 benefits.

You can also test whether converting Luma to Linear Light makes any difference.

[burfadel]

That true about THSAD :).

I've made test script with two new options for the testing only. To set the renoise variation there's options ver=1 through to ver=6. There is also spatial noise reduction option 'sn' that allows to specify any specific Removegrain mode, default 21, but 2, 4, 7, 9, 10, 17, 18 may also be worth testing. The different 'ver' settings will affect not only still images but how they look in motion, which can make it hard to tell. The intent is to have it look good in both still images and motion, whilst not causing excessive bitrate increase. Test script is YV12/YUY2/RGB32 only.

Separate question, how do you specify a script to return back to the original colour space if conversion occurs during the script? I know you can find the pixel type with the pixteltype() function, but there doesn't seem to be any way to actually use that information? Basically most filters require for instance, a planar format like YV12 or YV24. Ideally if in YV12 it can stay in YV12 (fine), what about RGB32 input? You would convert to YV24 which will (mostly) keep the chroma resolution, and convert back again, but how do you tell the script to automatically convert back to RGB32 or whatever the source was in, without guessing? Sure, you can get the pixeltype, and run the 'if' operator for every colour type based on the pixel format, but that's a considerable number of lines!

Test script for renoise variations and spatial clean type.

Note: Function mCleanT
# mClean Test Scipt
# Not for any other use than to test Renoise variations
# Function is mCleanT, T for Test.

# TEST ONLY!!!
# ver (renoise type) range 1 to 6
# sn (spatial noise reduction) test recommended values 2, 4, 7, 9, 10, 17, 18, 21 (default)


function mCleanT(clip c, int "thSAD", int "blksize", int "blksizeV", int "overlap", int "overlapV", int "enh", int "rn", int "sn", int "ver", int "cpu")
{
defH = Max (C.Height, C.Width/4*3) # Resolution calculation for auto blksize settings
thSAD = Default (thSAD, 450) # Denoising threshold
blksize = Default (blksize, defH<360 ? 8 : defH<750 ? 12 : defH<1200 ? 16 : defH<1600 ? 24 : 32) # Horizontal block size for MDegrain2
blksizeV = Default (blksizeV, blksize) # Vertical block size for MDegrain2, default same as horizontal
overlap = Default (overlap, blksize>4?(blksize/4+1)/2*2:0) # Horizontal block overlap
overlapV = Default (overlapV, blksize>4?(blksizeV/4+1)/2*2:0) # Vertical block overlap
enh = Default (enh, 20) # Detail enhancement (detail orientated sharpen) strength
rn = Default (rn, 0) # ReNoise strength from 0 (disabled) to 20
sn = Default (sn, 21) # Spatial noise type *****For TESTING ONLY*****
ver = Default (ver, 0) # Renoise variation *****For TESTING ONLY*****
cpu = Default (cpu, 4) # Threads for fft3dfilter

Assert(enh>=0 && enh<=102, """mClean: "enh" ranges from 0 to 102""")
Assert(rn>=0 && rn<=20, """mClean: "rn" ranges from 0 to 20""")
bits = bitspercomponent(c)
c = convertbits(c, 16)

# Spatio/temporal chroma noise filter
filt_chroma = fft3dfilter (c, bw=blksize*2, bh=blksizeV*2, ow=overlap*2, oh=overlapV*2, sharpen=0.12, bt=3, ncpu=cpu, dehalo=0.3, sigma=2.35, plane=3)

# Temporal luma noise filter
super = c.MSuper (chroma=false,hpad=16, vpad=16)
bvec2 = MAnalyse (super, chroma=false, isb = true, delta = 2, blksize=blksize, blksizeV=blksizeV, overlap=overlap, overlapV=overlapV, search=5, searchparam=5)
bvec1 = MAnalyse (super, chroma=false, isb = true, delta = 1, blksize=blksize, blksizeV=blksizeV, overlap=overlap, overlapV=overlapV, search=5, searchparam=3)
fvec1 = MAnalyse (super, chroma=false, isb = false, delta = 1, blksize=blksize, blksizeV=blksizeV, overlap=overlap, overlapV=overlapV, search=5, searchparam=3)
fvec2 = MAnalyse (super, chroma=false, isb = false, delta = 2, blksize=blksize, blksizeV=blksizeV, overlap=overlap, overlapV=overlapV, search=5, searchparam=5)
clean = c.MDegrain2 (super, bvec1, fvec1, bvec2, fvec2, thSAD=thSAD, plane = 0)

# Masks for spatial noise reduction and noise independent detail enhancement
noised = mt_makediff (clean, c, u=1, v=1)
noise = mt_binarize (clense(mt_makediff(mt_binarize(noised, u=1, v=1), mt_edge(sharpen(clean, 0.85), "prewitt", u=1, v=1), u=1, v=1), grey=true), u=1, v=1)

# Spatial luma denoising
clean2 = mt_merge (clean, removegrain(clean, sn, modeU=-1, modeV=-1), noise, u=1, v=1)

# Unsharp filter for spatial detail enhancement
clsharp = (enh>0<=100) ? mt_adddiff (mt_makediff(clean, blur(clean, 0.80*(enh/100), 0.50*(enh/100)), u=1, v=1), clean2, u=1, v=1) : clean
clsharp = (enh>=101<=102) ? mt_adddiff (mt_makediff(clean, gblur(clean, enh-100), u=1, v=1), clean2, u=1, v=1) : clsharp

# If selected, combining ReNoise
renoise = (ver==1) ? tweak(clense (blur(noised,1), grey=true), cont=1.010+(0.020*(rn/20)), bright=1.01+(0.04*(rn/20))) : nop
renoise = (ver==2) ? tweak(clense (noised, grey=true), cont=1.010+(0.020*(rn/20)), bright=1.01+(0.04*(rn/20))) : renoise
renoise = (ver==3) ? tweak(temporalsoften (noised, 3, 128, 0, scenechange=0, mode=2), cont=1.010+(0.020*(rn/20)), bright=1.01+(0.04*(rn/20))) : renoise
renoise = (ver==4) ? tweak(temporalsoften (noised, 4, 128, 0, scenechange=0, mode=2), cont=1.010+(0.020*(rn/20)), bright=1.01+(0.04*(rn/20))) : renoise
renoise = (ver==5) ? blur(tweak(temporalsoften (noised, 3, 128, 0, scenechange=0, mode=2), cont=1.010+(0.020*(rn/20)), bright=1.01+(0.04*(rn/20))), 1) : renoise
renoise = (ver==6) ? blur(tweak(temporalsoften (noised, 4, 128, 0, scenechange=0, mode=2), cont=1.010+(0.020*(rn/20)), bright=1.01+(0.04*(rn/20))), 1) : renoise


clean2 = (rn>0<=20) ? mergeluma (clean2, mt_adddiff(clean2, renoise, u=1, v=1), 0.3+(rn*0.035)) : clean2

# Combining spatial detail enhancement with spatial noise reduction using prepared mask
filt_luma = mt_merge (clean2, clsharp, mt_invert(mt_convolution(noise, u=1, v=1), u=1, v=1), u=1, v=1)

# Combining result of luma and chroma cleaning
mergechroma (filt_luma, filt_chroma)

return last.convertbits(bits, dither=1)
}

Renoise 'rn' is now much stronger as a minimum, seeing as it was far too weak in the lower numbers to actually be beneficial, and fractionally stronger effect at rn=20 than the previous rn=20. I can tweak that down slightly as well if found to be necessary!

EDIT: Just changed the noise again for testing :).

[MysteryX]

Sure, you can get the pixeltype, and run the 'if' operator for every colour type based on the pixel format, but that's a considerable number of lines!
That's the way to do it. One line to store PixelType, and one line for conditional conversion back with ? and :

[MysteryX]

mClean on its own doesn't work with AVS+ MT. Prefetch(2) works, Prefetch(4) crashes after a while and Prefetch(8) freezes on startup. (with 16-bit processing)

Even without MT, it doesn't work as a prefilter for FrameRateConverter, for some strange reason.

I was thinking of using Avisynth Virtual File System to feed the prefilter into the FrameRateConverter script, but AVFS doesn't work with Windows 10 x64, the folder C:\volumes stays empty.

So many bugs!!

As it stands, mClean isn't fast. It's too heavy to use as a 1080p prefilter.

[MysteryX]

Just how necessary is 16-bit processing? Here are some comparison images (not using the latest version but the other one before)

Video: AOA - Like a Cat (https://www.youtube.com/watch?v=qEYOyZVWlzs)

- Original
- mClean(rn=12) # 8-bit
- ConvertBits(16).mClean(rn=12).ConvertBits(8, dither=0)
- KNLMeansCL(D=2, A=2, h=1.4, device_type="GPU")
- ConvertToYV24().KNLMeansCL(D=2, A=2, h=1.4, device_type="GPU", channels="YUV")

https://s2.postimg.org/59kb8hgb9/Original.png (http://postimg.org/image/59kb8hgb9/) https://s2.postimg.org/i4ww8zkyd/m_Clean8.png (https://postimg.org/image/i4ww8zkyd/) https://s2.postimg.org/5c8s929cl/m_Clean16.png (http://postimg.org/image/5c8s929cl/) https://s2.postimg.org/3ryuwxbkl/KNL12.png (http://postimg.org/image/3ryuwxbkl/) https://s2.postimg.org/m8t9nqrit/KNL24.png (http://postimg.org/image/m8t9nqrit/)

KNLMeansCL tends to give a plastic effect when applied on YUV planes. There is considerable difference between mClean 8 or 16 but it's subtle details.

ff3dfilter is simply applied on chroma plane and then merged back, it doesn't need to be convert to 16-bit. I did a test converting to 16-bit AFTER MDegrain2 and then converting back to 8-bit when merging luma and chroma. This will be considerably faster. Quality is good.
https://s2.postimg.org/6tu8kme39/m_Clean_Conv.png (https://postimg.org/image/6tu8kme39/)

As for bit conversions, only convert to 16-bit and back if source is 8-bit

This modified version still crashes with Prefetch(8) saying "out of memory" on startup, so there's a bug that's not in MvTools2. I still can't use it as a prefilter with FRC.

Since your method works only on Luma and applies a simple FF3DFilter on chroma, it would be fair to compare with other filters with FF3DFilter on chroma.

KnlMeansCL with FF3DFilter. This one doesn't look very good.
https://s2.postimg.org/flag6dc5x/KNL-ff3d.png (http://postimg.org/image/flag6dc5x/)

With the changes converting to 16-bit only after MDegrain2, performance is good enough. Then there can be an option whether to dither back to 8-bit or stay in 16-bit. Since this runs at the beginning of the script, it makes sense to take a 8-bit input with 16-bit output.

FPS (min | max | average): 2.348 | 70862 | 9.560
Memory usage (phys | virt): 1264 | 1381 MiB
Thread count: 34
CPU usage (average): 49%

Of the various denoiser I'm testing, I still like this one better, although it's not finished.

[burfadel]

I'll update the script tonight with that info (9.30 am here), thanks! Do you have a preference in terms of the renoise 'version' or spatial removegrain method 'sn' from the test script? There are very subtle differences.