Detecting the amount of noise

[MysteryX]

Is there a way to detect the amount of noise in a frame? Either that, or get the average luma of the 85% darkest pixels on the frame.

The idea is to run 2 clips of KNLMeans with different 'h' parameter, and selecting between them based on the noise threshold or luma threshold.

[zorr]

Is there a way to detect the amount of noise in a frame?

A crude but surprisingly effective way is to run an edge detector -> threshold -> AverageLuma. The edge detector should be a sensitive one, such as VapourSynth's g41.EdgeDetect(clip, mode='TEdge') (not sure which filter represents that on AviSynth side).

Either that, or get the average luma of the 85% darkest pixels on the frame.

If you mean the darkest selection of pixels covering at least 85% of the screen then you need a luma histogram for that. I think a histogram would be very useful for many algorithms so hopefully someone implements a histogram plugin which writes the histogram data as an array frameProp.

[kedautinh12]

Here:
http://avisynth.nl/index.php/External_filters#Edge_Detection

[MysteryX]

A crude but surprisingly effective way is to run an edge detector -> threshold -> AverageLuma. The edge detector should be a sensitive one, such as VapourSynth's g41.EdgeDetect(clip, mode='TEdge') (not sure which filter represents that on AviSynth side).

It detects all the moving trees and barely detects the noisy areas. If I select only values below 50, there are still way too much everywhere and not in noisy areas.

[MysteryX]

DUH! My code is already doing noise-diff on several denoising passes.

Noise-diff... HELLO!??

Now... mt_makediff gives a mask with values around 128... how do I calculate the amount of noise from that? Need to calculate the amount of deviation from the center. Noise is up and down so AverageLuma will remain 128.

Is this formula correct, or is there a better way to calculate the noise from a noise mask? (*5 is just to make it visual for testing)


clean = core.std.Expr([core.std.Expr(noise, "x 128 -"), core.std.Expr(noise.std.Invert(), "x 128 -")], "x y + 5 *").std.PlaneStats().text.FrameProps(scale=4)


EDIT: the inverted noise mask is wrong.

then I can store this AverageLuma as frame property "Noise" and re-use it on the next denoising pass.

[Julek]

Instead of using mt_makediff and AverageLuma, why not just LumaDifference?

[MysteryX]

Instead of using mt_makediff and AverageLuma, why not just LumaDifference?
LumaDifference would work in Avisynth. What about VapourSynth?

[Julek]

LumaDifference would work in Avisynth. What about VapourSynth?

core.std.PlaneStats(clipa, clipb)

[kedautinh12]

LumaDifference would work in Avisynth. What about VapourSynth?

You can use PlaneCompare to replace LumaDifferenc
https://github.com/AmusementClub/mvsfunc/blob/03a9e588f7af9ad823c03c9a1bea7da35de3f150/mvsfunc.py#L1361

Example scripts:
https://github.com/WolframRhodium/muvsfunc/blob/8ba1436e6ff87598dea759e25f11bcf222a43ba8/muvsfunc.py#L5441

[MysteryX]

OK I'm surprised... MVTools2 and BM3D don't necessarily remove more noise on dark noisy scenes than on lighter scenes, so the noise mask isn't a reliable way to detect noisy scenes... In fact, the removal of dark noise may count less than the loss of bright textures with MVTools2. And BM3D is just very constant in what it removes.

Other approach: getting average luma of the 85% darkest pixels? Or measuring how many pixels above 50% luma, and if lower than 25% of the frame, calculate AverageLuma of the remaining pixels.

Here's the VapourSynth code that I used to calculate the noise from noise_diff as a frame property.


def WriteNoiseStat(n, f: list[vs.VideoFrame]):
noise = f[0].props["PlaneStatsAverage"] if "PlaneStatsAverage" in f[0].props else 0
f_out = f[1].copy()
f_out.props["xClean_Noise"] = noise
return f_out
noise_stats = core.std.Expr(noise_diff.fmtc.bitdepth(bits=8, fulls=True, fulld=True, dmode=1), "x 128 - abs").std.PlaneStats()
clean2 = clean2.std.ModifyFrame([noise_stats, clean2], WriteNoiseStat)

[StainlessS]

average luma of the 85% darkest pixels
How does this sound ?

Find luma level that partitions 85% [Y85=YPlaneMin(threshold=85.0)]
[EDIT: YPlaneMedian() is implemented as YPlaneMin(threshold=50.0) ]

Create mask using Masktools to set luma >= Y85 [EDIT: Should be <= Y85 ] to pure white, and use RT_Stats Masked function for average luma.

YPlaneMin [or 8 bit only RT_YPlaneMin]


RT_YPlaneMin(clip c,int "n"=current_frame,int "delta"=0,int "x"=0,int "y"=0,int "w"=0,int "h"=0,bool "interlaced"=false,
float "threshold"=0.0, int "Matrix"=(Width>1100||Height>600?3:2),clip "mask"=NOT_USED,int "MaskMin"=128,"MaskMax"=255)
Return int -1, if no valid pixels in Mask clip.
Returns int value minimum luma (0 -> 255) in frame(n+delta) for area x,y,w,h.
Threshold is a percentage, stating how many percent of the pixels are allowed below minimum (ignore extreme pixels ie noise).
Threshold is % of valid mask pixels processed if Mask supplied. The threshold is optional and defaults to 0.0.


Actually, I suppose you could cut out the mask making thingy altogether and use the frame luma itself as the mask,

Again, 8 bit only.

RT_AverageLuma(clip c,int "n"=current_frame,int "delta"=0,int "x"=0,int "y"=0,int "w"=0,int "h"=0,bool "interlaced"=false,
int "Matrix"=(Width>1100||Height>600?3:2),clip "mask"=NOT_USED,int "MaskMin"=128,"MaskMax"=255)
Return int -1, if no valid pixels in Mask clip.
Returns FLOAT value average luma (0.0 -> 255.0) in frame(n+delta) for area x,y,w,h.

Mask=SourceClip MaskMin=0 MaskMax=Y85



****************************************
******* MASKED Luma Y Functions ********
****************************************

Compile time/runtime functions, Planar, YUY2, RGB24 & RGB32. (RGB internally converted to YUV-Y).

The compiletime/runtime clip functions share some common characteristics:-

The 'n' arg is an optional frame number and defaults to 'current_frame' if not specified.
The x,y,w,h, coords specify the source rectangle under scrutiny and are specified as for Crop(), the default 0,0,0,0 is full frame.
If 'interlaced' is true, then every other line is ommited from the scan, so if eg y=1, then scanlines 1,3,5,7 etc are scanned,
if eg y=4 then scanlines 4,6,8,10 etc are scanned. The 'h' coord specifies the full height scan ie same whether interlaced is true
or false, although it will not matter if the 'h' coord is specified as eg odd or even, internally the height 'h' is reduced by 1
when interlaced=true and 'h' is even.

Matrix: Conversion matrix for conversion of RGB to YUV-Y Luma. 0=REC601 : 1=REC709 : 2 = PC601 : 3 = PC709,
Default = (Width > 1100 OR Height>600) then 3(PC709) else 2(PC601). YUV not used

The optional mask clip, governs which pixels are processed by the functions. Where a luma pixel in selected area of the the mask clip is
in range "MaskMin" to "MaskMax" inclusive, then those pixels will be processed. The Mask must also be Planar but not
necessarily the same colorspace as clip c, also must be same dimensions and have at least the same number of frames as clip c.
Calling without mask clip OR with MaskMin=0,MaskMax=255 will effectively ignore the mask and scan full x,y,w,h area.


EDIT: A few more peculiar ones that might be of use one day.


RT_YPlaneStdev(clip c,int "n"=current_frame,int "delta"=0,int "x"=0,int "y"=0,int "w"=0,int "h"=0,bool "interlaced"=false,
int "Matrix"=(Width>1100||Height>600?3:2),clip "mask"=NOT_USED,int "MaskMin"=128,"MaskMax"=255)
Return int -1, if no valid pixels in Mask clip.
Returns FLOAT value luma Standard Deviation (0.0 -> 255.0) in frame(n+delta) for area x,y,w,h.
Standard Deviation (Changed, from Sample Standard Deviation with Bessels Correction).
http://en.wikipedia.org/wiki/Standard_deviation

***
***
***

RT_YInRange(clip c,int "n"=current_frame,int "delta"=0,int "x"=0,int "y"=0,int "w"=0,int "h"=0,bool "interlaced"=false,
int "lo"=128,int "hi"=lo,int "Matrix"=(Width>1100||Height>600?3:2),clip "mask"=NOT_USED,int "MaskMin"=128,"MaskMax"=255)
Return int -1, if no valid pixels in Mask clip.
Returns FLOAT value (0.0 -> 1.0) being the amount of pixels in the range "lo" to "hi" (inclusive), 1.0 is equivalent to 100%.
Implemented as requested by Martin53 (thankyou), NOTE, differs from other funcs that return range 0.0 to 255.0.
NOTE, lo defaults to 128, "hi" defaults to "lo".

***
***
***

RT_YPNorm(clip c,int "n"=current_frame,int "delta"=0,int "x"=0,int "y"=0,int "w"=0,int "h"=0,bool "interlaced"=false,
float "mu"=0.0,int "d"=1,int "p"=1,int "u"=1,int "Matrix"=(Width>1100||Height>600?3:2),clip "mask"=NOT_USED,int "MaskMin"=128,"MaskMax"=255)
Return int -1 if no valid pixels in Mask clip.
Returns FLOAT value greater or equal to 0.0, being the "Minkowski P-norm" (range depends upon values of 'd' and 'u') for frame(n+delta)
area x,y,w,h.
mu, Float, default 0.0 (0.0 -> 255.0)
d, int, default 1 (1 -> 255) # downscale
p, int, default 1 (1 -> 16) # power
u, int, default 1 (1 -> 255) # final upscale before returning result (experimental)
Formula is: sum_over_pixels[ ((pixel-mu)/d)^p ]^(1/p) * u
or in words: d and u are scaling aids. The differences between the pixel values and mu are scaled, taken to the power of p and added up
over the frame. The sum is taken to the p-th root and finally rescaled.
mu=0, d=1, p=1, u=1 yields the average.
mu=average, d=1, p=2, u=1 yields the standard deviation (uncorrected sample standard deviation).
Implemented as requested by Martin53 (thankyou). *** EXPERIMENTAL ***
http://en.wikipedia.org/wiki/P-norm#The_p-norm_in_finite_dimensions
http://en.wikipedia.org/wiki/Minkowski_distance


EDIT: And theres also this

RT_Ystats(clip c,int "n"=current_frame,int "delta"=0,int "x"=0,int "y"=0,int "w"=0,int "h"=0,bool "interlaced"=false,
float "threshold"=0.0,int "lo"=128,int "hi"=lo,int "flgs"=255,string "prefix"="YS_",
float "mu"=0.0,int "d"=1,int "p"=1,int "u"=1, int "Matrix"=(Width>1100||Height>600?3:2),clip "mask"=NOT_USED,int "MaskMin"=128,"MaskMax"=255)
Returns multiple results as for above single frame Luma sampling functions as Local Variables (prefixed with the prefix string arg).
The args up to "interlaced", are as for all other clip functions, "threshold" used only for "RT_YPlaneMin", "RT_YPlaneMax" and
"RT_YPlaneMinMaxDifference" equivalent routines with same functionality.
"lo" and "hi" are used only with the "RT_YInRange" equivalent routine with same functionality.
"mu" and "d" and "p" and "u" are used only with the "RT_YPNorm" equivalent routine with same functionality.
The new arg "Flgs" selects which results you want returned and the string "Prefix" that is prepended
to the returned Local variable names.
The actual return result is a copy of the flgs arg with any non valid bits reset, ie the Local variables that were set.
Local variables are NOT altered for any function not selected in flgs.
Returns 0 if no pixels found in search area of mask within MaskMin and MaskMax.

Flgs_Bit_Number Add_To_Flgs Equivalent_Function Local_Var_Set_Excluding_Prefix
0 1($01) RT_YPlaneMin() "yMin" (0->255)
1 2($02) RT_YPlaneMax() "yMax" (0->255)
2 4($04) RT_YPlaneMinMaxDifference() "yMinMaxDiff" (0->255)
3 8($08) RT_YPlaneMedian() "yMed" (0->255)
4 16($10) RT_AverageLuma() "yAve" (0.0->255.0)
5 32($20) RT_YPlaneStdev() "yStdev" (0.0->255.0)
6 64($40) RT_YInRange() "yInRng" (0.0->1.0)
7 128($80) RT_YPNorm() "yPNorm" (0.0->??? depends upon d and u)
RT_Ystats() allows you to inquire multiple results simultaneously, with not much more overhead than calling a single individual
routine, however, you should not select sub functions that you dont need as there may be an additional unnecessary overhead.
The Default flgs=255($FF) are all bits set and so sets ALL Local vars at once.
RT_Ystats(flgs=1+2+16) would set Local vars "YS_yMin", "YS_yMax" and "YS_yAve" for full frame current_frame.

In addition to above Local Variables, RT_YStats() sets an int Local variable (where default prefix) of "YS_PixelCount" being
the number of pixels in mask area X,Y,W,H between MaskMin and MaskMax inclusive, or pixels scanned in X,Y,W,H area where mask
not used.
NOTE, If no valid flg bits set (eg $FF00), then returns 0, YS_PixelCount and all other variables remain as before call.
Assuming some valid flg bits, if no valid pixels were found in mask then function returns 0, and only YS_PixelCount would be set
to 0, no other variables are touched (remain as before call, undefined if not previously existing).
Example usage:
ScriptClip("""
got = RT_Ystats(c,mask=Mask,flgs=$10) # AverageLuma
(got != 0) ? RT_debug("AverageLuma = " + String(YS_yAve) + "PixelCount = " + String(YS_PixelCount)) : RT_Debug("NO VALID PIXELS FOUND")
""")

Also has similar-ish funcs for RGB.

EDIT: Damn!, Euromillions £185.5 Millions went to France, what a senseless waste. https://www.cosgan.de/images/midi/traurig/a082.gif