Automatically varying denoise strength based on the scene?

[bkman]

Hi,

At the moment I am encoding a tricky source with quite some variation in grain strength in different scenes. I am currently applying different strengths of fft3dfilter to trimmed sections of the film, then splicing them together at the end.

So what I was wondering is there might conceivably be some way to handle this automatically with a script, or even withing a denoiser. Perhaps something along the lines of computing the combined entropy of the last sequence of frames, and enabling stronger denoising once it passes a certain threshold?

So what do the denoiser authors think about this idea?

[Mug Funky]

perhaps you could use something like this approach:

1. perform strong denoise (ie what you'd use for the noisiest scene in the source, or just something at max settings).

2. subtract this from the source (mt_lutxy(clean,noisy,expr="x y - abs") or something similar).

3. perform a strong temporal smooth on this new clip, with scenechange threshold set very low indeed, and noise thresholds set very high (ie temporalsoften(5,255,255,1). the "1" might need to be raised a bit).

4. use scriptclip/frameevaluate to crossfade the noisy/clean clips based on averageluma of the "noise mask" clip. the maths here might be a little tricky, but all do-able.

you could also use one of those change-limited denoisers, but it's not exactly the same (i've considered scripting a change limiting function that's arse-backwards from the type used in degrainmedian and dust, but have been too lazy).

[Didée]

Also try PeachSmoother ... it is (or tries to be) adaptive to the actual noiselevel.

[bkman]

Ok, I've attempted your suggestion, MugFunky, but it simply doesn't work very well on account of the difference between clean and noisy frames is similar when a lot of noise has been removed to when a lot of detail has been removed.

I also tried peachsmoother, and the results are mixed. The noise adaptation seems very good, but the picture is less stable than I would like, and seems blurry in motion.

Are there any better options?

[*.mp4 guy]

there are some other options, but they get progressively slower. You could try vaguedenoiser+Degrainmedian, vaguedenoiser has an adaptive mode. You could also try motion compensated denoising with an overall high strength becuase it tends to keep detail intact, but this would be really really slow, you would prolly be better of using trims to set denoising "zones" of different strnegths.

[bkman]

I've been fiddling with this for a whole now, and I've ended up going back to MugFunky's idea, but with a different spin.

I though I would share my findings here, in case anyone is interested.

orig=last
denoised=fft3dfilter(bt=4,sigma=0.8, sigma2=0.4, sigma3=3, sigma4=4, plane=4)
sharp=denoised.limitedsharpenfaster()
diff=mt_lutxy(orig, denoised, expr="x y - abs").temporalsoften(5,255,255,1)
sdiff=mt_lutxy(denoised, sharp, expr="x y - abs") #.temporalsoften(5,255,255,5) #slows it down.
scriptclip("merge(orig, sharp, averageluma(diff)*averageluma(sdiff)/0.3)")
removegrain(mode=1)

Now what I've come up with here, is that I'm partially merging a denoised and sharpened version of the source based on the amount of "noise" that was removed weighed with the amount of detail still left after the denoising, normalized to 0.3 (may depend on the source). If there is not much noise, then not much denoising is done unless there is a lot there to sharpen. If there is noise, but not much detail in the scene, then the noise is left intact. So in theory (and in practice to a certain extent) the final result keeps or sharpens most of the detail while smoothing out noise when it can.

It works fairly well, but does anyone have any suggestions on how to improve on this idea?