Selectively Denoise High/Low Frequencies

[Dali Lama]

Hi everyone,

I was wishing to selectively denoise high and low frequency noise in a source, while preserving middle frequency noise. From my subjective testing on various images, I felt the middle frequencies contained the most detail and also afforded a more natural image, instead of a totally smooth image (I used image-editing programs).

I was wondering if there are AVISynth filters that could do this, probably it would be useful as a wavelet denoiser like vaguedenoiser. I'm pretty sure this can be done with the filters we currently have, but couldn't seem to figure out how to apply it.

Thanks for the help,

Dali

Edit: Ok,
It looks like FFT3DFilter can now differentially denoise frequencies at different sigma (or strength) values. That just might work.

Edit2:
So, it seems that differential denoising strengths based on frequencies works well. Let me explain my experiences a little. To my eyes totally smoothed images are nice at first glance, but when you encode it it gets dithering artifacts especially in the background/static areas. This may be relieved by increasing the bit depth, but that isn't always possible or effective. I started to use the image-denoising program neatimage to clean a static image from the source and realized that the high frequency noise is often troublesome grainy/ringing type noise. On the other hand, the large blocky and smudgey (especially in dark areas) noise is due to low/very low frequency noise. When I removed the middle fequency noise I found that detail that I wanted in the image was gone and that smooth/artificial effect was achieved. Of course I couldn't encode that image.

So here I am with the tool thanks to Fizick and it seems that this idea is very effective (obviously people know about it). However, I havn't read anything on Doom9 stating that this was a very effective way to denoise. Just finished some testing with FFT3DFilter with differential frequency dependant thresholds and the encoded source is very low noise with NO dithering or unwanted artificial smoothness.

Well, just thought I'd let everyone in on my experiences, hope you can find similar or better results.

Thanks Fizick,

Dali

[Dali Lama]

Ok,

It looks like FFT3DFilter can now differentially denoise frequencies at different sigma (or strength) values. That just might work.

[movax]

Wavelet 3 in VagueDenoiser (newer version), Villasenor-Belzer-Liao 6/10, though rarely used and often giving craptacular results on some things, can work true miracles on others. Give that a shot if you haven't already.

[cwk]

Howdy Dali Lama,

As if I have any advice to offer to the Dali Lama... ;)

FFT is a fantastic denoiser. I'm a fan! However I find the multiple sigmas confusing. Here is the way I read it. When using Sigma2, Sigma3, etc, the frequencies between the stated sigmas are denoised at a value based upon linear averaging between the other values. From the FFT doc:

"The pattern coefficients will be created internally from these sigmas values by interpolation."

If I read that correctly and have Sigma2=5 and Sigma3=1, then some intermediate frequency that corresponds to Sigma2.5 will be denoised at a sigma of 3 = ((5+1)/2). What do you think?

Also, in high motion sequences, motion compensation of the previous and next frame seems to allow higer sigmas without oversmoothing. This requires MVTools 0.9.9.1 and can be SLOW. Looking something like this:

clip = last
bvec = MVAnalyse(clip, blksize=4, lambda=1000, delta=1, pel=2, isb=true)
fvec = MVAnalyse(clip, blksize=4, lambda=1000, delta=1, pel=2, isb=false)
backw = MVCompensate(clip, bvec, mode = 1).Deblock(quant=15, aOffset=4, bOffset=4)
forw = MVCompensate(clip, fvec, mode = 1).Deblock(quant=15, aOffset=4, bOffset=4)
interleave(backw,clip,forw)
FFT3DFilter(sigma=3, bh=60, bw=60, oh=30, ow=30, wintype=1, plane=0, bt=3)
SelectEvery(3,1)

I basically pulled this out of a post from Heini011 here, substituting FFT for FluxSmooth:

http://forum.doom9.org/showthread.php?t=84770&page=5

DeGrainMedian also responds well to motion compensation.

ck

[Dali Lama]

Hi cwk,

Motion compensation using MVTools seems like a good idea to improve the denoising. I will give it a try. The only thing about your posted script is that deblock is being used too, which can be a pretty aggressive deblocking algorithm coming from h.264 post-processing. Maybe I'll try it without that.

Interesting you mentioned DeGrainMedian too, because when I use that as a pre-processor before FFT3D it helps to remove even more grain. Man, Fizick is making some good filters isn't he?

And your idea about the sigmas was exactly what I was wondering. I have a feeling your right, but Fizick would know for sure.

Hi Movax,

Its good to know that Vague has that option now. I will give a go too, but after I thought of Vague, I was hoping for 3D noise reduction with frequency discrimination and then FFT3D poped up in my mind.

-Dali

[Fizick]

BTW, some time ago i create a my almost first complex script for different frequencies denoising.

Function DeGrainPiramid(clip clipYV12, int "limit0", int "limit1", int "limit2", int "mode") {
# Script function for big grain removal with 3-scale image piramid (by Fizick)
# plugin used: MaskTools (by Manao), DeGrainMedian (by Fizick)
# input clip must be YV12
limit0 = default(limit0,3)
limit1 = default(limit1,3)
limit2 = default(limit2,3)
mode= default(mode,1)
a0=clipYV12
w=width(a0)
h=height(a0)
a1=reduceby2(a0)
a2=reduceby2(a1) # lowest
b1=pointresize(a2,w/2,h/2)

b0=pointresize(a1,w,h)
c0=yv12lutxy(a0,b0,"x y - 128 +", "x y - 128 +", "x y - 128 +", y=3,u=3,v=3) # high
c1=yv12lutxy(a1,b1,"x y - 128 +", "x y - 128 +", "x y - 128 +", y=3,u=3,v=3) # middle

c0=c0.degrainmedian(mode=mode, limity=limit0, limituv=limit0) # here we can filter high
c1=c1.degrainmedian(mode=mode, limity=limit1, limituv=limit1) # here we can filter middle
a2=a2.degrainmedian(mode=mode, limity=limit2, limituv=limit2) # here we can filter lowest

e0=pointresize(c1,w,h) # middle upscaled
d0=pointresize(a2,w,h) # lowest upscaled

s0=yv12lutxy(d0, e0,"y x + 128 -", "y x + 128 -", "y x + 128 -", y=3,u=3,v=3) #
res= yv12lutxy(c0, s0, "y x + 128 -", "y x + 128 -", "y x + 128 -", y=3,u=3,v=3) # result
return res
}


aviSource("grain8mm.avi")
i=converttoyv12()
degrainpiramid(i,2,4,3)

[Poutnik]

BTW, some time ago i create a my almost first complex script for different frequencies denoising.

Function DeGrainPiramid(clip clipYV12, int "limit0", int "limit1", int "limit2", int "mode") {
............
c0=yv12lutxy(a0,b0,"x y - 128 +", "x y - 128 +", "x y - 128 +", y=3,u=3,v=3) # high
.......
return res
}
I just learn to use MaskTools. I am confused what is the difference between
yv12lutxy used as above and internal fuction Subtract ?
I think they do the same. Or is speed the reason ?
If I understand Masktools are MMX optimized.
If yv12LutXY is faster then Subtract, does it mean the latter ( and maybe other ) function is not even MMX optimized ?

But I can be completely wrong, as I used to be :)

[Fizick]

I do not remember :) Seems, speed. try yourself :)

[Didée]

Well, I also mostly use custom YV12Lutxy expressions, even when dedicated plugins are available. This is partly because of habit, partly because the speed difference isn't that big (at least in complex scripts, where using LUTs instead of something else perhaps is only 0.1% slower at the end of the day) ... and partly it has *true* reasons. ;)


Example Subtract() :

Subtract works fine, and supposely is a tad faster than YV12Lutxy (never did actually measure the speed difference). Problem with Subtract is that

a) it gives a result that is centered around Y=126 , UV=128, where Y=128 is more usual at least when working with MaskTools' filters, and

b) it clips the luma input to [16,235] , which often makes it unusable when the input actually is a mask already.

So, instead of having to think about IF Subtract can be used in a given case, and HOW to treat its result properly IF it can be used, I always use YV12Lutxy and can concentrate on the problem I want to solve, instead on the darn filters. :)


Example LimitChange(): (from the other thread)

Sure, for simple "clipping-the-effect" one could use LimitChange instead of building a YV12Lutxy expression. Again, LimitChange supposely is a tad faster.
Now, I could argue that IMHO LimitChange puts the two clip arguments backwards - but that's not an argument. ;)

But what I do very often is not to do a "hard clipping", but clipping with some modifications. An action often taken is, say, "apply effect by 50%, but not more than +/- 3". With LUT, this is:

a=last
b=blur(1.58)
yv12lutxy(a,b,"x 5 + y < x 3 + x 5 - y > x 3 - x 50 * y 50 * + 100 /")

With LimitChange, this is not possible to do directly, but would require another additional filtering step.


For these and similar reasons, you'll see me use YV12Lutxy instead of other possible filters very often. It simply is easier for me, and more versatile.

Last not least, YV12Lutxy in a sense is "my" baby ... IIRC it was me who suggested Manao to implement it into the MaskTools :D

[Fizick]

Didée,
Thanks for your baby, I exploit it. :)

[Poutnik]

nice to have such parents among us..... I like yv12lutxy concept, so does RPN,
even I am far to be used to it.....

[Poutnik]

Well, I also mostly use custom YV12Lutxy expressions, even when dedicated plugins are available. This is partly because of habit, partly because the speed difference isn't that big (at least in complex scripts, where using LUTs instead of something else perhaps is only 0.1% slower at the end of the day) ... and partly it has *true* reasons. ;)
thank you for your long post. :goodpost: It is always good to know the way how others think. Even if your are far more experienced and skilled than I am in video processing, our thinking about ways to problem solution are very close.
I have wanted to be sure about yv12lutxy ideas, not to persuade you to do things in other way.
Example Subtract() and LimitChange():
another examples of clearing of Poutnik's mind.