What denoiser is the worst ? Objective test attempt

[shaolin95]

OK, so whos "winning" at the moment?

[tritical]

Some results on the flower clip with noise std of 10 (var 100):

tested filters:

sw-3ddct(NoiseParam=9)  <= sw-3ddct  (not an avisynth filter)

dctfun4b(24.0)  <= dctfun4b

frfun7(lambda=2.0,T=14.125)  <= frfun7

frfun3d(t=13.3)  <= frfun3d

fft3dfilter(sigma=7.9, bt=4, bw=16, bh=16, ow=8, oh=8, plane=0)  <= fft3dfilter

dfttest(sigma=3.9,bsize=16,osize=12,tsr=1,ssr=0,max2dblocks=3)  <= dfttest

vf=noisy.mvanalyse(pel=2,blksize=8,isb=false,idx=1,overlap=4,sharp=2,truemotion=true)
vb=noisy.mvanalyse(pel=2,blksize=8,isb=true,idx=1,overlap=4,sharp=2,truemotion=true)
vf2=noisy.mvanalyse(pel=2,blksize=8,isb=false,idx=1,delta=2,overlap=4,sharp=2,truemotion=true)
interleave(\
mvcompensate(noisy,vf2,idx=1,thSCD1=800)\
, mvcompensate(noisy,vf,idx=1,thSCD1=800)\
, noisy\
, mvcompensate(noisy,vb,idx=1,thSCD1=800))
FFT3DFilter(sigma=7.4, bt=4, bw=8, bh=8, ow=4, oh=4)
denoised=selectevery(4,2)  <= fft3dfilter+emc

vf1=noisy.mvanalyse(pel=2,blksize=8,isb=false,idx=1,overlap=4,sharp=2,truemotion=true)
vf2=noisy.mvanalyse(pel=2,blksize=8,isb=false,idx=1,delta=2,overlap=4,sharp=2,truemotion=true)
vb1=noisy.mvanalyse(pel=2,blksize=8,isb=true,idx=1,overlap=4,sharp=2,truemotion=true)
vb2=noisy.mvanalyse(pel=2,blksize=8,isb=true,idx=1,delta=2,overlap=4,sharp=2,truemotion=true)
interleave(\
mvcompensate(noisy,vf2,idx=1,thSCD1=800)\
, mvcompensate(noisy,vf1,idx=1,thSCD1=800)\
, noisy\
, mvcompensate(noisy,vb1,idx=1,thSCD1=800)\
, mvcompensate(noisy,vb2,idx=1,thSCD1=800))
dfttest(sigma=3.6,bsize=8,osize=6,ssr=0,tsr=2,max2dblocks=5)
denoised=selectevery(5,2)  <= dfttest+emc

I tested only luma denoising (chroma from source was merged into the denoised clip before computing metrics). The bold writing above indicates the name I gave to each set of settings for making the graph and listing the results. (emc = external motion compenstation, imc = internal motion compensation... external as in a separate filter and internal as in part of the filter itself).

I optimized the settings for best ssim (or at least tried to), so read what you will into the psnr results.

PSNR:  
name:                        Overall -  (Y-min,   Y-avg,   Y-max)
dfttest+emc:                 35.2520 -  (32.3217, 33.4911, 34.0736)
fft3dfilter+emc:             34.8975 -  (32.1083, 33.1366, 33.7318)
sw-3ddct:                    34.2174 -  (31.9290, 32.4565, 32.7581)
dfttest:                     33.9319 -  (30.8815, 32.1710, 32.4758)
fft3dfilter:                 33.7370 -  (30.5839, 31.9761, 32.3819)
dctfun4b:                    32.6397 -  (30.6978, 30.8788, 31.1775)
frfun7:                      32.2570 -  (30.3373, 30.4960, 30.7684)
frfun3d:                     32.0432 -  (30.1331, 30.2823, 30.5516)
noisy clip:                  30.2587 -  (28.4802, 28.4978, 28.5098)

SSIM (frames 0-29):
name:                        Scaled SSIM Value
dfttest+emc:                 79.876
fft3dfilter+emc:             77.437
sw-3ddct:                    74.833
dfttest:                     73.102
fft3dfilter:                 71.763
frfun7:                      65.280
frfun3d:                     64.830
dctfun4b:                    64.262
noisy clip:                  30.906

SSIM (frames 2-27):
name:                        Scaled SSIM Value
dfttest+emc:                 80.837
fft3dfilter+emc:             78.402
sw-3ddct:                    75.162
dfttest:                     73.580
fft3dfilter:                 72.700
frfun7:                      65.345
frfun3d:                     64.885
dctfun4b:                    64.319
noisy clip:                  31.098

Here's a graph of the per-frame SSIM Values.



And for those who prefer pictures to numbers, here's a pic of frame 21 showing the original source, the noisy source, and denoised versions.

[foxyshadis]

Is dfttest so far only optimized for uniform gaussian noise, or does it work just as well if you toss your noisy subject video into an encoder at some middling bitrate first?

[tritical]

I've only tested on gaussian noise so far, but I would expect it to perform similarly to fft3dfilter on other types of noise. I am definitely going to start testing on some other sources soon. Would it be more desirable to test on a noisy source generated by:

1.) sequence -> add gaussian noise -> mpeg2

2.) sequence -> mpeg2 (dvd) -> dvd player -> huffyuv capture (composite)

3.) sequence -> add gaussian noise -> mpeg2 (dvd) -> dvd player -> huffyuv capture (composite)

? I am also thinking about switching to a clip that isn't so perfect for block based motion compensation.

[Richard Berg]

I think #2 would be most interesting.

I have a large collection of old (read: noisy) capture cards but no DVD player Best I could contribute would be the same program captured from both a good and bad card.

[Mug Funky]

@ tritical: try old footage from archive.org.

the kind with strong luminance flicker... this'll make block-based motion compensation retch.

[frednerk33]

How about Pookie's http://forum.doom9.org/showthread.ph...906#post865906 ?

backward_vec2 = last.MVAnalyse( isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1 )
backward_vec1 = last.MVAnalyse( isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1 )
forward_vec1 = last.MVAnalyse( isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1 )
forward_vec2 = last.MVAnalyse( isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1 )
a = last.MVDegrain2( backward_vec1, forward_vec1, backward_vec2, forward_vec2, thSAD=400, idx=1 )
b = a.DeGrainMedian( mode=1 )
SeeSaw( a, b, NRlimit=4, NRlimit2=5, Sstr=1.5, Slimit=9, Spower=9, Szp=16, sootheT=80, bias=40 )

Referred to in a visual test at http://forum.doom9.org/showthread.ph...972#post865972

Pretty results from noisy input: http://forum.doom9.org/showthread.ph...872#post865872

[Didée]

"Visually pleasing" and "good metrics" are two different pairs of shoes.

I tried FFT3D+motioncompensation (top-notch combo in this thread) on a grainy DVD source, and the result looked kind of boring ("emptied-out"), while not being able to truly kill low-frequency flicker. OTOH, a custom-made MC'ed temporal averaging looked sharp & detailed, and highly stable: no more LF flicker ... though the same script gives a 2 points lower SSIM (!) on the source of this thread, heh.

Me, I dont care for high SSIMs or PSNRs, if the result is blurry and keeps flickering ...

[tritical]

You're saying that your method looked better on the current test clip then the fft3d+mc method even though it scored 2 points lower on ssim? ... or just that it looked better on the grainy dvd source?

It's not surprising to me that fft3d+mc (and the like) has so far dominated this comparison (ssim/psnr wise) based on removing zero mean gaussian noise from a clip with pretty consistent global motion. It is also not surprising to me that it would not win in other noise removal tests (either different scoring system or different types of noise).

Start rant about metrics:

The thing to remember about ssim/psnr is that they are 'image' quality assessments, so for video they compare single frames and then pretty much just average (in the case of overall psnr it's slightly different) the results of all frames. In the original ssim paper there was a proposal for simple motion weighting (give less weight to frames in areas of large motion), but that doesn't come close to handling a lot of cases. For example, consider two 6 frame sequences with ssim scores of:

80.005 80.005 80.005 80.005 80.005 80.005
(.9725 .9725 .9725 .9725 .9725 .9725)

86.611 72.139 86.611 72.139 86.611 72.139
(.9850 .9600 .9850. 9600 .9825 .9600)

both would get an overall score of: 80.005 even though the first sequence would in all likelihood be ranked higher from a subjective quality perspective. ssim/psnr will also not account for things such as flickering/moving blocks in flat areas between frames, etc...

Even on the single image level, PSNR has been shown to not correspond well with subjective human tests as far as image quality is concerned. SSIM has been shown to be better, but is still not perfect. There are newer versions of ssim (cwssim, wcwssim) that come closer to human rankings of quality for certain types of artifacts (one is low frequency structural noise, such as block mean shift).

Another problem with using these types of metrics is that it assumes that the original source is the optimum. For example, assume you have a source that is noise free, you then add some noise to it, afterwhich you denoise the image using two different filters. One filter perfectly removes the noise and does nothing else. The other removes the noise, but also slightly sharpens, increases saturation, performs aa, or whatever... From a metric perspective the first denoiser is the best, but a person might prefer the look of the second denoiser (due to the fact that they would prefer a slightly modified version of the original vs the original itself).

end rant:

Anyways, I am sure almost everyone here already knows the limitations/issues of using psnr/ssim/etc... metrics. However, it is still interesting to perform these types of tests, if only to see how poorly the metrics compare to human perception (to me at least).

[Soulhunter]

Ok, let's compare temporal filters with temporal filters... ^^

Using tritical's settings for motion compensation on the flower sequence:


Noisy=Source.AddGrain(25,0,0)

Measured on frames 0-30


Denoised=MC.FFT3DFilter(Sigma=3.7,bt=4,bw=8,bh=8,ow=4,oh=4)
> Average SSIM = 91.19802121

Denoised=MC.DCTFun7(Ty=2.9)
> Average SSIM = 92.42250950
> Average SSIM = 92.68476356 [on frames 2-27]

The resul tritical published for DFTTest+EMC
> Average SSIM = 92.052


Noisy=Source.AddGrain(100,0,0)

Measured on frames 2-27


Denoised=MC.FFT3DFilter(Sigma=7.4,bt=4,bw=8,bh=8,ow=4,oh=4)
> Average SSIM = 78.28281343

Denoised=MC.DCTFun7(Ty=2.9*2)
> Average SSIM = 81.90503727

The result tritical published for DFTTest+EMC
> Average SSIM = 80.837


In short...

Some noise :

92.422 for DCTFun7
92.052 for DFTTest+EMC
91.198 for FFT3DFilter

Some more noise :

81.905 for DCTFun7
80.837 for DFTTest+EMC
78.282 for FFT3DFilter

Conclusion: MVTools wins! :]


Trivia about DCTFun7 [is a little bit less XD]

It's an hard thresholded TI 4x4x4 ICT using a multiplierless and branchless core!

[Backwoods]

Do you have a download link for DCTFun7?

[Fizick]

http://forum.doom9.org/showthread.ph...535#post864535

[Soulhunter]

It's coming...

Same goes for DCTFun7! ^^


Bye

[frednerk33]

Is it wrong to post a link to http://forum.doom9.org/showthread.ph...048#post868048 which has visual results but not objective per above ? (I am not smart enough to do the above, but someone else here might be.)

[DryFire]

What if you all used the lossless version of elphant's dream as a starting source? I'd suggest the 640x360 one, unless you just feel like playing with 20gb's worth of hd png's (the is the same source from saggatire's hd comparison test):
http://orange.blender.org/blog/origi...rce-available/

Or some other, clean predefined clip.

[Fizick]

O.K., now I have secret weapon (bt=5) too

interleave(\
mvcompensate(noisy,vf2,idx=1,thSCD1=600)\
, mvcompensate(noisy,vf,idx=1,thSCD1=600)\
, noisy\
, mvcompensate(noisy,vb,idx=1,thSCD1=600)\
, mvcompensate(noisy,vb2,idx=1,thSCD1=600))
FFT3Dfilter(sigma=3.6, bt=5, bw=8, bh=8, ow=4, oh=4)
denoised=selectevery(5,2)

Average SSIM= 91.75

But fft3d is still the worst of secret filters.

[cwk]

O.K., now I have secret weapon (bt=5) too

Should we be expecting FFT3dFilter v 1.8.6??????

[Didée]

Originally Posted by Fizick

O.K., now I have secret weapon (bt=5) too

Pfffft. There was already bt=7 in use.

Well okay, [(bt=4)+(bt=4)]*0.5 is not exactly bt=7 ... just what a poor scripter can do.

[frednerk]

Originally Posted by Fizick

O.K., now I have secret weapon (bt=5) too

Er, bt=5 gives me an error message saying values must be -1,0,1,2,3,4 ... am using FFT3Dfilter Version 1.8.5 - 4 December 2005.

Am using

ConvertToYV12(interlaced=FALSE) 
#http://forum.doom9.org/showthread.php?p=873307#post873307 
#O.K., now I have secret weapon (bt=5) too 
noisy = LAST 
vf1=noisy.mvanalyse(pel=2,blksize=8,isb=false,idx=1,overlap=4,sharp=2,truemotion=true) 
vf2=noisy.mvanalyse(pel=2,blksize=8,isb=false,idx=1,delta=2,overlap=4,sharp=2,truemotion=true) 
vb1=noisy.mvanalyse(pel=2,blksize=8,isb=true,idx=1,overlap=4,sharp=2,truemotion=true) 
vb2=noisy.mvanalyse(pel=2,blksize=8,isb=true,idx=1,delta=2,overlap=4,sharp=2,truemotion=true) 
interleave(\ 
mvcompensate(noisy,vf2,idx=1,thSCD1=600)\ 
, mvcompensate(noisy,vf1,idx=1,thSCD1=600)\ 
, noisy\ 
, mvcompensate(noisy,vb1,idx=1,thSCD1=600)\ 
, mvcompensate(noisy,vb2,idx=1,thSCD1=600)) 
FFT3Dfilter(sigma=3.6, plane=4, bt=5, bw=8, bh=8, ow=4, oh=4) 
selectevery(5,2)

Can you share the secret here ? I won't pass it on.

[Fizick]

v.1.9.2 is coming somedays ...

EDIT:
Released for public.