What denoiser is the worst ? Objective test attempt

[Fizick]

What denoiser is the best? It is good, but not-allowed question here at the forum.
Why? Becourse it is subjective, and dependent from your impression and the source type, and the noise type, codec, etc, etc.
Subjective rating is not well defined, and is possible probably by some poll only.
Legal question: what denoiser from the list is the currently better for some given video with given noise value and type, by some objective criterium?
But what compare, and how to compare?
In scentific publications it is not a new question.

Typical procedure:
1. Use some standard test sequence (clean, free of noise).
2. Add given noise amount, with normal distribution.
3. Clean this noisy sequence by denoiser.
4. Compare cleaned result with original sequence, calculate some metric (signal to noise ratio by PSNR, SSIM, VQM criterium)

(This method is also often used in image processing with test images "Lena", etc.)

I suggest use similar method here.
Lets make our own comparison!
(I often use it for my filter development).

Step 1.
There are many standard freely-redistributable test sequences (used in video codec development, etc).
I found and downloaded some of them.
Lets use so called "Flower" sequence. It has quite small file size and has some movement in it.
The original is at
http://bmrc.berkeley.edu/ftp/pub/mul...eg/EncodeData/
First part (frames 0 to 29) is in the file "flower.0-29.tar.z"
Direct link: http://bmrc.berkeley.edu/ftp/pub/mul...wer.0-29.tar.z
This archive file can be unpacked by 7-zip, Winrar or some other archiver tools.
Archive contains image files sflowg.0.yuv to sflowg.29.yuv.
This raw YUV files has I420 format (planar 4:2:0 format similar to YV12, but with swapped U, V planes). Width=352, height=240

(not interlaced).
The file sequences can be open with ImageSequence plugin by WarpEnterprises
(Get it at http://www.avisynth.org/warpenterprises)
and we get normal YV12 video.

rawsequence("h:\flower\sflowg.%d.yuv",width=352,height=240,pixel_type="I420").trim(0,29)

Realy, video is not quite normal (it has colors range from 0 to 255), so lets convert it to recomended TV range (16-235):

source=coloryuv(levels="PC->TV")

Step 2.
Lets add some grain with AddGrain plugin by trbarry (or use modified version AddGrainC by Foxyshadis for YUY2, chroma).
(Get it at http://www.avisynth.org/warpenterprises)
Added noise amount is controlled by first parameter. It is really sigma*sigma (or noise variation),
where sigma is the noise standard deviation.
Let's use only luma (Y) channel for testing.
Lets use sigma=5 for test. It is rather big value, and the noise is visible.
(In scientific articles larger sigmas are usually used, about 10-20).
Noise variation parameter = 5*5=25.

noisy=source.addgrain(25,0,0)

For temporal denoisers, noise must not be the same for all frames, so we can not use seed option of AddgrainC.

Step 3.
Use your favorite denoiser here!
The null-transform denoiser:

denoised=noisy

Of course, I use FFT3DFilter as example.

denoised=noisy.FFT3DFilter(sigma=4.0, bt=4, bw=32, bh=32, ow=16, oh=16)

Step 4.
Lets use SSIM metric for camparison of denoised with original source images. (SSIM is better than PSNR).
Well known SSIM plugin by Lefungus
(Get it at http://www.avisynth.org/warpenterprises).
EDIT: It seems, that various versions of SSIM plugin exists, without source.

SSIM(source,denoised,"nul","nul")

I firstly used "nul" as dummy output files.
Now I use full info:

SSIM(source,denoised,"ssim.txt","ssimA.txt")

Open script in VirtualDubMod.
We get SSIM values typed on frame.
Skip some first frames 0,1,2,3 (for temporal denoisers), lets will consider frame 4.
The values for noisy video (null-transforn denoiser) is:
n=4
Y=0.9343 U=1.0000 V=1.0000
SSIM=0.9474 Wieght=1.0000

In this test, by my first suggestion,
we are interested not in summary SSIM, but in Y (luma) SSIM value (=0.9343) only! Chroma is not changed.
Reported last summary SSIM is average of channels (0.8*Yssim + 0.1*(Ussim + Vssim)).
But since this plugin write summary SSIM (not luma) to output file SSIM.TXT,
we now will use this summary per-frame SSIM for comparing.
So, play scipt in any player, and output files will be created.
File SSIMA.TXT will contain average clip SSIM.

--------------------------------------------------
Full test script:

rawsequence("h:\flower\sflowg.%d.yuv",width=352,height=240,pixel_type="I420").trim(0,29)
coloryuv(levels="PC->TV") 
source=last
noisy=source.addgrain(source,25,0,0)
denoised=noisy#.DENOISER(parameters) # put your real denoiser script here
SSIM(source,denoised,"ssim.txt","ssimA.txt")

Here is result for FFT3DFilter v.1.85.
My best settings is:

denoised=noisy.FFT3DFilter(sigma=4.0, bt=4, bw=32, bh=32, ow=16, oh=16)

FFT3D result:
n=4
Y=0.9794 U=1.0000 V=1.0000
SSIM=0.9835 Wieght=1.0000


DeGrainMedian best script (almost default!):

denoised=noisy.degrainmedian(mode=1,limity=4,limitUV=0)

But result is not so great:
n=4
Y=0.9525
SSIM=0.9620

So, FFT3Dfilter SSIM score is above DegrainMedian.
FF3DFilter is better in this thread.
I will tune Vaguedenoiser, MVDegrain soon.

Please download the test clip "Flower" (link is above), tune your favorite denoiser plugin (or script) and post results!


You may also post speed results (but they are system-dependent).

[tritical]

With the way you are doing it (adding noise to the sequence with addgrain() during each run) wont the noise be different for each test? Sure, it will have the same variance, but it wont be exactly the same for each filter tested, which I think is important for comparison of the results (correct me on this if I'm wrong). It would be nice if we could generate a few lossless noisy versions of the sequence (maybe with noise std of 2.5, 5, and 10) along with a lossless clean version (just to make it a single clip that can be loaded with avisource()) and then everyone could download and use those for the tests... that way the noise would be the same for all. Since the flower sequence is only 30 frames the clips shouldn't be very large (a few MBs), and a fast lossless codec should be fine (lagarith would be my pick). I would be able to make and host the clips if needed.

Also, for fft3dfilter and degrainmedian you posted the result only for frame 4. Why look at only a single frame instead of results over the entire sequence?

Anyways, I definitely think this would be interesting as there have been a lot of new denoisers (both scripted and stand-alone filters) recently

[Ferux]

Why would one use FFT3DFilter if FFT3DGPU does the same, but faster?

[Mug Funky]

hmm. a standard test video is a good idea.

how about burning it to a rewritable DVD, playing it out through analog and recapturing, or dubbing to VHS and capturing, etc. could give a good idea of what people could expect from their various sources.

[flood555]

Originally Posted by Ferux

Why would one use FFT3DFilter if FFT3DGPU does the same, but faster?

One and only reason:Limitation of GPU RAM if script needs too many ...

[Guest]

What denoiser is the best? It is good, but not-allowed question here at the forum.

And since you know that, your violation is intentional. You had no need to mention "best" and could have simply referred to your objective test. Therefore, struck for rule 12.

Do not discuss this matter here, take it to PM or challenge it through proper channels. Violators will get a rule 3 strike.

[Mug Funky]

i believe this is an attempt at an objective test (says in the thread title and several times in the post), so at least in my own understanding "best" is applicable. if the situation is controlled enough, surely one can refer to the measurably better performing denoiser as the "best"?

i'm all for this test though - noise comes in all different forms, and it can't hurt to define what denoisers work best in what situations. this can really help the community, as there's a fair bit of confusion i've observed as to what denoiser to run and when.

one example i toy with often in my own work is the difference between removedirt and degrainmedian:

they both work in a similar manner and accomplish similar goals, but they still have their strengths and weaknesses, for instance removedirt will remove spots (often quite big ones) and stabilise shaky footage to a small extent, but on the flipside it'll spoil the picture if there's a regular shaking going on. degrainmedian doesn't do that, removes lots of grain, but will also leave in spots and not stabilise the motion as much. based on that we can't say what's "best", but if we define our goals rigidly toward spot removal or artefact prevention, suddenly we can validly say one is better than the other in this situation.

to reiterate: i think this thread is useful and should continue, perhaps with some re-wording to fit the forum rules better.

[tritical]

@Fizick
Just to give you an idea of what I was talking about earlier, I created/uploaded a rar file here that includes:

the original source, 3 noisy versions of the source, ssim/yv12-psnr avs filters, and a test script

The original source clip was created using:

rawsequence("c:\flower\sflowg.%d.yuv",width=352,height=240,pixel_type="I420")
coloryuv(levels="PC->TV")

and compressed in vdub (fast recompress) using lagarith 1.3.9 in yv12 mode. The noisy clips were created using:

rawsequence("c:\flower\sflowg.%d.yuv",width=352,height=240,pixel_type="I420")
coloryuv(levels="PC->TV")
addgrainc(25,25,0,0)

where 25 was replaced with 9 and 100 in two of the instances. I used v1.3 of addgrain by foxyshadis, and, again, lagarith was used for lossless compression.

I know that ssim is usually preferred over psnr, but I think it would be interesting to see psnr results as well so I included psnr calculation in the provided script. Anyways, I think that having a single package that people can download, that contains the clips/needed comparison filters/test script would help quite a bit.

@Mug Funky

Originally Posted by Mug Funky

hmm. a standard test video is a good idea.

how about burning it to a rewritable DVD, playing it out through analog and recapturing, or dubbing to VHS and capturing, etc. could give a good idea of what people could expect from their various sources.

Sounds like a good idea, though from Fizick's post I was under the impression that he just wanted to use zero-mean gaussian white noise in this test (since he was using addgrain to add the noise).

[tritical]

Sorry for hijacking your thread Fizick, but I'm gonna be w/o internet for the next 4 days so have to get everything done quick . Some results using the noisy clip with noise variance of 25 from the rar file I posted:

tested filters:

sw-3ddct(NoiseParam=4)  <= sw-3ddct  (not an avisynth filter)

dctfun4b(12)  <= dctfun4b

tnlmeans(h=3.5,bx=1,by=1,sx=3,sy=3,ax=6,ay=6,az=2)  <= tnlmeans

frfun7(lambda=2.0,T=7.5)  <= frfun7

frfun3d(t=7.0)  <= frfun3d

fft3dfilter(sigma=3.8, bt=4, bw=16, bh=16, ow=8, oh=8, plane=0)  <= fft3dfilter

dfttest(sigma=1.87,bsize=16,osize=12,tsr=1,ssr=0,max2dblocks=3)  <= dfttest

vf=noisy.mvanalyse(pel=2,blksize=8,isb=false,idx=1,overlap=4,sharp=2,truemotion=true)
vb=noisy.mvanalyse(pel=2,blksize=8,isb=true,idx=1,overlap=4,sharp=2,truemotion=true)
vf2=noisy.mvanalyse(pel=2,blksize=8,isb=false,idx=1,delta=2,overlap=4,sharp=2,truemotion=true)
interleave(\
mvcompensate(noisy,vf2,idx=1,thSCD1=600)\
, mvcompensate(noisy,vf,idx=1,thSCD1=600)\
, noisy\
, mvcompensate(noisy,vb,idx=1,thSCD1=600))
FFT3DFilter(sigma=3.7, bt=4, bw=8, bh=8, ow=4, oh=4)
denoised=selectevery(4,2)  <= fft3dfilter+emc

dfttest(sigma=1.87,bsize=8,osize=6,tsr=2,ssr=6,max2dblocks=4)  <= dfttest+imc

vf1=noisy.mvanalyse(pel=2,blksize=8,isb=false,idx=1,overlap=4,sharp=2,truemotion=true)
vf2=noisy.mvanalyse(pel=2,blksize=8,isb=false,idx=1,delta=2,overlap=4,sharp=2,truemotion=true)
vb1=noisy.mvanalyse(pel=2,blksize=8,isb=true,idx=1,overlap=4,sharp=2,truemotion=true)
vb2=noisy.mvanalyse(pel=2,blksize=8,isb=true,idx=1,delta=2,overlap=4,sharp=2,truemotion=true)
interleave(\
mvcompensate(noisy,vf2,idx=1,thSCD1=600)\
, mvcompensate(noisy,vf1,idx=1,thSCD1=600)\
, noisy\
, mvcompensate(noisy,vb1,idx=1,thSCD1=600)\
, mvcompensate(noisy,vb2,idx=1,thSCD1=600))
dfttest(sigma=1.87,bsize=8,osize=6,ssr=0,tsr=2,max2dblocks=5)
denoised=selectevery(5,2)  <= dfttest+emc

I tested only luma denoising (chroma from source was merged into the denoised clip before computing metrics). The bold writing above indicates the name I gave to each set of settings for making the graph and listing the results. (emc = external motion compenstation, imc = internal motion compensation... external as in a separate filter and internal as in part of the filter itself).

I optimized the settings for best ssim (or at least tried to), so read what you will into the psnr results.

PSNR:  
name:                        Overall -  (Y-min,   Y-avg,   Y-max)
dfttest+emc:                 39.6374 -  (36.9429, 37.8764, 38.4442)
dfttest+imc:                 39.5612 -  (36.8100, 37.8003, 38.2116)
fft3dfilter+emc:             39.5062 -  (37.1861, 37.7453, 38.2638)
dfttest:                     38.8855 -  (36.1973, 37.1246, 37.3587)
fft3dfilter:                 38.6796 -  (35.8878, 36.9187, 37.2048)
sw-3ddct:                    38.6577 -  (36.5600, 36.8968, 37.1035)
dctfun4b:                    38.0481 -  (36.1463, 36.2872, 36.5296)
frfun7:                      37.8178 -  (35.9513, 36.0569, 36.2783)
frfun3d:                     37.7291 -  (35.8747, 35.9682, 36.1844)
tnlmeans:                    37.2052 -  (34.3804, 35.4443, 36.1713)
noisy clip:                  36.6127 -  (34.8308, 34.8518, 34.8684)

SSIM (frames 0-29):
name:                        Scaled SSIM Value
dfttest+emc:                 91.836
dfttest+imc:                 91.166
fft3dfilter+emc:             91.124
sw-3ddct:                    89.606
dfttest:                     89.263
fft3dfilter:                 88.590
tnlmeans:                    88.172
frfun7:                      86.789
dctfun4b:                    86.694
frfun3d:                     86.640
noisy clip:                  67.518

SSIM (frames 2-27):
name:                        Scaled SSIM Value
dfttest+emc:                 92.052
fft3dfilter+emc:             91.374
dfttest+imc:                 91.318
sw-3ddct:                    89.766
dfttest:                     89.435
fft3dfilter:                 88.906
tnlmeans:                    88.340
frfun7:                      86.835
dctfun4b:                    86.731
frfun3d:                     86.684
noisy clip:                  67.644

Here's a graph of the per-frame SSIM Values. I've scaled the values to the 0-100 range using the same equation that the ssim filter uses when it outputs the overall ssim value (scaled_value = 100*pow(original_value,8)).



After running the tests I noticed that lumimasking for ssim didn't really change anything on this clip (weights for all frames were > 0.999 with both lumimask=1 and 2)... so I just used the results for lumimask=0.

I'll try to add the execution times for each of the tested setting combinations...

[Soulhunter]

*cough*

[Didée]

To not conflict with the rules, rename the thread to perhaps "what denoiser gives the best metrics" (or "the highest") ...
(Though quite I liked the joky intention of the original wording )

About the noise - pure white gaussian noise surely is a typical means for such a comparison. But it is a highly artifical kind of noise in that sense that in practice, one is almost never dealing with such noise. In most cases, the original noise has been malformed by DCT compression, and therefore has another characteristic. In particular, withh AddGrain(x,0,0) there is no low-frequency noise present at all; it's all only high-frequency noise.

I think that either an additional step of compression after noise-adding should be done, like Mug Funky suggested, or the adding of noise should be slightly more complex, in order to cover a wider frequency range.
(quick idea: make a 2nd layer of noise, process it (only the noise layer) with DCTFilter, and apply that one, too. ?)

[Backwoods]

Originally Posted by Didée

...In most cases, the original noise has been malformed by DCT compression, and therefore has another characteristic...

I've been running into this lately and it can make some scenes, in dark areas or flat one color surfaces, look worse degrained/noised than the original grain/noise.

[Soulhunter]

Originally Posted by Backwoods

I've been running into this lately and it can make some scenes, in dark areas or flat one color surfaces, look worse degrained/noised than the original grain/noise.

You talk about... banding? :]


Bye

[Backwoods]

It's a jumbled mess spinning around. It's not like the banding I've seen on TV or some DVDs, but I guess you could consider it banding. I'll post a sample in a minute.

EDIT: Sample



Right click Save As, 1sec HuffYUV

FRFun7(Lambda=3,T=2,Tuv=1)
aWarpSharp(depth=8,cm=4)
Tweak(bright=3.5,sat=1.185)
LimitedSharpenFaster(Lmode=2,Smode=3,strength=282,overshoot=1,wide=true,ss_x=2,ss_y=2)
FastLineDarken(80,200)

On this source I'm getting different degrees of that mess with different denoisers.

[Didée]

No surprise. That's what happens when somewhat stronger grain/noise is processed with just a spatial filter (like here, overpraised frfun.)
Treating such noise with just spatial filtering will kill out only HF noise, and leave you back with lots of LF noise. After encoding *that* with a codec using motion vectors, things will get worse, because ME most likely will find even more motion where should be none. LF noise is big poison for motion search, in particular in "flat" areas where you'll see the effect of erroneous motion vectors most easily.
Told this already several times, seems no-one listens.

[Fizick]

tritical,
thanks for results!
O.K., lets use summary SSIM instead of Y.
Many frames info is important, but the more info, the more confusing.
For example, first 2 and last 1 frame is not full procesed in fft3dfilter for bt=4.
so, average sigma is more informative for trimmed frame range.

I used now for fft:

SSIM(source.trim(2,28),denoised.trim(2,28),"ssim.txt","ssimA.txt",false)

Here is updated per-frame (2 to 28) results for fft3dfilter for noise=25 (with Addgrain, not tritical's avi).

denoised=noisy.FFT3DFilter(sigma=4, bt=4, bw=32, bh=32,
ow=16, oh=16)

SSIM from file (not scaled)
0.982694
0.982774
0.983487
0.982951
0.983048
0.983332
0.983549
0.984140
0.984105
0.983672
0.983977
0.984027
0.984097
0.984429
0.984682
0.984191
0.984057
0.984143
0.984185
0.984713
0.984575
0.984306
0.984170
0.984299
0.984422
0.984390
0.984735

Average SSIM= 87.87 (it is scaled)

by the way, tritical, what version SSIM do you use? my version has lumimasking parameter as bool, not as integer.
i have got if from Lefungus site as SSIM-0.24src.rar, it is the same as at m.f.al site.


Here is results for vagedenoiser v0.35.1.
My best settings (without auxclip):
denoised=noisy.vaguedenoiser(threshold=9, method=3, chromat=0.01, wiener=true, wavelet=3)

0.976896
0.977495
0.977606
0.977716
0.978127
0.978130
0.978188
0.978439
0.978670
0.978279
0.978184
0.978304
0.977856
0.978276
0.978102
0.977974
0.978394
0.978593
0.978511
0.978867
0.978980
0.978637
0.978385
0.978741
0.978427
0.978425
0.978199

Average SSIM= 83.86 (scaled)

BTW, wavelet=3 good results (in wiener mode) was a little surprisingly for me
So, the better second stage wavelet, the better result is.

[Fizick]

May be Foxyshadis can modify AddgrainC (option) to provide same noise for given seed (but different for frames).

[Backwoods]

Originally Posted by Didée

No surprise. That's what happens when somewhat stronger grain/noise is processed with just a spatial filter (like here, overpraised frfun.)
Treating such noise with just spatial filtering will kill out only HF noise, and leave you back with lots of LF noise. After encoding *that* with a codec using motion vectors, things will get worse, because ME most likely will find even more motion where should be none. LF noise is big poison for motion search, in particular in "flat" areas where you'll see the effect of erroneous motion vectors most easily.
Told this already several times, seems no-one listens.

You know I follow your posts well enough, but I hope that snide comment wasn't geared towards me. In my folder I am trying alot of different denoisers on this source and it seems they all do it. Even if you're not fond of FRFUN, it still does a good job on most of the anime sources I have tried it on. Just appartently not this one, which doesn't matter b.c others have done the same. I would guess this is from DCT compression. Unfortunately working from the original negatives or even a BETA tape isn't an option here.

Also I think this is getting off subject of the original thread, maybe this can be split or just forgotten about (or even PM). I wouldn't want to derail.

[foxyshadis]

Oddly enough I've had such a thing kicking around for a while, so I went ahead and uploaded it. seed=<random seed>; constant=true for the behavior seed used to have.

http://foxyshadis.slightlydark.com/random/AddGrainC.zip

I definitely agree that encoding the output with xvid or quenc is the way to go. For simulating dot crawl/rainbowing, try analog capturing a good sequence. (Even as simple as wiring a cable from one computers's output card to another's input.)

[Fizick]

foxyshadis,
thanks, will try.

Ferux,
We waiting fft3dgpu (different modes ?) SSIM results from you