Speed and compression of different denoisers [Archive]

N_F

15th April 2003, 23:47

These are some tests I recently ran on different denoisers I figured I'd share.

Noisy clip: 4000 frames of Sex and the City (NTSC, 4:3)
Clean clip: 4000 frames of Fellowship of the Ring (PAL, 16:9 (2.35))

* = Added on first edit
** = Added on second edit
*** = Last added

Denoiser Time Size

Noisy clip
None 5:33 (100%) 64,0 (100%)
*Unfilter(-5,-5) 5:33 (100%) 56.0 (88%)
*Undot() ´ 5:35 (101%) 61.4 (96%)
temporalsoften(2,3,3,2,6) 5:47 (104%) 62,5 (98%)
Mergechroma(blur(1.3)) 5:48 (105%) 61,4 (96%)
**DNR2() 5:57 (108%) 62.0 (97%)
*TemporalCleaner() 5:57 (108%) 62.0 (97%)
temporalsoften(3,5,5,2,10) 5:59 (108%) 59,8 (93%)
**STMedianFilter(8,15,4,7) 6:04 (109%) 60.3 (94%)
temporalsoften(4,8,8,2,10) 6:04 (109%) 58,0 (91%)
*Deen() 6:08 (111%) 53.3 (83%)
fluxsmooth() 6:11 (111%) 57,4 (90%)
convolution3d("moviehq") 6:14 (112%) 62,2 (97%)
convolution3d("movielq") 6:14 (112%) 59,7 (93%)
**Peachsmoother() 6:28 (117%) 53.7 (84%)
SpaceDust() 6:46 (122%) 54,0 (84%)
****convolution3d("vhsbq") 7:05 (128%) 57.6 (90%)
**SmoothHiQ(7,20,25,230,5) 16:54 (305%) 50.3 (79%)
FaeryDust() 26:17 (474%) 55,6 (87%)
PixieDust() 26:36 (479%) 48,7 (76%)
GoldDust() 26:49 (483%) 40,4 (63%)

Clean clip
None 3:20 (100%) 26,3 (100%)
*Undot() 3:21 (101%) 25.8 (98%)
*Unfilter(-5,-5) 3:22 (101%) 23.5 (89%)
Mergechroma(blur(1.3)) 3:30 (105%) 25,9 (98%)
**DNR2() 3:31 (106%) 26.7 (102%) (!?)
temporalsoften(2,3,3,2,6) 3:34 (107%) 25,5 (97%)
temporalsoften(3,5,5,2,10) 3:42 (111%) 25,1 (95%)
**STMedianFilter(8,15,4,7) 3:43 (112%) 26.2 (100%)
*TemporalCleaner() 3:45 (113%) 25.9 (98%)
temporalsoften(4,8,8,2,10) 3:47 (114%) 24,9 (95%)
fluxsmooth() 3:50 (115%) 24,2 (92%)
*Deen() 3:56 (118%) 22.4 (85%)
convolution3d(movielq) 3:58 (119%) 25,0 (95%)
convolution3d(moviehq) 3:59 (120%) 25,6 (97%)
**Peachsmoother() 4:03 (122%) 24.3 (92%)
SpaceDust() 4:07 (124%) 22,2 (84%)
***convolution3d("vhsbq") 4:23 (132%) 25.1 (95%)
FaeryDust() 12:14 (367%) 23,9 (91%)
GoldDust() 12:45 (383%) 20,5 (78%)
PixieDust() 12:48 (384%) 22,3 (85%)
**SmoothHiQ(7,20,25,230,5) 14:10 (425%) 20.6 (78%)

If there are any conclusions to be drawn someone else will have to draw them for now. I’m having a cold at the moment and I’ve been feeling dizzy all day; my mind is not at its sharpest.

One thing to note though is Convolution3D. It seems to be a common misconception that it’s very slow, but as you can see it’s not.

All tests were run with Avisynth 2.51. Dust with the help of Warpsharp’s LoadPluginEx()

Also, if you’re missing a filter leave its name and setting here and I’ll add it

* Take note of Undot and Unfilter (both by Tom Barry IIRC). How is it possible they hardly affect encoding speed, yet in Unfilter's case (and to much lesser extent Undot) changes the picture's size greatly? It would seem there are hardly any extra calculations done or something like that :confused: Not that I'm complaining... :)

irasnyd

16th April 2003, 00:15

If you've got the time, two more I suggest are deen and undot.

I just use them with the defaults:
deen()
undot()

irasnyd

manono

16th April 2003, 10:58

Hi-

Nice test N_F. I'd like to see how TemporalCleaner() compares to the rest of them.

frodoontop

16th April 2003, 11:38

This test is really helpfull. If you have some time, please look at Unfilter(-5,-5). I use it a lot.

Valky

16th April 2003, 14:46

My conclusion is that there is no point using Fairydust anymore on noisy source, cause it takes about same time than Pixiedust().
Or then this might be caused by Warpsharp’s LoadPluginEx().

Prettz

16th April 2003, 15:30

Nice test, but can you tell us your system specs too, possibly to compare performance with different CPUs (i.e. athlon vs P4)?

PerCIVaL

16th April 2003, 17:06

how about smart smoother HQ ? I use it a lot in virtualdub. you should be able to load it into avs.

Also, I find compression important but quality more so. Smart smoother gives me very good results. So does convolution3d.

N_F

16th April 2003, 23:31

I'll try to add all the filters you've mentioned within a day or two.

@PerCIVaL - I haven't had any success with sshq in avisynth yet, but I'll see if I can get it to work.

@Prettz - Well my basic specs are Athlon 1300, 256 MB mem, Win XP. But I don't think it'll do you any good, there are so many other things that makes a difference

trbarry

16th April 2003, 23:53

If you are doing an exhaustive set of tests you might also want to include STMedianFilter(8,15,4,7). ( www.trbarry.com/Readme_STMedianFilter.txt )

- Tom

JuanC

17th April 2003, 00:06

How about including PeachSmoother ? :) :J

12monkeys

17th April 2003, 11:25

Another vote for PeachSmoother, I use it all the time :)

majerle

17th April 2003, 12:19

ehmmm...

how about quality ? a psnr confrontation could be usefull
you can use psnr4avi ( http://members.xoom.virgilio.it/_XOOM/majerle/psnr4avi.exe )

Thanks

Andres

sh0dan

17th April 2003, 15:13

PSNR doesn't say anything about quality - just as little as compression does. A PSNR test would IMO be a waste of time. ;)

DDogg

17th April 2003, 15:25

N_F, even if it would be a totally subjective judgement, I wonder if you might consider assigning a "quality" factor and then plot a graph in Excel so that time, compression and quality (again, I know that is voodoo) could indicate the general trending of the better combinations?

Probably a real pain in the backside for you :)

Piper

17th April 2003, 16:13

While we're at it, let's not forget DNR2 as Shayne kindly pointed out in: http://forum.doom9.org/showthread.php?s=&threadid=44335

I've been toying with this filter over the past few days, comparing it with Pixiedust. I'm quite impressed - the results between the two filters are very close to each other with DNR2 preserving more detail (IMHO) while running much faster.

N_F

18th April 2003, 00:11

@DDogg - I've been thinking about that, but thing is, I have ~30 clips, and it's damn hard seeing any difference without spending a considereble amount of time comparing them. Still, I'll think about it.

Could anyone tell me what sshiq version to use? There seems to be like 10 different versions made by 10 different people available. I've found one by Sh0dan, but it works in RGB IIRC and involves some kind of vdf inport.

^^-+I4004+-^^

18th April 2003, 01:08

numbers really mean less for this kinda comparisson,and for images,you would have to include at least few settings for each filter (yes,peachsmoother and grapesmoother too...hehe)
so it's a tremendous job
for example;there's a source video screenshot,spliced with screenshot after filtering (split screen),this is cropped heavily,but the frame is picked so it had details and some uniform parts too....(this image portion is retained after cropping too)
for every filter/settings there's one such jpeg (very small)....
(i think somone already did such test..with few denoisers)
but as i said too many filters,combinations,numerus settings for each...tough job

TNM

18th April 2003, 09:24

Originally posted by Piper
While we're at it, let's not forget DNR2 as Shayne kindly pointed out in: http://forum.doom9.org/showthread.php?s=&threadid=44335

I've been toying with this filter over the past few days, comparing it with Pixiedust. I'm quite impressed - the results between the two filters are very close to each other with DNR2 preserving more detail (IMHO) while running much faster.
Can u pls post which setting of this filter can give close result to pixiedust? :)

FuPP

18th April 2003, 10:59

"(i think somone already did such test..with few denoisers)"

It was me :) but pictures showed were not always very relevant (mostly due to a lack of time). Sorry...

"While we're at it, let's not forget DNR2 as Shayne kindly pointed out"

Yes, I'd like to play a little bit with that one. Does anybody have some sample scripts with that filter, to use them as a starting point (and then play with parameters)?

Regards,
FuPP

PerCIVaL

18th April 2003, 16:55

@N_F

you should be able to get the latest version here:
http://shelob.mordor.net/dgraft/hiq/smoothhiq.html

As for vdf import, IIRC there is no avisynth version of the filter yet. The page above has detailed instructions how to correctly import the vdf though.

Thanks for the great tests btw!

Shayne

18th April 2003, 20:41

Very good test thank you

Sort of keeps our visual testing in perspective.

I do think you are right with regard to trbarry filters sort of the swiss army on filters.

Peace

sh0dan

18th April 2003, 20:58

I made a 2.0 version of the SmoothHiQ - it hasn't been updated for a while because there are several better smoothers out there, and I'm currently drafting a new one. Find it via avisynth.org as usual.

@N_F: Great to have all the filters listed here!

crusty

18th April 2003, 21:52

N_F:
* Take note of Undot and Unfilter (both by Tom Barry IIRC). How is it possible they hardly affect encoding, yet in Unfilter's case (and to much lesser extent Undot) changes the picture's size greatly? It would seem there are hardly any extra calculations done or something like that Not that I'm complaining...

The effect of Unfilter is pretty easy to explain:
The high speed is basically because it has the same effect as different resize filters on the clip: Different resize filters make the clip look either softer or sharper. The settings you used make the clip look softer.
This also explains the difference in filesize: a softer looking clip compresses much better.
Also, you're using light settings.
Try Unfilter with (5,5) or (10,10). This will increase sharpness considerably and should increase filesize as well.

Why Undot is so fast: I have no idea :D

Also, you don't mention subjective quality of the clip. My experience is that Pixiedust tends to produce artifacts, at least on fast-motion source.

My favorites at this time are Undot, Unfilter, Fluxsmooth and Peachsmoother. I would like to try Deen but I haven't been able to find a version for 2.5. Anyone?

PerCIVaL

18th April 2003, 22:02

Originally posted by sh0dan
I made a 2.0 version of the SmoothHiQ - it hasn't been updated for a while because there are several better smoothers out there, and I'm currently drafting a new one. Find it via avisynth.org as usual.

@N_F: Great to have all the filters listed here!

there's a 2.11 version on the link I posted, is that yours or a different one?

Dark-Cracker

18th April 2003, 23:05

hi,

@percival
there is in edeen a function that produce aproximatively the same result than smoothHiq. this will perhaps help u waitting sh0dan finish to convert sshiq for 2.5 :) :)

@N_F:
could we have some screenshot or a quality rating to know the quality of the result ?

Ps: nice test, nice work :)

thank u :)
Bye.

Piper

19th April 2003, 00:15

TNM & FuPP

I've been using the sample Shayne provided:
DNR2(8,4,6,3,6,3,"dnr1")
and comparing it with
Pixiedust(limit=3)

To my eyes DNR2 using the above settings retains more detail than does Pixiedust(limit=3). I'll need to spend more time getting to know the settings before I decide whether I'll start using it full time. So far I like what I see and it may just give Convolution3D (my mainstay) a run for its money. We'll see..

In the meantime, here again is the translated readme Shayne provided. I'd be interested to hear what others come up with when experimenting with this filter.

DNR2 - Dynamic Noise Reduction 2 (YUY2/RGB32/YV12 (CS_YV12, CS_I420))
Download DNR2 v0.7 (included Source files)

VirtualDub plugin - dynamic noise reduction (transplantation edition of MMX) by Steven Don and Avery Lee

DNR2 (threshY_hi, threshY_lo, threshU_hi, threshU_lo, threshV_hi, threshV_lo and type)

ThreshY_hi Brightness of the pixel of the frame data where one ago was accumulated (Y) threshold value 1
[ Default = 8 ]

ThreshY_lo Brightness of the pixel of the frame data where one ago was accumulated (Y) threshold value 2
[ Default = threshY_hi / 2 + 1 ]

ThreshU_hi Color difference information U of the pixel of the frame data where one ago was accumulated (the Cb) threshold value 1
[ Default = threshY_hi * 3 / 4 (YUY2/YV12): ThreshY_hi (RGB) ]

ThreshU_lo Color difference information U of the pixel of the frame data where one ago was accumulated (the Cb) threshold value 2
[ Default = threshU_hi / 2 + 1 ]

ThreshV_hi Color difference information V of the pixel of the frame data where one ago was accumulated (the Cr) threshold value 1
[ Default = threshY_hi * 3 / 4 (YUY2/YV12): ThreshY_hi (RGB) ]

ThreshV_lo Color difference information V of the pixel of the frame data where one ago was accumulated (the Cr) threshold value 2
[ Default = threshV_hi / 2 + 1 ]

Type " DNR2 " (or "") [ default ] OR " DNR1 "

In the CPU which supports MMX MMX is used.

Recommendation value 7 of threshold_hi - 12

Furthermore, in case of the RGB32, it corresponds to the respective Y:U:V = R:G:B.

Processing summary
Each point of the present frame and the point of the same position of the DNR processing end frame of one ago are compared,
It increases levels. Whether or not the difference within threshold with processing differs.

Case1: Type= " " OR " dnr2 " (default)
Presently the pixel = current of the frame, it makes the pixel = previous of the frame of one ago when
If |current - previous|
If |current - previous| > threshold_lo

Then new = (2 * current + previous) / 3 //current values to approach
Else new = (current + 3 * previous) / 4 //previous values to approach

Else new = current

Case2: Type= " dnr1 "
Presently the pixel = current of the frame, it makes the pixel = previous of the frame of one ago when
If |current - previous|
If |current - previous| > threshold_lo

Then new = (2 * current + previous) / 3 //current values to approach
Else new = previous
Else new = current

As for the DNR1 of the type and the difference of the DNR2, the DNR2 is high picture quality, but it does not become help of MPEG compression,
It becomes help of the high pressure reduction in regard to the point, the DNR1 MPEG compression.

Sample
DNR2(8,4,6,3,6,3,"dnr1")

N_F

19th April 2003, 00:24

Originally posted by sh0dan
I made a 2.0 version of the SmoothHiQ - it hasn't been updated for a while because there are several better smoothers out there, and I'm currently drafting a new one.

Glad to hear that. Saves me the trouble of learning the settings of another advanced (as opposed to for instance fluxsmooth which only has 2 parameters) filter at this time.

N_F

19th April 2003, 00:35

The original intent of this test was actually 95% aimed at the speed of the filters, I just added the size as an extra bonus.

I don't have time to do any extensive visual tests as I'm graduating from what you'd call high school (I think... 16-18 yrs ?) in ~5 weeks and have a lot of work to do before that. Tell you what, as soon as I'm back in school to a faster connection I'll cut a few seconds from each clip and make them available. Don't know how much good it'll do as a few seconds is rather limited, but it's the best I can do at the time.

I'm glad you found my tests useful/interesting.

trbarry

19th April 2003, 03:24

Why Undot is so fast: I have no idea

Partly cause it's written in assembler but mostly because it doesn't do anything except squash stray dots, single pixels that might stand out. It's just a subset of the STMedianFilter() code.

- Tom

High Speed Dubb

19th April 2003, 07:37

From the numbers in N_F's test, it looks like something other than the speed of the smoother (disk speed, maybe?) is the real limiting factor for the faster smoothers. It just doesn't make sense that Unfilter or Undot would be instantaneous.

You might want to add in an extra computationally intensive step. That could keep the processor busy enough that smoothing will directly affect the speed.

trbarry

19th April 2003, 23:37

I guess I'm assuming in this case we have a mildly noisy clip and we would rather have a softer clip that compresses well. I tested a series of my own filters that I thought might do this, using a 3 minute sequence from the SDTV version of Return of the Jedi, 704x368, 2.35:1, not resized (but should be).

I used the following set of filters, doing the test with fixed quants both with and without all 5 filters.

UnDot() # Cheap, but only marginal help here
STMedianFilter(8,20,4,8) # Remove noise and grain
UnFilter(-40,-40) # Remove sharp edges
FillMargins(0,4,0,6) # Blot any border garbage
DctFilter(1,1,1,.8,.4,0,0,0) # Remove high frequency components

The strong UnFilter() was needed because I'm also using a strong DctFilter() and don't want edge noise, though I'm not sure it had to be that strong. The FillMargins() was needed because I had some garbage at the top and bottom and wanted to keep a multiple of 16. I haven't tested it but FillMargins() is another filter that should run very fast since it only touches the changed bytes and doesn't copy the buffer.

Because I didn't know how the first test was done I tested both with and without the filter pack at both fixed quant=2 and fixed quant=4, with Avisynth 2.5 and Xvid, at Xvid defaults.

But be warned the results are indeed very soft, so this is maybe an extreme case. But it looks like the Q=4 ended up being about 318 kbps. That's maybe not bad for a 704x368 clip, soft but no obvious macro blocks. Though it was mostly not a very high motion sequence.

You can see the space saving benefits of filtering, and DctFilter specifically, tend to diminish as quants go up and quality and size go down anyway.

Denoiser Time Size

Fixed Quant=2
None 2:29 (100%) 24954 (100%)
All 5 Filters 3:30 (141%) 15916 ( 64%)

Fixed Quant=4
None 2:28 (100%) 9042 (100%)
All 5 Filters 3:26 (139%) 6974 ( 77%)

- Tom

N_F

20th April 2003, 00:57

Added 4 new filters.

DNR2 and STMedianFilter doesn't seem work well (or they do, depending on how you look at it) on very clean sources.

If you only look at the numbers (haven't even looked at the actual clip yet) Peachsmoother seems very nice.

SSHiQ doesn't have any default values and I don't have the time to find optimals, so I just looked through the readme and tried following the guidelines. If they seem way wrong, add a "better" settings here and I'll add it.

Both Peachsmoother and SSHiQ were run with the help of Warpsharp's LoadPluginEx() as they are 2.0x filters.

If I've forgotten any filter or used a strange setting, let me know.

High Speed Dubb

20th April 2003, 08:16

N_F,

What kind of processor and memory do you have? The time results will probably depend a fair amount on that.

The Peach results are a little surprising -- I'd have expected larger file sizes, but faster run times (relative to the other filters in the list).

Since these clips are from movies, did they start out in the YV12 format? It could be that the YV12->YUY2->YV12 conversion took up a significant amount of the processing time.

In terms of size, the Peach takes a significant amount of time to get an estimate of the noise. If either of the clips had a lot of motion, it's likely that the smoother wasn't able to get a decent estimate during the clip, and was smoothing too much. (Its starting values are chosen for over the air reception, not for a DVD.) That could explain why it's giving small file sizes.

JuanC

20th April 2003, 09:20

Originally posted by High Speed Dubb
The Peach results are a little surprising -- I'd have expected larger file sizes, but faster run times (relative to the other filters in the list). I agree. I always use Peach for my TV captures, and as far as I remember, it's been considerably faster than c3d or Flux. I haven't looked at compressibility thou, but the image quality is great!. What I always do is selecting a couple of still/"slow" scenes I let it show me its estimates for NoiseLevel and BaseLine so when I finnally encode I mannually set them to avoid "oversmoothing" (and maybe making it faster?).

@HSD: It would be great to have an "autoloadable" version (compiled for AVS 2.5X?) whithout the need for LoadPluginEX(). Thanks, It's a hell of a filter!!;)

N_F

20th April 2003, 23:52

Athlon 1300, 256 MB RAM, Win XP.

Yeah, they were originally in YV12. 4 or 5 of the filters (yours are one of them) in the list have a YV12 --> YUY2 --> YV12 conversion, and on top of that LoadPluginEx(), which I don't know how fast/slow it is.

Most part of the noisy clip is an intro, with many fast scene changes and varieting noise. Perhaps that would explain it?

^^-+I4004+-^^

21st April 2003, 01:51

i'm running peach (and grape)in pure YUY2 environment,and
when i disable it i see no speed-up in processing speed....
that's how fast it is.....(and i believe it's some sort of DScaler port,so it should be fast...)

didn't tried tom's filters,but when i see how peach "learns" what is noise and what's not(after few frames),it brings a smile to my face every time..hehe

/ivo

PS. i might do DNR2 vs. peach/grape visual test on some captures,if there's some confusion...
i believe i tested DNR2 long time ago and it was not so good...back in VDub days.....

N_F

22nd April 2003, 09:15

Originally posted by ^^-+I4004+-^^
i'm running peach (and grape)in pure YUY2 environment,and
when i disable it i see no speed-up in processing speed....
that's how fast it is.....(and i believe it's some sort of DScaler port,so it should be fast...)

didn't tried tom's filters,but when i see how peach "learns" what is noise and what's not(after few frames),it brings a smile to my face every time..hehe

/ivo

PS. i might do DNR2 vs. peach/grape visual test on some captures,if there's some confusion...
i believe i tested DNR2 long time ago and it was not so good...back in VDub days.....

When I read this Peach seems very interesting. I very much liked deen, but perhaps I should try Peach more thoroughly

N_F

22nd April 2003, 09:20

Originally posted by N_F
Tell you what, as soon as I'm back in school to a faster connection I'll cut a few seconds from each clip and make them available. Don't know how much good it'll do as a few seconds is rather limited, but it's the best I can do at the time.

I've changed my mind a bit on this. As there are many clips I would say are quite uninteresting (for whatever reason, doens't mean I think it's bad), I'll cut a longer part from the ~6 most interesting clips. If you're interested in this, say so. If not, don't say anything :) No point in me spending energy on cutting and uploading it if noone wants it.

sh0dan

22nd April 2003, 10:00

Originally posted by PerCIVaL
there's a 2.11 version on the link I posted, is that yours or a different one?

That one (which I also made) is for Virtual Dub and only works in RGB mode.

WarpEnterprises

22nd April 2003, 15:19

About the speed measurements:
you get the real filter speed quite well if you add something like
Crop(0,0,8,8)
at the end of the script. Then there is at least no time spent on transferring data to e.g. VirtualDub.

High Speed Dubb

23rd April 2003, 04:52

Most part of the noisy clip is an intro, with many fast scene changes and varieting noise. Perhaps that would explain it?

Yeah, that could well be the reason for the small files — It probably smoothed those clips too much. (Like any filter which reduces noise, Peach Smoother should also reduce compressed file size somewhat. But that wasn’t a design goal, so I’d be surprised to see it work as well for compression as filters which were meant to reduce file sizes.)

Of course the Peach doesn’t really run instantaneously. From benchmarking with the DScaler version, my estimate is that (with spatial smoothing turned on) it uses about 400 MHz per 60 640x240 fields.

The fastest way to run it is using JuanC’s method — Figure out the baseline and noise level for your video source, then provide those as parameters. That only works if your video source has a constant amount of noise — But that seems to be true for most content.

N_F

23rd April 2003, 09:41

Originally posted by WarpEnterprises
About the speed measurements:
you get the real filter speed quite well if you add something like
Crop(0,0,8,8)
at the end of the script. Then there is at least no time spent on transferring data to e.g. VirtualDub.

I don't understand this. I used a normal script where I loaded the source, cropped, resized and added the filters. Why would I want to add another crop?

... on second thought, won't Crop(0,0,8,8) crop the picture to a very small 8x8 clip from the upper right corner? Was that the point?

... on third thought (getting tired? :)), I just realised that the filter will be applied to the whole picture while a very small part of the picture will be sent to virtualdub, eliminating that time. That's the point! :)

WarpEnterprises

23rd April 2003, 12:30

1 - 2 - 3 - you got it :) (btw first I wrote "clip" instead of "script" which was a little sloppy)

Please tell me the differences you see because I was seeing enourmous amounts of time spent on the transfer to VD.

And maybe you need to add a AssumeFPS(100) or such at the end if you system gets up to realtime playback (VD don't previews faster than the clip-rate !)

Shayne

24th April 2003, 00:32

Originally posted by trbarry
"If you are doing an exhaustive set of tests you might also want to include STMedianFilter(8,15,4,7)."Are these still your recommended start point i thought i read somewhere these are a bit aggressive and STMedianFilter(8,10,4,7) was recommended.

PS have you jumped a pond lately?

N_F

24th April 2003, 12:14

Originally posted by WarpEnterprises
Please tell me the differences you see because I was seeing enourmous amounts of time spent on the transfer to VD.
Is there really any point to this? I mean for me as a regular user I only find the numbers useful in comparesion to others under realistic circumstances.

Wouldn't your suggestion only be of possible interest to the developers themselves, not the 99,9% of the rest of the community?

You can never get rid of the data transfer to VirtualDub (or another program), or can you?

trbarry

24th April 2003, 15:49

Are these still your recommended start point i thought i read somewhere these are a bit aggressive and STMedianFilter(8,10,4,7) was recommended.

The second parameter is very forgiving. It probably wouldn't do much damage even at 100, at least in the YV12 version. A previous release also did a small amount of averaging based upon this parm but now it's a pure median filter.

PS have you jumped a pond lately?

Huh? :confused:

- Tom

WarpEnterprises

25th April 2003, 07:35

The transfer rate to VD or other apps is constant and not dependent of the filter chain.
So of course you can't get rid of it, but if you want to compare the speed of a (single) filter you must try to minimize all other influences.
The total time needed for one frame is something like:

disc_access + mpegsource + YOUR_FILTER + export_to_VD + display_in_VD

Now if the other numbers are much higher than the time for the FILTER you can't measure it reliably.
E.g. disc access could be a big factor if you would use an uncompressed RGB source, export could be a big one if there is some slow colorspace conversion involved.

So it's well applicable to the general user.

High Speed Dubb

25th April 2003, 22:12

Yeah, WarpEnterprises’ suggestion sounds like a good idea to me. A measurement of the total encoding time is only useful to someone with the same configuration (processor speed, memory speed, and disk speed) and running the same script.

With a crop command at the end, you’ll be able to measure the processing time for the filter (mostly) independent of your disk speed and the script. (It’ll still depend on the processor and memory speed, but you can’t get around that.) That will make it easier for people with different computers and different scripts to use your benchmarks, and to compare those values with their own.

It’ll also avoid the misleading time results toward the top of the table. Your question in another thread tells me you’ve been confused by this, yourself —
1) Would you say it's a good idea using undot() in every script one uses (as it doesn't affect speed)?

Any filter which processes all of a frame will have a measureable affect on speed. (Just accessing that much memory takes a significant amount of time — I’d estimate about 120 MCycles for 60 640x240 fields on an Athlon 1.2 GHz machine with SDR memory.) The reason it isn’t showing up in the test is that some other step which is being run in parallel — I’d guess the time to write the result to the disk — Is taking longer than that. The difference will show up once you add in some more video processing steps.

PS: I just remembered — Filter execution time is also going to depend on its order in the script, because of the way memory cache works. So even WarpEnterprises’ test method will have somewhat inconsistent results.

N_F

26th April 2003, 00:50

Ok, I think I understand what you guys are saying now. Basically, I should try to remove as many variables as possible, correct?

Isn't it possible many of the filters are optimized in some way for a reasonable resolution? Isn't it possible I'll get strange results from some filters with just a 8x8 picture?

I will try your suggestion on a few appropiate filters. If I get identical numbers (percentage wise, as the speed will be much faster) I'll leave it as it is. If I get different numbers I'll try to update the list.

Does that sound ok?

High Speed Dubb

26th April 2003, 01:33

Isn't it possible I'll get strange results from some filters with just a 8x8 picture?

The compression results will be messed up, but the speed measurement should be fine. I think that’s why WarpEnterprises suggested that the Crop run at the end of the script.

Does that sound ok?
;) Yes, certainly. The original tests were okay, too. You did notice something very interesting, there — That if your processing steps are faster than the output steps, then they are effectively free.

I mostly just wanted to squish the idea that any filter (which processes whole frames) can run instantaneously.

trbarry

26th April 2003, 02:03

I mostly just wanted to squish the idea that any filter (which processes whole frames) can run instantaneously.

Of course it can't (though I like to pretend ;) )

But it also depends upon what you are trying to optimize. If N_F's jobs are typical of his tests, he was considering adding just one single filter, and he wanted to minimize how much longer the job would run (I do) then the tests were probably pretty appropriate.

For instance I started checking out Deen for the first time based upon the above results.

And even before the above tests I'd gotten in the habit of throwing in UnDot() in almost all my scripts. Now I usually add UnFilter() also.

- Tom

Kurosu

26th April 2003, 02:06

I've attached a plugin that outputs as a debug string (check txt file included) the execution time for a filter chain block (both average and current duration).

[edit]
Attachment deleted because of some weird bug (time was measured in fact between start of frame n and stop of frame n+1)

Shayne

26th April 2003, 16:07

I have to say undot and unfilter are also in all of my scripts lately. It would be great to have these incorportated into AviSynth.

@trbarry

Sorry for the English but for some reason i thought you where from England now i see USA?

PerCIVaL

26th April 2003, 17:10

[QUOTE]Originally posted by Dark-Cracker
[B]hi,

@percival
there is in edeen a function that produce aproximatively the same result than smoothHiq. this will perhaps help u waitting sh0dan finish to convert sshiq for 2.5 :) :)

thanks Dark-Cracker,

but edeen doesn't look as good as smart smoother to me. I've played around with the settings but I keep getting terrible smearing.

I also get 'value too big' errors for the wrong values.

Beta I guess

trbarry

26th April 2003, 18:37

I have to say undot and unfilter are also in all of my scripts lately. It would be great to have these incorportated into AviSynth.

They are probably both stable enough that I would have no objection. But UnDot supports only YV12 and neither supports RGB. I don't have any current plans for changing these so I'm not sure what policy is for embedding them with only partial support.

And I'm don't know if there is any real advantage anyway over just throwing them in your plugins folder. But it would be fine with me of others wanted to.

- Tom (who actually speaks only bad English and pidgin nerd ;) )

N_F

26th April 2003, 23:56

Originally posted by trbarry
For instance I started checking out Deen for the first time based upon the above results.
I'm very glad to hear that. (That my tests were also of some interest to more experienced guys like yourself, High Speed Dubb and WarpEnterprises)

N_F

29th April 2003, 09:22

Ok, here's a few tests where crop(0,0,8,8) is used. The scripts I used looks like this:

Clean source
LoadPlugin("E:\PROGRA~1\GordianKnot\mpeg2dec3.dll")
loadplugin(<FILTER>)
mpeg2source("D:\RIP\clean\FotR.d2v")
trim(0,4000)
crop(4,78,712,420)
BicubicResize(640,272)
<FILTER>
crop(0,0,8,8)

Noisy source
LoadPlugin("E:\PROGRA~1\GordianKnot\mpeg2dec3.dll")
loadplugin(<FILTER>)
mpeg2source("D:\RIP\noisy\SatC - 101 - Sex and the City.d2v")
Telecide().Decimate(5)
trim(5000,9000) #At the time I'm not sure it was a good idea using trim after the IVTC, but I wrote
crop(4,0,710,480) #the script several weeks ago and at the time I obviously did
BicubicResize(512,384)
<FILTER>
crop(0,0,8,8)

For fun I also decided to change the codec and its settings. Previously I used DivX 5.03; quant 2 & B-frames. For this test I used XviD (Koepi's latest); quant 4; Search precision 5; H.263; VHQ 0 & no special features.

As you can see it's more optimized for speed now. To the speed (size will be omitted as it's a non-issue)...

Filter Time

Clean source
None 01:41 (100%)
Unfilter(-5,-5) 01:49 (108%)
Undot() 01:51 (110%)
Deen() 02:24 (143%)
Pixiedust() 11:05 (654%)

Noisy source
None 03:40 (100%)
Undot() 03:46 (103%)
Unfilter(-5,-5) 03:47 (103%)
Deen() 04:25 (120%)
Pixiedust() 25:04 (687%)

I'm not sure what to say about the numbers... They do show that all filters do take some more time (of course...). Still, my first tests in this thread were made with a pretty average computer and are IMO more interesting.

Perhaps Warpenterprises or High Speed Dub or someone else will add some useful comments, I have none. :)

BTW, High Speed Dub: As I understand it, your filter will not work well in a comp. test, correct? Perhaps if one specify the Baseline and Noise Level?

High Speed Dubb

1st May 2003, 01:23

The run time results for Peach Smoother should be fine using the default options. To evaluate picture quality, though, you’d either need to specify Baseline and NoiseLevel or run a clip which has parts which aren’t moving. (It’s something of an inconvenience to users of the filter, but I don’t think there’s any way around it if you want to infer the noise level of the video.)

It’s not worth redoing the tests for it (you’ve already tested an awful lot of stuff), but I think the very best way to measure processor time for an extention would be using Kurosu’s filter (once it’s working well) without any daemons running in the background. That would remove the effect of both the read and the write time for the video clip.

sh0dan

16th September 2003, 21:16

VagueDenoiser is missing! I've foundl it to be truly excellent at ermoving noise. Other than that it has become quite fast recently!

Shayne

17th September 2003, 00:36

Would you care to share any of ur setting on different sources the insight would be great

Piper

17th September 2003, 14:01

Yes, please do. Once I know a little more about VagueDenoiser I'll subject it to my own comparison tests that I'm currently doing with deen, peach & c3d on analogue sources.

N_F

18th September 2003, 20:01

Yay! I got my own very first sticky :)

Next step: Moderator. Of course, I've probably written less than 10 posts the last 4-5 months, so that might be hard :D

Wilbert

18th September 2003, 20:23

Do you have time to do the next round?

Including:
- VagueDenoiser
- NoMoSmooth
- a combo script: QMF (for example the one at http://www.avisynth.org/index.php?page=ConditionalFilter, commenting out the resizers)
- MipSmooth

Kurosu

18th September 2003, 20:37

<ad mode=shameless>
SmoothUV (http://kurosu.inforezo.org/avs/Smooth/index.html)?
</ad>

[EDIT]
And soon, VagueDenoiser 0.27.0 :D

N_F

18th September 2003, 22:08

Sorry, but there's no way I have the time to do anything similar within a reasonable time frame. Just went off to college a few weeks ago and I just have so much school work to do. That together with the fact that I'm on a 10 mbit line nowadays pretty much fills up my day :D

Wilbert

18th September 2003, 22:56

Ok, I think I will put something together next week :)

North2Polaris

24th September 2003, 03:10

Originally posted by sh0dan
VagueDenoiser is missing! I've foundl it to be truly excellent at ermoving noise. Other than that it has become quite fast recently!

VagueDenoiser 0.27 can be found at:

http://perso.wanadoo.fr/reservoir/avisynth.html

Wilbert

24th September 2003, 16:15

:) What Sh0dan meant was that VagueDenoiser is missing in the comparison.

North2Polaris

24th September 2003, 22:23

Thanks for the clarification.

shredness

2nd November 2003, 02:18

I've been trying to strike a balance between speed, quality and compressibility, so I've been watching this thread and it's cousin (denoisers and quality comparison), with avid interest. I've found something very curious however and I'm wodernig about your collective opinions.

I ran the following 2 scripts on a clip of the GQ Awards on the SPIKE network in the US. After encoding them both with DivX 5.05, not only did I get the exact same files size, but to my eye they looked the same!

In other words:

undot().deen("a3d",10,12)

gives me the same file size (and apparently same visual quality) as:

undot().unfilter(-5,-5)

I realize that without a sample to post up here, you folks cannot respond to my particular clip, but I was wondering perhaps one or some of few had tried something similar. Any thoughts?

N_F

2nd November 2003, 15:54

You mean they give you the exact same filesize? My guess would in that case be that the unfilter and deen filters haven't been used for some reason.

Try using just undot() without unfilter or deen and see if you reach the exact same filesize again.

I'm not entirely sure in what (if any) way

undot().deen("a3d",10,12)

differs from

undot()
deen("a3d",10,12)

Wilbert

2nd November 2003, 20:23

My guess would in that case be that the unfilter and deen filters haven't been used for some reason.
Not necessarily. Have a look at the cousin thread (600 kpbs CBR):

Undot() ´ 1:40 12.148.736 73.96
Mergechroma(blur(1.3)) 1:42 12.148.736 73.92
Fluxsmooth(1,1) 2:09 12.148.736 74.13
TemporalSoften(3,5,5,10,2) 2:11 12.148.736 73.97
Convolution3d("movielq") 2:18 12.148.736 73.88 # YV12 version
VagueDenoiser(0.8, 1, 6, true) 4:15 12.148.736 73.85
NoMoSmooth() 4:27 12.148.736 74.02 # converted to YUY2

STMedianFilter(5,15,4,7) 1:52 12.150.784 73.19
STMedianFilter(8,15,4,7) 1:52 12.150.784 72.66
Peach(30, 40, false) 1:56 12.150.784 71.98
Fluxsmooth() 2:06 12.150.784 73.80

How about that? Seven smoothers which result in the exact same file size! Coincidence? I have no idea, but it's spooky ...

Sulik

2nd November 2003, 21:09

If you use CBR, the file size will pretty much be the same no matter what the source is.
The fact that the size is *exactly* the same might mean that either:
- filters have no or little effect
- the source is simple towards the end, and the encoder is using padding

When doing such comparisons, it's always better to use constant quantization:

- want to compare quality: use constant file size (CBR)
- want to compare compression efficiency: use constant quantization

N_F

2nd November 2003, 21:11

Hmm... Perhaps I fail to understand something here.

You're encoding bitrate based, which means the codec will try to reach this specific size. In this case perhaps that specific size the codec tried to reach was 12.148.736, which it did for many of the encodes.

I assumed shredness used quality based encoding.

In my view and experience talking about filesize is quite irrelevant when encoding bitrate based.

Wilbert

2nd November 2003, 23:45

In my view and experience talking about filesize is quite irrelevant when encoding bitrate based.

Sure.

You're encoding bitrate based, which means the codec will try to reach this specific size.
Of course!

The clip is 160 seconds at 600 kbps (XviD) => 96000 kbit = 96.000.000 bits = 12.000.000 bytes.

SILICON

20th April 2004, 21:42

I made test with several denoisers.
I used a 15 minutes sample of film "die hard", 2.35:1 aspect ratio, don´t anamorfic. Medium quality source ripped of DVD.

I compresed with CCE 2.66.01.07 (demo version) with Bitrate min = 300, max = 2350, Quality = 30, MPEG2 (CVD). All CCE filters on off.
The version of denoisers are the last at 8/02/2004 in the YUV12 flavour if are avaliable.
I used Avisynth 2.54
The CPU as AMD K7 2200+

3D FILTER SIZE(BYTES) SPEED (FPS)
----------------- ----------- -------
GoldDust() 82.272.040 0.61 (LoadPluginEx2)
PixieDust() 91.717.800 0.61 (LoadPluginEx2)
MipSmooth(preset="movieHQ") 93.159.424 1.06
Deen() 93.301.800 1.27
MipSmooth(preset="movieHQ2") 93.693.380 1.10
FluxSmooth(7,7) 96.275.124 1.31
NoMoSmooth() 96.349.288 0.90 (YUY2)
PeachSmoother() 97.484.800 1.22 (YUY2)
STMedianFilter(8,15,4,7) 97.995.892 1.35
Convolution3d(preset="movieHQ") 98.369.668 1.26
NONE 99.940.388 1.43

Temporal FILTER SIZE(BYTES) SPEED (FPS)
----------------- ----------- -------
FaeryDust() 94.209.460 0.61 (LoadPluginEx2)
TemporalSoften(4,4,8,15,2) 95.322.092 1.22
atc() 95.340.012 1.19
TemporalCleaner() 96.688.972 1.29
GrapeSmoother() 97.582.316 1.37 (YUY2)
cnr2() 99.940.388 1.24 (No size change ¿Broken?)
antiflicker() 104.109.728 1.23 (YUY2)(More size)

Spatial FILTER SIZE(BYTES) SPEED (FPS)
----------------- ----------- -------
SmoothHiQ(7,20,25,230,5) 90.603.520 0.44 (YUY2)
MSmooth() 93.319.796 0.62
SpaceDust() 93.457.196 1.22 (LoadPluginEx2)
VagueDenoiser() 94.060.176 1.06
Unfilter(5,5) 95.039.200 1.39
deen("a2d",2,10,12) 95.358.448 1.27
DNR2() 96.240.152 1.40
DctFilter(1,1,1,1,1,1,.5,0) 98.769.368 1.38
UnDot() 99.201.016 1.39
Mergechroma(blur(1.3)) 99.274.728 1.34
Blockbuster(method="blur") 100.005.980 1.37 (Increase size)

SansGrip

28th July 2004, 15:00

Interesting comparison. Since this is a sticky, I'd be curious to see how the new version of FluxSmooth (both FluxSmoothST and FluxSmoothT) stacks up against the old in your tests. Hopefully you still have the test material... ;)

trbarry

8th September 2004, 12:56

SILICON -

Be aware that UnFilter(5,5) as you used it will actually sharpen a bit, not soften or filter. If you want a noise filter and better compression then you might instead want negative parm values, like UnFilter(-5,-5). My own preference is to use UnFilter() that way following either UnDot() or STMedianFilter().

- Tom

jsquare

11th September 2004, 17:40

My favorite so far for low bitrate encodes:
Undot().Unfilter(-5,-5).FluxSmoothT().Unfilter(-5,-5).Undot()

Chainmax

24th September 2004, 02:28

It would be interesting to see personal opinions on how much detail each option removes. I'm personally looking for a good filter/filterchain that can remove as much blocking as possible while preserving as much detail as possible.

Piper

24th September 2004, 14:24

For analogue sources, my favourite denoising sequence is as follows:

CNR2()
UnDot()
RemoveDirt()
# Resize if req'd - Lanczos4Resize(352,480)
Blockbuster(method="noise",variance=0.3,seed=1)

I found it when doing a search for something totally unrelated to what I was looking for at the time. A quick search again failed to find it, so unfortunately I don't know who to give credit to, sorry.

It strikes a very good balance between removing noise while retaining details, esp on my own TV caps. I've not yet done any compressibility tests with it though.

k0r0n4

27th November 2004, 09:24

thanks Dark-Cracker,

but edeen doesn't look as good as smart smoother to me. I've played around with the settings but I keep getting terrible smearing.

I also get 'value too big' errors for the wrong values.

Beta I guess
Hrm, I havn't had trouble with edeen, but then again I probably wasn't using the same version as you were. Might wanna give it another try if you still have the clips. Also, I'd always start with the default settings and work from there on.

Just remember, you have to be careful with it, but you probably already know that :D

trolltuning

28th February 2005, 19:00

As I understand the original tests you took mpeg2 ran the filters and then saved to avi in Virtual Dub. Would you expect the length of time to be porportional on all the filters when encoding AVI to mpeg2 or will some of them be vastly different?

SCIF

4th March 2005, 08:45

Where from i can download the latest version of Dust(GoldDust , PixieDust..)?

Wilbert

4th March 2005, 11:28

http://www.avisynth.org/warpenterprises/

There's no newer version.

Raziel6969

29th June 2005, 23:28

Hello folks,

hqdn3d(), i like this for anime stuff, i use it to clean before put 2 anime movies in AVI to DVD5 :P
I'm sure there are many new filters to update the list!
Like Fizick tools, FFT3D, Tritical tools, RemoveDirt, RemoveGrain.

Bye.

superdump

27th November 2005, 18:09

I have a request, for everyone's sake. Could people make suggestions as to current denoisers and light/medium/heavy denoising settings so that this thread can be updated?

I don't mind eating some of my cpu cycles so that this thread may become current again. :)

chrisjim

28th November 2005, 00:34

Scintilla

12th December 2005, 23:04

I don't remember the settings used, but recently I've been fond of SmoothD, HQDN3D, and the LRemoveDust(_YV12) script function.

TBilateral can produce very nice results as well, but with any substantial radius, it's INSANELY slow...

(I should mention that I work almost exclusively with anime footage)

plomon

28th January 2006, 03:28

temporalsoften(2,3,3,2,6) 3:34 (107%) 25,5 (97%)
temporalsoften(3,5,5,2,10) 3:42 (111%) 25,1 (95%)
**STMedianFilter(8,15,4,7) 3:43 (112%) 26.2 (100%)
*TemporalCleaner() 3:45 (113%) 25.9 (98%)
temporalsoften(4,8,8,2,10) 3:47 (114%) 24,9 (95%)

Should the order of filters be constant. I mean can I change the order in which the filters are being used.

In the above description, u have used temporalsoften() 3 times. Please explain why 3 times, and what is the reason behind using them intermittently after the usage of certain other filters like STMedianFilter & TemporalCleaner.

foxyshadis

28th January 2006, 03:48

That's not a script, that's just a list, ordered by speed. Using all those together would turn your video to mush. :p The three TS instances are comparing different results of using different arguments.

Boulder

28th January 2006, 10:16

Be careful if you test the parameters yourself, they appear to be incorrect in the table in the first post of this thread. That is, scenechange threshold and mode are mixed up.

plomon

29th January 2006, 04:58

Eventually learning this script and using it to eliminate the existing artifacts is becoming very tough for me.

Is there any complete guide(apart from AVISynth Documentation) which describes the best settings for each of the filters and which explains their purpose and usage in an easy manner.

foxyshadis

29th January 2006, 05:07

most of the denoisers are third-party and can be downloaded from http://www.avisynth.org/warpenterprises/ , and most of those include a readme which explains what they do, how to use them, and usually provides sample use cases. (Docs for many 3rd party filters are also included with Avisynth.) Once the revamp of the wiki is complete that will also be a direct resource.

The reason they have parameters at all is that there is no best setting, every source is unique, but the defaults are tuned to be the "best" for the most sources.

If you need help beyond that, searching this forum for a filter name is guaranteed to find you more information on how to use it.

frednerk

28th February 2006, 10:11

Like superdump and chrisjim, I'm willing to run tests with a view to posting (or linking to, if that's the right thing to do) images showing before/after comparisons of the effects of updated filters and various settings, eg on (a) minor noise (b) moderate noise (c) heavy noise such as poor VHS capture. Sort of like an updated Wilbert's http://www.geocities.com/wilbertdijkhof/analog_comp/comparison.htm done in Feb 2004, by the looks. http://forum.doom9.org/showthread.php?p=791089#post791089

I know The reason they have parameters at all is that there is no best setting, every source is unique, but the defaults are tuned to be the "best" for the most sources is very true, however a visual comparison using suggested "optimal" settings for "standard test sources" might be handy for some. And incredibles like Didee and scharfis_brain and others are always coming up with some amazing new thing to try.

If I could find free web space I'd also link to before/after clips. I'm not sure where to look for links to smallish clips which people may suggest fall into each of these categories (and which ideally require no other filtering)... any suggestions (people seem to have done many tests so I guess some "standards" for testing could exist) ?

Voodoochild

15th May 2006, 09:30

Hi I tried lot's filters including the new and fast removedirt() , which by now became a function not a dll no more.
the problem is almost none of the filters I tried, has the quality of hqdn3d().
the new removedirt() is fast but hide to much details sometimes... I am using hqdn3d(1.5) for DVD movies.
I would like to know if you know of any other filters with same quality but faster..
10x in advance.

WorBry

15th May 2006, 10:42

the problem is almost none of the filters I tried, has the quality of hqdn3d().

From my experience, I would agree with this statement for denoising clean-to-slightly-noisy live DV footage also (interlaced and deinterlaced). As for speed, well that's somewhat of a relative term as it depends on what other 'rate limiting' filters you might have in your processing chain as well as your final rendered format. From my own tests, hqdn3d(3) is pretty much on par with Deen("a3d",1,3,4,3) in terms of relative processing speed and compression boost. I usually use it in combination with (and before) RemoveGrain(2). LRemoveDust_YV12(17,2) alone smoothes a bit too much for my taste.

I have yet to find a 'faster' denoiser (or, more correctly, a denoiser setting) that achieves the precison of hqdn3d, and would be interested to know if you do.

FRFun, a fractal denoising filter, seems to be attracting attention of late, but I've yet to do comparative tests myself.

http://forum.doom9.org/showthread.php?t=110200

I cant see any of the new/experimental motion-compensated denoisers matching a good temporal-spatial denoiser (combo) for speed, but would be delighted if they eventually do.

One other thing you might try is the 'denoise3d' filter in the FFDShow decoder (blur & DN) options, from which hqdn3d was adapted. For interests sake, I'll do some tests myself when I have time.

Voodoochild

15th May 2006, 12:32

FRFun, a fractal denoising filter, seems to be attracting attention of late, but I've yet to do comparative tests myself...

tried it ... with virtualdub to see how it work , took 100% of my cpu, and the movie jumps all the time cause of high cpu ...

I never thought of combining it with different denoisers since, I use it in DVD-rb to backup dvd's and the more plugins you use, the more time it takes. Did Dragon -Tiger with hqdn3d(2), result were great! though took 3 hours :-)

10x Elad.

WorBry

15th May 2006, 14:39

Dragon-Tiger - that would probably be anime right? Cant say I have any experience there. For DVD back-ups and conversion (TMPGenc) of my PAL DV home movies to DVD for TV viewing, I rarely use more than RemoveGrain(2) anyway, as mild noise is usually less apparent than loss of detail.

For PC viewing however, I usually deinterlace and convert my DV's to DivX (sometimes XviD). In this case I'm usually aiming for relatively high quality (Quant 2.7 - 3.0) with no B-frames at a comfortable bitrate (around 3300 -3600 kbps at full 720x544 res). For me, thats where a precise Temporal-Spatial denoiser (combo) makes the difference. I wouldnt combine more than two filters though. Whilst some multiple-filter denoiser functions give quite pleasing results and appear to work wonders with noisy video, they usually involve some sharpening in an attempt to recover lost detail, but at the expense of some compressibility....and, of course, speed.

Voodoochild

16th May 2006, 08:48

as you told me, I tried removegrain(2), removegrain(5) also.
removegrain(2) a bit more drastic then hqdn3d(2).. How ever
I discovered removegrain(5) is almost identical to hqdn3d(2) but consume way less cpu usage below 50% !, while hqdn3d consume between 80-90%... I'll give it some tests and report..
10x Elad

WorBry

16th May 2006, 14:21

I'm surprised that you find RemoveGrain(2) alone ''a bit more drastic than hqdn3d(2)'' as with my DV sources I have always found it to be quite mild. Indeed, the documentation cites mode=2 (default) as relatively ''risk free'' in terms of preservation of thin line detail and avoidance of artifacts, which are more or less features of the higher, more agressive denoise modalities. Mode=1 basically functions like Undot() in that it it is primarily intended for removal of stray hot pixels. For sure, mode=5 is said to rank somewhere between the two so it is perhaps not suprising that you are observing a milder effect than mode=2. Ironically though, with my deinterlaced DV sources I found mode=5 to exert a significantly lower compression boost than mode=1 when testing RemoveGrain in the absence of other denoisers. Strange huh? Evidently, the behaviour of this filter (and probably other denoisers too) appears to depend somewhat on the nature of the source material, which in turn presumably reflects differences in the prevalence of different types of noise. Other, more informed authorities might be able to comment on this further?

One thing to bear in mind though is that RemoveGrain is a purely spatial denoise filter and modes 1-10, 17 and 18 are primarily intended for use with progressive material. As such, the success of your tests with DVD sources might well be dependant on whether you are dealing with purely progressive, interlaced or hybrid video. If there is any possiblity of interlaced content I would definitely recommend applying the filter to the separated fields and then re-weaveing - but you probably know that already?

Actually, RemoveGrain can also be made to act as a temporal denoiser using the following function:

function RemoveTempGrain(clip input, int _mode)
{
rg = RemoveGrain(input, mode=_mode)
return TemporalRepair(rg, input)
}

You need to load the Repair.dll provided with RemoveGrain for this.

I tested RemoveTempGrain(2) with my deinterlaced DV material but found no noticable improvement and slightly larger output file sizes (lower compression) than with RemoveGrain(2) alone. I have no idea how it behaves with DVD sources. Why not give it a try with mode=5.

My limited knowledge of denoisers is hereby exhausted.

Voodoochild

16th May 2006, 18:03

My limited knowledge of denoisers is hereby exhausted

Wow... I'm impressed your "limited knowledge" is not so limited :-).. it's more then mine that for sure..
I have all the plugins i need i'll give your function a try with mode 5, asap. :thanks: so much Elad.

i'll even upload sample images that show the comparision between the options..

WorBry

17th May 2006, 07:00

The RemoveTemporalGrain function is nothing new - it's all there in the filter documentation. Check-out also the RemoveGrain thread in this forum:

http://forum.doom9.org/showthread.php?s=&threadid=79751

Another one to try is DeGrainMedian, a Spatio-Temporal denoiser:

http://forum.doom9.org/showthread.php?s=&threadid=80834

Good luck with your tests.

Your_Idol

13th September 2006, 03:56

Well, inspired by this thread i did some tests myself.

I was wanting to run a filter over a clean video in an attempt to make it compress better without any noticeable difference in quality from the source.

I used 9 different filters and made 9 different videos from the one clean mpeg2 source. And I compared the results very carefully for many hours, looking for differences. Then I did it again and again with different sources just to make sure.

I have attempted to attach the results to this post with a word document (Compression and quality of different filters.doc)

I concluded that I would use TemporalSoften(3,5,5,18,2) because it scored high with good compression and high with retaining quality and it’s fast.

I also tried to use DNR2, Dust and CondTemporalMedian. I downloaded them from warpenterprises but the files did not come with a good readme about how to use it and I couldn’t figure them out.

I posted this here because I thought it might help people decide which denosier to use. I know I would have liked to find a post like this to save me a lot of time testing.

Didée

14th September 2006, 13:38

I concluded that I would use TemporalSoften(3,5,5,18,2) because it scored high with good compression and high with retaining quality and it’s fast.
It's not done with spending lots of time with looking at lots of tests with lots of samples.
The trick is to look at the *right* samples - i.e. those that show you something.

You concluded TemporalSoften is "good quality, no detail loss (blur)" ?? Well, have a look at this case example (http://s000.tinyupload.com/index.php?file_id=26172320586343465698) (tinyupload). I'd say, in this case TemporalSoften gives a pretty bad result with lots of smearing and loss of detail. And this isn't even a too drastic example (have only very few samples on this PC here). I've seen cases where the results were much more desastreous!

TemporalSoften() is a quite "generic" (read: "dumb") filter. Just because you use it with low thresholds, it doesn't mean it's safe. The only really safe way is to set the thresholds as low as (0,0,0...), i.e. to not use TemporalSoften at all. :)

Your_Idol

15th September 2006, 08:40

I did my test for me, i thought some people might want to know about the stuff i found out.

I liked my test and i am sticking with the results.

if you disagree please do your own test and post the results because people want to know.

Of course if size does not matter don’t use any filters.

Scintilla

16th September 2006, 19:43

Of source if size does not matter don’t use any filters.
Actually, even then, I'm sure a lot of people would still use filters designed to correct problems such as rainbowing, dot crawl, etc.

Bh4i

5th October 2006, 19:39

Removegrain seems to be a very popular one (i think..)

Why isnt it included in this test?

unskinnyboy

5th October 2006, 19:46

Removegrain seems to be a very popular one (i think..)

Why isnt it included in this test?
Because when N_F did his test, Removegrain didn't exist. Later on in the thread, people do comparatively speak about Removegrain. You only have to read.

Blue_MiSfit

14th April 2008, 23:04

Well, I hate to cast ThreadNecromancy(level=2) ;)

But, I'm thinking it might be time for some updates in this area. There's been a lot of interesting development over the last few years in the denoise / degrain area.

A few that come to mind:
MVDegrain
DeGrainMedian
FRFun
MC_Spuds
SPresso

Also, people seem to be keen on motion compensating denoisers by using MVTools. I've heard fft3dfilter/gpu mentioned several times in this context.

I will be doing some tests over the next few days (using static and high motion clips), and hopefully updating this thread a bit!

I may also investigate the impact of these filters on HD encoding, since this seems to be a new trend on Doom9. Super-great filters like MVDegrain2 seem to be unusably slow on 1080p, but provide incredible results.

More to come.

~MiSfit

moviefan

14th April 2008, 23:11

I'll be waiting for your tests. For me quality is more important than speed, so I would like to have tests about 1080p super-great filtering with incredible results :)

Blue_MiSfit

15th April 2008, 00:14

Oh man.. if you don't care about speed, then try MVDegrain2 on a really grainy HD-DVD or Blu Ray. It's _very_ slow - something on the order of 2-3 seconds per frame single threaded on my Q6600. Maybe 1fps if I MT it, but that uses 100% of CPU. While encoding, who knows!

But yes, good stuff coming.

Here's going to be my methodology:

1) Rip DVD to VOBs, index, and make simple script with MPEG2Source
2) Trim a section, encode to lagarith YV12
3) Make several scripts referencing this Lagarith AVI, and execute various filters
4) Save results as CRF18 x264 with some standardized setting
5) Report various file size differences, and comment on subjective impressions of encoded quality.

I will likely post screenshots, and possibly make the filtered samples available. I welcome input from anyone regarding methods - or suggested sample DVDs. I have so far chosen a section towards the end of Terminator 2: Extreme Edition, which is a 'clean' 2.35:1 anamorphic title. I haven't picked one for a high detail static scene. I will likely pick a third title that's "very bad". We'll see!

--crf 18 --level 4.1 --keyint 240 --min-keyint 24 --ref 6 --mixed-refs --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto
--subme 6 --trellis 1 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto --thread-input --progress --no-psnr --no-ssim --output "output" "input"

Will be my x264 command line (executed through MeGUI)

~MiSfit