Some thoughts on a new denoiser comparison

[FlimsyFeet]

The Sticky at the top of this forum, "Speed and compression of different denoisers", was written in April 2003. There is also a newer article "denoiser comparison for analogue caps" in February 2004.

Now though, these comparisons are out-of-date, as several new filters are available (e.g. FFT3DFilter, LRemoveDust, and something about using an image sequence process) that people say give very good results, but they are not included in these comparisons.

The trouble is, it's very time consuming for one person to take it upon themselves to do all the tests, experiment with each filter, select the appropriate settings, analyse the results and so on.

A member called mg262 once suggested:
"I have thought that a good way of doing this without requiring too much work from any one person would be to run a "blind" comparison -- i.e. to post a source video, have everyone filter it however they liked and send back the scripts, then to post all the versions labelled A, B, C, etc. then have people make comments. But, in practice, I think very few people would send back scripts!"

Well ath the end of the 2004 article Wilbert asks for other scripts to be sent to him, but I don't know if he ever got any?

Anyway, I though that if I could make available 4 "typical" noise test clips as follows:
- Analogue capture from VHS movie
- Analogue capture from laserdisc
- Consumer DV cam in low light
- Clip from a grainy DVD
who would be interested in taking on the "denoiser challenge" and submitting a script for each of these cases? Unfortunately I can't promise any prize money like the football clip guy in the general forum!

And should I also include a clip from 8mm camcorder, for the interlaced video case?

(What about anime? I didn't include it because anime post-processing scripts seems to be rather specialised to each individual project. But I could include a clip from a DVD of an old '80s cartoon, which if I recall was quite a bad transfer (noise and ghosting).)

Now, I may still feel like a noob in the presence of the experts here, but I've been around the Doom9 forums for a while and I reckon I could have a go at doing a new denoiser comparison.

I could make the test clips available by email, or maybe ftp or one of these free file hosting sites. I hope that the usual suspects would submit AVS scripts. Maybe someone could suggest settings for simple inbuilt filters like TemporalSoften to see how they compare with the more advanced scripts. Is it worth opening up the challenge for VirtualDub filters as well? I've used Smart Smoother before and liked the results.

I want to compare denoising filters only, however I think it should be allowable to apply secondary filters to counteract any side-effects of the denoising (e.g. deblocking for pixiedust, or sharpening to restore detail lost by spatial filtering).

I would then encode the scripts to DVD-compliant MPEG-2. I would say that's what most people would be interested in, but of course the requirments for things like XviD might be very different. Where relevant, I would also encode the original clip with no processing for comparison purposes. Something like a 4-pass VBR with CCE, average bitrate 4000kbps. All scripts would be encoded with the same settings.

The primary consideration is final quality. Although time is important, so I'll include comment on the speed of each script in the final summary, but each clip would be assessed on quality. How do you scientifically assess which one looks "best"? (aargh not that word!) Well, I could post a poll for each clip, and ask people to vote (and make comments) on which one they prefer. This would be, as mg262 suggested, blind, so there is no way for the voters to know who each clip belongs to or what filters were used.

After the polls close, I would then reveal all the scripts and who submitted them. Then I'd write a summary of the findings and post a new denoising comparison article. I promise it will be better written than this post, which is really just throwing my thoughts at the screen to see what sticks.

Sorry about the long post. I've asked several questions in the text above, so if you have any ideas, please post. Or if you think I'm going about this in totally the wrong way, I'd welcome any thoughts about what should be done.

[pelle412]

I uploaded a Star Trek TNG VOB sample (no audio) to:

http://www7.rapidupload.com/d.php?file=dl&filepath=6463

It's not too noisy but it's difficult to get it to look good. Feel free to use this in a denoiser comparison. The objective here is not to find a way to IVTC the CGI sequences into a smooth 24fps motion but general denoising.

The vob sample has 2631 frames in it. I'm not sure if providing this is legal or not. If anyone of you know that it isn't, let me know so I can remove it.

[mg262]

A few notes... first, I hadn't discarded the idea completely, but I wanted to wait for the proper release of Didée's LTSMC, which is IMO an indispensable contender, and then leave a a few more months for us to get the hang of setting its parameters correctly.

Second, I don't think that trying to run four categories simultaneously is ideal... it takes a long time to fine tune a script for a source, and very few people are likely to have time to manage four sources. So one animation clip (which I think LTSMC isn't meant for) would be a good place to start.

Third, you don't want the clip to have artefacts that should be removed by a different kind of plug-in (e.g. rainbows, field blending, ghosting). After all, there's no shortage of source material to choose from!

Fourth, you really must evaluate the results of the denoising process independently from the compression process. If you use 4000 kbps MPEG for the output, you may well find that clips that killed more detail end up looking better because they compress better. But the deeper point is that we want to test the quality of the denoisers rather than the peculiarities of any particular compression method. Depending on the length of the source clips -- and incidentally, I was thinking several clips from the same source to provide representativity -- I would say provide lossless versions (as well as compressed versions for those on dial-up, but using something better than 4000 kbps MPEG).

Incidentally, VirtualDub filters can be imported into AVISynth script, and I think we should restrict their use to this method... otherwise it's too fiddly to reproduce.

[MfA]

The objective here is not to find a way to IVTC the CGI sequences into a smooth 24fps motion but general denoising.IMO you should use a pure progressive source then.

[FlimsyFeet]

A few notes... first, I hadn't discarded the idea completely, but I wanted to wait for the proper release of Didée's LTSMC, which is IMO an indispensable contender, and then leave a a few more months for us to get the hang of setting its parameters correctly. Oh, OK, sorry I didn't mean to tread on anyones toes. I just saw that there seemed to be a lot of options for denoising and it's very difficult without experience to know which are more suited to different types of noise, and some up to date guidance should be provided. I didn't realise this was in hand.
Second, I don't think that trying to run four categories simultaneously is ideal... it takes a long time to fine tune a script for a source, and very few people are likely to have time to manage four sources. So one animation clip (which I think LTSMC isn't meant for) would be a good place to start. Well, I was thinking the comparison would be more on the general methods/filters than fine tuning the settings. Thus, each script would be suitable to the general case in question, but would always need fine tuning for each source.
Third, you don't want the clip to have artefacts that should be removed by a different kind of plug-in (e.g. rainbows, field blending, ghosting). After all, there's no shortage of source material to choose from!Yes agree, and it seems to be that anime suffers the most from these kind of defects.
Fourth, you really must evaluate the results of the denoising process independently from the compression process. If you use 4000 kbps MPEG for the output, you may well find that clips that killed more detail end up looking better because they compress better. But the deeper point is that we want to test the quality of the denoisers rather than the peculiarities of any particular compression method. Depending on the length of the source clips -- and incidentally, I was thinking several clips from the same source to provide representativity -- I would say provide lossless versions (as well as compressed versions for those on dial-up, but using something better than 4000 kbps MPEG).OK, I can see what you're saying. But the main point of denoising is to make the video suitable for compressing. (And to make it more pleasing to view, but who views stuff that ir saved losslessly?) Maybe it should be 6 or 7Mbps instead, and also provide some PNG stills from the script before encoding?
Incidentally, VirtualDub filters can be imported into AVISynth script, and I think we should restrict their use to this method... otherwise it's too fiddly to reproduce.Yes that's agood point, and I agree with you on that.

[Audionut]

My 2 cents.

Don't use bitrate encoding, it can be very hard if not impossible to tell the difference of different encodes, "encoded" at the same bitrate.
edit: it also rules out any posibility of finding out the compression gain of the denoiser. At least I think so. :)

May I suggest to use a constant quantinizer.

I frames only. "To make it easier to post a screen shot of the exact same frame"

Example: Encode source at constant quant, no denoising. Note, "Filesize", "Speed"

Than reencode with chosen denoiser at same quant. Note, "Filesize", "Speed".

Do the same for all the denoisers in the comparision.
You could post a screen shot of each different encode at the same frame.

You would then know.

1. Compression gain with said denoiser.
2. Speed of said denoiser.
3. You would have the ability to post a screen shot of the exact same "I-Frame" for people to also see the quality difference between denoisers.

If I can find any sort of time soon, I would certainly be up to the task.

[Manao]

I frames only. "To make it easier to post a screen shot of the exact same frame"No, you have to use pframes ( at least ), and I'd consider bframes mantatory too. Their statistical properties are too different to be ignored. Noise tends to mess up the motion estimation, so you'll have a lot more gain on p & bframes than on iframes.

Constant quantizer isn't that good an idea, because you end up with files having different sizes and different looks. No way to compare them anymore. Something must be constant. Since quality obviously can't, it'll be filesize.

[Audionut]

No, you have to use pframes ( at least ), and I'd consider bframes mantatory too. Their statistical properties are too different to be ignored. Noise tends to mess up the motion estimation, so you'll have a lot more gain on p & bframes than on iframes.

Believe it or not I agree. I-Frames only makes it extremely easy to post screen shots of the same frame. ie, Same frame number, same frame type. Why screen shots? See below.

Constant quantizer isn't that good an idea, because you end up with files having different sizes and different looks. No way to compare them anymore. Something must be constant. Since quality obviously can't, it'll be filesize.

Ah, but that would not be one of the things worth comparing with different denoisers, Quality. NO

For me, being able to see the compression of each denoiser as well as a screen shot, to see the quality of said denoiser. Is. I guess what I want.

edit: I hold the compression of each denoiser in high reguard when chossing which one to use.
edit 2: If you were to run an encode at constant quant 2 with Denoiser A and the resulting quality was shit house, then it makes no difference what filesize it is, it's still shit house.

I would have thought for all money, that doing an encode at a fixed filesize would introduce the codec itself in the comparisson.
The only thing you would want to change in the comparission is the affect of each denoiser. Not the codec's ability to compress the video to a fixed filesize after it has been denoised.


I'm really grasping for straws here, as for how to decribe what i'm meaning. :)

[mg262]

Don't get me wrong -- you're not "stepping on my toes" at all. You're more than welcome to run whatever you like... but I personally wouldn't want to put in the (several evenings of) work needed to get a script I was satisfied with until LTSMC was one of the options in my toolbox.

In any case, your emphasis is likely to be very different to mine:
But the main point of denoising is to make the video suitable for compressing.I categorically disagree... I would say the main point of denoising is to remove noise to improve the visual quality of the video.* I'm a pure quality freak; many people will fall somewhere between the two of us, and some to the far side of you.

*I don't care about suitability for compression at all; obviously I don't store things losslessly, but I start by obtaining something that looks satisfactory, then I play extensively with encoder settings until I can find something that doesn't visibly mess up the output. That can take some time. One of the problems with playing with video is that the longer you do it the more sensitive you become; I can see artefacts in most commercial DVDs, and there are many people on this forum more sensitive than me... By the way, in case you're wondering, I process a very low volume!

The difference in emphasis is important... for example, one of the criticisms often levelled against PixieDust is that it softens a clip. Now this is a definite hit on the quality mark, but will tend to improve compression. So how you evaluate this depends on your relative emphasis on the two factors.

IMO this makes a joint test very difficult... but that's not my problem :p. The one piece of advice I would underline is that you shouldn't use sources with extraneous stuff in, including scratches, telecine, rainbows, etc... evaluating denoisers is hard enough on its own!

Final thought... bear in mind that in order to evaluate their own scripts before submission people will need to be able to duplicate your encoding process precisely.

[Richard Berg]

I agree that we should test with constant-quant encoding. Then, the produced file size becomes an extra dimension of measurement. That makes it harder to graph the results, but the data will be there for people who care about compressability (and I think we all do at one point or another).

[pelle412]

I think the output of the denoiser should be evaluated with a lossless codec (e.g. huffyuv or ffv1), and compressibility should be evaluated using a constant quantizer encode.

[Isochroma]

agreed

[lexor]

I think the output of the denoiser should be evaluated with a lossless codec (e.g. huffyuv or ffv1), and compressibility should be evaluated using a constant quantizer encode.
I totally agree, denoiser is there to first of all denoise (and it is that quality and speed that I care most for), compresibility is a distant second DVD-5 are dirt cheap.

[foxyshadis]

Denoiser should also be evaluated based on the type, some are meant to flatten small differences (cartoon/anime filters), some are meant to remove grain, some random noise and artifacts, some target only Luma or Chroma, so there's a few different categories of "denoiser" with some overlap that should be dealt with. I think some kind of subjective "smear factor" should be adknowledged, as in, how well it retains fine details in stills and motion with strong removal.

Comparing denoisers is almost as bad as doom9's comparing codecs, except there's no rate control, but there's also no one best setting depending on source and what you want... sigh. On the upside, I keep finding better combinations.

[Fizick]

As i wrote before, I am sure, that we need in objective comparison.
The only way is: take clear source (good DVD), create artifical noisy clip (for example, by analog capture fromn DVD, or by AddGrain),
apply various denosers, and compare denoised clip with a source (original DVD).
We can use SSIM or other metric for objective comparison, and visual subjective comparison too.
I used similar approach when I develop last Vaguedenoiser and FFT3dfilter versions.

[Elic]

pelle412
> the output of the denoiser should be evaluated with a lossless codec (e.g. huffyuv or ffv1)
NOT ffv1 if you please. As ffv1 is part of constantly-changing ffdshow it breaks compatibility time by time (and my favorite ffdshow is quite old :) ).

[Elic]

Fizick
> take clear source (good DVD), create artifical noisy clip (for example, by analog capture fromn DVD, or by AddGrain), apply various denosers
Hmm... And how to make clip similar to, for example, old Soviet animation films (semi-faded colors, extremely big grains and clear imprints of someone's boots :) )?

[Audionut]

The thought of taking a clean source, then dirting it, then trying to clean it, sounds awfull.

Would it not be better to find a dirty source to begin with that closely represents, what is found in your local dvd collection.

Say, 3 different comparissons.
1 with a clean dvd source. To see which denoiser is best at increasing compressability, without smoothing and/or destroying/creating artifacts.
1 with a typical* dvd, that has your typical* grain. Perhaps something like Terminator 2 special edition, pal version. At a guess.
And perhaps 1 with just a plain shit noisey vhs source.

I think it would also be appropriate to ask the creators what they think are the best settings for each source.

edit: I'm really interested in doing this comparission. I would really like to get how best to run this comparission sorted first. I don't want to just do a comparission my way, only to have people bitching at me cause I did this and that instead of something else.
All thoughts/comments/suggestions now will be appreciated.