A little quality test

[Audionut]

Source

mpeg2source("E:\SAWII\VIDEO_TS\test.d2v",ipp=false,info=3)
colormatrix(hints=true)
crop(8,14,-8,-14)
lanczos4resize(688,368)
trim(30,1030)
http://www.savefile.com/files/1175306


Aiming for approx 1500kbps.


Xvid.
First pass
Quantizer ratio 1.60
Quantizer offset 0.00
Full quality first pass
VHQ mode 4
Use VHQ for b-frames
Max I-frame interval 250
Trellis quantization
Custom matrix
8,11,11,11,12,13,16,19
11,11,11,11,12,14,17,20
11,11,12,13,15,17,20,24
11,11,13,16,19,22,26,31
12,12,15,19,25,30,36,42
13,14,17,22,30,39,49,60
16,17,20,26,36,49,65,83
19,20,24,31,42,60,83,110

14,14,14,14,15,17,21,26
14,14,14,14,15,18,23,27
14,14,15,17,20,23,27,33
14,14,17,21,26,30,36,44
15,15,20,26,35,42,51,60
17,18,23,30,42,46,58,87
21,23,27,36,51,58,78,122
26,27,33,44,60,87,122,162

2nd pass
Same settings as first with
I-frame boost 0
All overflow options 30
High bitrate degration 10

http://www.savefile.com/files/8558429
Result
approx 14fps both passes
File size 7,701,126 bytes
Average SSIM= 78.75173578


NeroDigital AVC.
Maximum Definition profile.
All settings standard execpt,
Maximum vector range -512 to 511.75
5 reference frames
Maximum gop size 250
Automatic sharp deblocking

http://www.savefile.com/files/6198004
Result
approx 4-5fps both passes
File size 7,512,304 bytes
Average SSIM= 80.07793265


X264
--ref 5 --mixed-refs --bframes 3 --b-pyramid --bime --weightb --analyse all --8x8dct --direct auto --threads 2 --thread-input --progress --no-psnr --output "E:\SAWII\VIDEO_TS\test.mp4" "E:\SAWII\VIDEO_TS\test.avs"
With a fast first pass.

Lost link. If you want it, I will re-upload it.
Result
approx 24fps first pass and 10fps 2nd pass
File size 7,303,115 bytes
Average SSIM= 81.31906215


X264 tuned settings
--ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --filter -1,-1 --subme 7 --aq-tcplx 4 --trellis 2 --analyse all --no-dct-decimate --8x8dct --ipratio 1.0 --qcomp 0.8 --direct auto --me umh --threads 2 --thread-input --progress --no-psnr --nr 50 --output "E:\SAWII\VIDEO_TS\test.mp4" "E:\SAWII\VIDEO_TS\test.avs"

http://www.savefile.com/files/8059260
Result
approx 24fps first pass and 3fps 2nd pass
File size 7,444,873 bytes
Average SSIM= 82.67125741

Some quick thoughts.
First encode was with Nero.
With the other codecs I just tried to get close enough to the file size of the Nero encode.

[foxyshadis]

Don't forget to post timings, which are quite important to most people! (Although if you do, you should add your system specs as well.) Even though that avisynth would probably be about 15-20% in xvid and nero's case, or 10% in x264's, it makes a perfect real life test.

[Audionut]

Added samples.

[GodofaGap]

Is this tuned for metrics or visual quality? --b-rdo --subme 7 --trellis 2 --me umh is a lot slower than it reasonably has to be. Also, why ipratio 1.0?

For XviD: using a CQM will probably only hurt metrics.

So I'm a bit confused... what is it you trying to show here?

[Audionut]

--b-rdo --subme 7 --trellis 2 --me umh is a lot slower than it reasonably has to be.

You will notice that I used those options only in the second X264 encode. To show what sort of a gain in SSIM can be attained.

For XviD: using a CQM will probably only hurt metrics.

Emphasis on the word "probably".
I will run another encode tommorrow with the default matrices.

So I'm a bit confused... what is it you trying to show here?

What do you see?
I see xvid way behind x264 in SSIM.
I see x264 running at the same speed if not a smidgen faster than xvid, while providing better quality.
I see nerodigital as slow as sh*t, and no better than x264 for SSIM.
I see that with x264, enabling all the "extra" features, does gain 1.3db in SSIM, but at the expence of encoding speed.

edit: And i'm waiting for siddharthagandhi to show me his encodes/results. He "claims" that both nerodigital and xvid are better than x264.

[nm]

Nice comparison. The ND AVC encode actually has more blocking than XviD, but ND is also more detailed. That's probably due to the "sharp deblocking" setting. The tuned x264 encode is sharp and detailed without blocking.

There seems to be a problem with the URL of the first x264 sample.

[Audionut]

Xvid encode.

H263 matrix SSIM = 79.93962393
MPEG matrix SSIM = 79.59657294

There seems to be a problem with the URL of the first x264 sample.

It's lost in the abyss, and there apears no way to search for it. If anyone really wants it, I can upload it again.

[Audionut]

X264 revision 508 used in test.
filesize 7,303,115 bytes
Average SSIM= 81.31906215

X264 revision 512B.
filesize 7,478,196 bytes
Average SSIM= 81.56315674

NeroDigital AVC
Same settings as above but with "automatic smooth deblocking"
Average SSIM= 81.07374597

[Sagittaire]

--aq-tcplx 4

IMO AQ at this time don't improve visual quality and certainely not SSIM


--ipratio 1.0

quality for Iframe must be always superior. IMO default setting improve quality and SSIM

--nr 50

it's a internal denoiser from libavcodec. IMO it's not the oblective for video codec to use internal denoiser. It's better to use high quality denoiser like convolution3d or hqdn3d in pre-process avisynth script if you want. Internal denoising don't improve metric like PSNR or SSIM.

[foxyshadis]

To compare a result with noise reduction, perhaps you should compare to an x264 lossless encode, using the same noise reduction value.

That would compare to the original in the same way a pre-filtered video would, which would be more indicative of the codec's performance and not the overall codec+filtering stage quality. On the other hand, the extra smoothness would give x264 an instant leg up in quality since it then isn't as hard to encode to. So I don't think there's any fair way to include it in a competition.

[akupenguin]

x264's denoising is part of quantization, it's disabled in lossless. So you'd have to use qp=1, and introduce even more discrepancy.

[Soulhunter]

It's better to use high quality denoiser like convolution3d or hqdn3d in pre-process avisynth script if you want. Internal denoising don't improve metric like PSNR or SSIM.
They are fast, but not HQ... In a small test I got a 0.5dB PSNR improvement over "simple" denoisers like HQDn3D or TemporalSoften by using "advanced" denoisers like FFT3D or FRFun! A bigger, better and more reliable denoiser test is in work btw... ^^


Bye

[Sagittaire]

They are fast, but not HQ... In a small test I got a 0.5dB PSNR improvement over "simple" denoisers like HQDn3D or TemporalSoften by using "advanced" denoisers like FFT3D or FRFun! A bigger, better and more reliable denoiser test is in work btw... ^^


source A -> denoising (source B) -> video B
source A -> video A

you obtain 0.5 dB improvement ... you are sure for that ... ???

be carefull metric test must be between [source A - video A] vs [source A - video B] and not between [source A - video A] vs [source B - video B]

Manao try that with TemporalSoften and with very particular source but metric improvement was really small and don't work in general case for general source.

[Soulhunter]

Yeah, I think thats what I did... Non denoised source -vs- denoised (denoisers were tweaked for equal NR) and compressed (max bitrate difference was below 1%) source. This was to test the codecs quantization (source -> NOP -> compressed) -vs- the quantization of the different denoisers (source -> denoised -> compressed at the same bitrate as the NOP). But the upcoming test will be completely different! There I test the raw denoising efficiency... Clean source -vs- Clean source -> noised -> denoised... with multiple sources, denoisers, settings... Its a good way to test the image reconstruction capability's of denoisers (the current PSNR results match my visual impression to 99% btw). Im not even half finished yet, but I can already tell that the results will be very interesting! Differences of 1dB and more... ^^


Bye

[BigDid]

@ Audionut: Impressive tests :thanks:

Concerning Siddharthagandhi, maybe his position is evolving, his signature now states:
"Favorite- ND/Ateme MP4 AVC
Rec. Codecs- (ranked by quality): x264, Xvid, Divx" :)

@Soulhunter; becareful if you don't post your tests, Audionut will also wait for you (as for siddharthagandhi) to prove it :D

Did

[Sagittaire]

Yeah, I think thats what I did... Non denoised source compressed -vs- Denoised source (denoisers were tweaked for equal NR) compressed at the same bitrate as the non denoised (max bitrate difference was below 1% iirc). But the upcoming test will be completely different! There I test the raw denoising efficiency... Clean source -vs- Clean source -> noised -> denoised... with multiple sources, denoisers, settings... Its a good way to test the image reconstruction capability's of denoisers (the PSNR results match my visual impression to 99%). Im not even half finished yet, but I can already tell that the results will be interesting! Differences of 1dB and more... ^^


Bye

your protocol are bad and useless because you test non equivalent source.

[Soulhunter]

your protocol are bad and useless because you test non equivalent source.

I dont understand what you mean... ^^;

Btw, I edited my previous post to explain it better!


Bye

[Sagittaire]

Yeah, I think thats what I did... Non denoised source compressed -vs- Denoised source (denoisers were tweaked for equal NR) compressed at the same bitrate as the non denoised (max bitrate difference was below 1% iirc). But the upcoming test will be completely different! There I test the raw denoising efficiency... Clean source -vs- Clean source -> noised -> denoised... with multiple sources, denoisers, settings... Its a good way to test the image reconstruction capability's of denoisers (the PSNR results match my visual impression to 99%). Im not even half finished yet, but I can already tell that the results will be interesting! Differences of 1dB and more... ^^


Bye

your protocol are bad and useless because you test non equivalent source.

In fact your test is just denoiser efficiency test I think. If you want make that it's not necessary to make encoding.

source A -> artificial noise -> denoising -> source B

compare simply source A and source B

[Soulhunter]

your protocol are bad and useless because you test non equivalent source.
I dont see where I used a non equivalent source! I always used the same source... Source -vs- Source -> NOP -> Compressed, Source -vs- Source -> Denoiser1 -> Compressed, Source -vs- Source -> Denoiser2 -> Compressed, Source -vs- Source -> Denoiser3 -> Compressed, and so on... what do you mean!?



In fact your test is just denoiser efficiency test I think. If you want make that it's not necessary to make encoding.

- The old test was codec quantization -vs- denoiser quantization

- The new test is denoiser efficiency -vs- denoiser efficiency



source A -> artificial noise -> denoising -> source B

compare simply source A and source B

Yeah, thats what I do... ^^


Bye

[BigDid]

@ Sagittaire, may I point out that Soulhunter already made a few CQM comparisons and tests in the past?

@Soulhunter, don't get discouraged by Sagittaire, it may be his cartesian way of posting :D (Hey don't crush it's a joke)
I, for one, am awaiting the results of your tests.
Cheers

Did

[Soulhunter]

Well, still ~150 filter runs to go! Considering that some filters run at 3fps or less, you probably have to wait two weeks till Im done... ^^;


Thanks n' Bye

[Sagittaire]

Like I say it's just a denoiser comparison and not coding improuvement with denoiser. IMO it's impossible in general case to improve metric with denoising and for particular case certainely impossible to improve 1db with simple denoising. In this thread we speak about real metric improvement and not comparison between denoiser.

Real metric improvement
source A -> denoising (source B) -> video B
source A -> video A
in this case metric(source A, video B) must be superior to metric(source A, video A)

Simple denoinsing comparison
source A -> artificial noise -> denoiser 1 -> source 1
source A -> artificial noise -> denoiser 2 -> source 2
best denoiser N obtain simply best metric (source A, source N)

[Soulhunter]

In this thread we speak about real metric improvement and not comparison between denoiser.

Real metric improvement
source A -> denoising (source B) -> video B
source A -> video A
in this case metric(source A, video B) must be superior to metric(source A, video A)

Yeah, thats exactly what I did in the first test! FFT3DFilter had an improvement of ~0.5dB above HQDn3D with the first source, and ~0.6dB with the second... But the improvement above the NOP was much lower, ~0.1dB with the first source, and ~0.02dB with the second! I just wanted to point out that C3D and HQDn3D are not "HQ" (as you stated/suggested) coz they produced even worse metric results than a NOP, and that there are much better denoisers, visually and metric-wise!



Simple denoinsing comparison
source A -> artificial noise -> denoiser 1 -> source 1
source A -> artificial noise -> denoiser 2 -> source 2
best denoiser N obtain simply best metric (source A, source N)

Again, thats exactly what I do in my second comparison!

And thats where I got variances of more than 1dB...


Bye

[akupenguin]

Yeah, thats exactly what I did in the first test! FFT3DFilter had an improvement of ~0.5dB above HQDn3D with the first source, and ~0.6dB with the second... But the improvement above the NOP was much lower, ~0.1dB with the first source, and ~0.02dB with the second! I just wanted to point out that C3D and HQDn3D are not "HQ" (as you stated/suggested) coz they produced even worse metric results than a NOP, and that there are much better denoisers, visually and metric-wise!
Say what? You're measuring how well a denoiser preserves noise.
I can buy the 2nd comparison, but the 1st comparison also needs to use artificial noise and compare against the clean source.

[foxyshadis]

I think you need other metrics than "how identical is this to the original", like SSIM, once you introduce denoisers. If grain is undesired then it makes no sense to compare against a source with grain! I know MSU started down this road with their blurring (and blocking) measure, but it's rather crude. Something like "how much interesting vs uninteresting detail is lost" or "how many artifacts (overblurring, blocking, ringing, blending, rainbows, etc) are present", somehow. But then you have to define what's desirable, and that changes for every person...

[IgorC]

Related poll http://forum.doom9.org/showthread.php?t=110504

[Soulhunter]

Say what? You're measuring how well a denoiser preserves noise...

Like I wrote before, I tested the codecs quantization -vs- the quantization of various denoisers... First I tweaked the settings of all denoisers for equal compressibility (tested via fixed quant encoding) then I tested a NOP (just the codecs quantization) against the tweaked denoisers (equal denoiser quantization + equal codec quantization). The goal was to find out if codec side pre-processing (denoising) can give better results than direct encoding! My result: Yes, but only by using relative complex/slow denoising schemes which would double or triple the encoding time... But thats just true for PSNR! Visually you can gain much more, even "simple" denoising should give better results than the codecs quantization, at least thats what my eyes tell me... Well, its also possible to use denoisers in a tricky way: In one of my recent experiments for example, where I applied denoising just to high motion scenes (scenes that are already blurry, not counting the grain), I got a 10% compressibility improvement without really noticing a difference (well, in difference to block artefacts its nearly impossible to notice a loss of grain in a high motion scene, eh?) to the non denoised version (both encoded at the same fixed quantizer). And the good thing is that the "motion detection + simple denoiser" combo runs relative fast, which is sorta surprising considering that it was scripted by mf... ^^;


I think you need other metrics than "how identical is this to the original", like SSIM, once you introduce denoisers. If grain is undesired then it makes no sense to compare against a source with grain...

Well, in the first test it made sense imo as the noise/grain was not "undesired" no, it was more the other way around, its was: Which stays closer to the source at a certain level of compression, direct encoding or encoding + pre-processing? In the second test again, the noise/grain is absolutely undesired, yes, but there I compare the denoised result against the clean source (no compression at all btw). And to measure the overall image reproduction of a denoiser, PSNR measuring should be enough, at least thats what some experienced filter dev told me... ;D


Bye

[Audionut]

Clean video <--clip a
Clean video with noise added <--clip b
Clean video with noise added and then denoised <-- clip c

Then compare clip a to clip b.
SSIM result say 80db.
Clip a to clip c.
SSIM result increased to say 85db.
And continue comparing clip a to clip c with different denoisers.

edit: this would be to compare the denoising ability of filters correct?
Personally I prefer the ablity of denoisers to increase encoder efficiency at low bitrates.
Say target bitrate 1000kbps.
No denoising = 80db
denoiser a = 82db
denoiser b = 83db etc.

[Soulhunter]

Clean video <--clip a
Clean video with noise added <--clip b
Clean video with noise added and then denoised <-- clip c

Then compare clip a to clip b.
SSIM result say 80db.
Clip a to clip c.
SSIM result increased to say 85db.
And continue comparing clip a to clip c with different denoisers.

Yeah, thats what I do at the moment, just that I use PSNR, so the results are more like... 42dB for ClipA -vs- ClipB, and 47dB for ClipA -vs- ClipC... Of course, the results change with the amount off noise you add! I used a mixture of AddGrain and GNoise (produces gaussian noise) to simulate the noise you find on a average DVD. Well, maybe Ill do a second test in future when I found a satisfying way to simulate film grain!? Btw, sorry for polluting your thread, really... ^^;


Bye

[Audionut]

Wouldn't you need to encode a clip with added grain to a lossless format?
And then compare against it.

Doesn't filter's like blockbuster produce different noise.

[Audionut]

@Sagittaire

The original command line in the first post with rev512b.
Average SSIM= 82.70546960
Size 7,469,524 bytes

Same command line minus --aq-tcplx 4 and --nr 50 with a default ipratio.
Average SSIM= 82.76732782
Size 7,465,656 bytes

[Soulhunter]

@ Audionut

Yes, you can add the noise to the source, encode it lossless, and use this as input for the denoising (you just have to make sure that its encoded/decoded in the same colorspace as the other stuff) or you can simply do it all via avs script (load source -> add noise -> do denoising) and use this as "B" input for the computation (but then you need to re-do the noising step each time, so its a little bit slower...). For BlockBuster: With this filter you are able to add noise only to edges (leaves flat areas clean) or only to flat areas (leaves the edges clean) for example. Well, here (http://img159.imageshack.us/img159/1279/015cv.jpg) is a image which visualizes the effect... But to simulate grain you have to take care of much more things, like different amounts of grain and different grain sizes for each luma level and color... Dark areas should get rougher grain than bright areas for example, and blue areas should get more grain than red or green ones... sorta hard to do via avs scripting alone, at least for me... :\


Bye

[Audionut]

My biggest concern is that doing it all by avs script (load source -> add noise -> do denoising), you would get different amount of grain each time.

Hence an unreliable comparision.

[Soulhunter]

Yeah, while GNoise offers a option to produce always the same noise, other filters like AddGrain use a random pattern iirc, so you should get a minimal difference! Well, I used the lossless compression way to be on the save side, and to save processing time (considering how much filter runs I do, it should save me some hours overall).


Bye

[Sagittaire]

@Sagittaire

The original command line in the first post with rev512b.
Average SSIM= 82.70546960
Size 7,469,524 bytes

Same command line minus --aq-tcplx 4 and --nr 50 with a default ipratio.
Average SSIM= 82.76732782
Size 7,465,656 bytes

it's an real metric improvement for you ... lol

try with source without noise and post result ... !!!

I quote myself ...

IMO it's impossible in general case to improve metric with denoising and for particular case certainely impossible to improve 1db with simple denoising.

[Audionut]

However. If you are encoding to a low bitrate, denoising will help.

Things like the various removegrain modes only decrease SSIM by a very small margin, but can noticably reduce the bitrate required to encode video.
It is a very effective and easy way to help encoders when shooting for a small target bitrate.

[Sagittaire]

However. If you are encoding to a low bitrate, denoising will help.

Things like the various removegrain modes only decrease SSIM by a very small margin, but can noticably reduce the bitrate required to encode video.
It is a very effective and easy way to help encoders when shooting for a small target bitrate.

DCT + quantisation (+ inloop for AVC) make denoising itself and particulary if you use high quant (low bitrate encoding)