PSNR & SSIM test of 1.0b

[mevensson]

Hi!

I've done PSNR and SSIM tests on a couple of short clips with diffrent parameters and thought I should share my results. All tests are made in 2 pass mode with MPEG quantizers. To decide the target size for each clip I encoded it with the default settings at constant quant 4. All clips are from PAL DVD's.

Clip 1 is from American Pie 2, fairly low motion with some panning.
Length 1 min 9 s. Target bitrate 1499 kb/s. Size 704x368.

PSNR
Options Y U V Sum SSIM Avg Quant
-----------------------------------------------------------------
vhq4 40.08 45.20 44.87 41.19 73.99 3.960
vhq4+aq 39.87 45.12 44.80 41.00 74.07 4.489
vhq4+cm 40.08 45.21 44.88 41.20 74.01 3.962
vhq4+co 40.07 45.19 44.86 41.19 73.99 3.960
vhq4+gmc 40.19 45.35 45.04 41.32 74.40 3.883
vhq4+qpel 40.72 45.47 45.18 41.79 75.57 3.584
vhq4+trellis 40.14 45.15 44.78 41.23 73.93 3.819


Clip 2 is from High Fidelity, really low motion pretty dark.
Length 1 min 18 s. Target bitrate 515 kb/s. Size 704x368.

PSNR
Options Y U V Sum SSIM Avg Quant
-----------------------------------------------------------------
vhq4 44.54 49.01 48.95 45.59 83.76 3.745
vhq4+aq 44.33 48.93 48.86 45.39 83.71 3.866
vhq4+cm 44.53 49.02 48.95 45.58 83.74 3.756
vhq4+co 44.54 48.91 48.89 45.57 83.75 3.753
vhq4+gmc 44.65 49.13 49.07 45.70 83.96 3.679
vhq4+qpel 44.37 49.00 48.96 45.43 82.58 3.613
vhq4+trellis 44.58 49.04 48.98 45.62 83.78 3.642


Clip 3 is from Matrix, a part of the lobby scene.
Length 1 min 38 s. Target bitrate 1275 kb/s. Size 704x288.

PSNR
Options Y U V Sum SSIM Avg Quant
-----------------------------------------------------------------
vhq4 41.52 49.37 51.39 43.00 68.69 3.850
vhq4+aq 41.47 49.35 51.33 42.95 68.91 3.904
vhq4+cm 41.51 49.39 51.40 43.00 68.69 3.844
vhq4+co 41.52 49.38 51.40 43.00 68.70 3.850
vhq4+gmc 41.53 49.37 51.41 43.01 68.74 3.859
vhq4+qpel 41.16 49.32 51.45 42.66 67.07 4.023
vhq4+trellis 41.53 49.41 51.43 43.01 68.69 3.786


Clip 4 is from Spiderman, a bright high motion clip.
Length 1 min 28 s. Target bitrate 3587 kb/s. Size 704x368.

PSNR
Options Y U V Sum SSIM Avg Quant
-----------------------------------------------------------------
vhq4 39.81 43.25 43.04 40.66 72.85 3.916
vhq4+aq 39.63 43.17 42.93 40.50 72.66 4.114
vhq4+cm 39.81 43.25 43.04 40.67 72.85 3.913
vhq4+co 39.81 43.23 43.02 40.66 72.84 3.918
vhq4+gmc 39.85 43.29 43.08 40.70 72.95 3.952
vhq4+qpel 39.62 43.22 43.01 40.50 72.01 3.960
vhq4+trellis 39.88 43.19 42.99 40.71 72.81 3.850


Clip 5 is from Star Wars I, fairly high motion with bright colors and lots of CGI.
Length 1 min 27 s. Target bitreate 2098 kb/s. Size 704x288.

PSNR
Options Y U V Sum SSIM Avg Quant
-----------------------------------------------------------------
vhq4 40.98 43.77 45.21 41.86 71.61 3.872
vhq4+aq 40.84 43.71 45.14 41.73 71.45 4.137
vhq4+cm 40.99 43.78 45.22 41.87 71.65 3.862
vhq4+co 40.98 43.76 45.21 41.86 71.62 3.872
vhq4+gmc 40.99 43.78 45.22 41.86 71.63 3.886
vhq4+qpel 40.63 43.72 45.20 41.57 70.09 3.951
vhq4+trellis 41.02 43.72 45.20 41.88 71.56 3.788


Some observations: Chroma Motion doesn't help either PSNR or SSIM significantly on any clip.
GMC helps both PSNR and SSIM significantly on most clips and doesn't hurt on any.
Quarter pixel hurts both PSNR and SSIM signficantly on most clips except one where it instead helps signficantly. I have no idea whats so special about that clip, any sugestions?.
Trellis helps PSNR a little bit on most clips but doesn't help SSIM on any clip. It actually hurts SSIM a little on most clips.


Any comments or questions?

[Tommy Carrot]

My only comment: don't trust in these quality measurers like PSNR or SSIM too much. They are interesting and all, but not the real indicators of the quality.

[Sharktooth]

It's interesting to see how the various options "change the numbers".
Sure the numbers cant be taken as a final word on quality but they can catch what our eyes sometimes cant.
They give a general indication of the encoder (not the encode) quality. If activating an option higher the PSNR it means something is "less different" from the original image. That means the encoder did a "better good job" (than without that option) in reproducing the source but the final result maybe better or maybe not (it depends on personal preference) but still the encoder did a better job.

[JimiK]

First of all, I don't trust PSNR either. Still I find it interesting that QPel doesn't perform to well. It's not one of my favorite options (matter of taste), but now I could say: hey, I always knew it. It's like with clairvoyance. You don't really believe in it, but if a prophecy is good for you, why not :)
Then it's interesting to see that GMC is really working (not that I ever doubted it). Now I'll have to do some tests to see how much it slows down encoding to know if it's worth enabling it.
Best regards,
JimiK

[Selur]

Thanks for the work :)

Would be nice to see how the combination of the settings influence PSNR&Co,.. ;)

Cu Selur

Ps.: Could also be nice to see how reduced resolution influences the values ;)

[Joe Fenton]

:rolleyes:

What I saw in the table was that with vhq4, any other setting made virtually no difference at all.

Not a single rating was more than a few tenths more or less than any other. Seems you proved that vhq4 makes everything else worthless.

Since the other settings aren't worthless, it just demonstrates (again) how worthless these PSNR and other calculations really are in judging compression.

[KpeX]

Originally posted by Joe Fenton
Since the other settings aren't worthless, it just demonstrates (again) how worthless these PSNR and other calculations really are in judging compression. I completely agree. What was the last time you were watching a movie and thought to yourself, "gee the psnr seems a bit low". To me the only quantitative measure of quality would be first pass filesize, i.e. compressibility. So if you really want to compare features do a bunch of 1-pass quant 2 encodes. That's what I did in the Aloha thread (http://forum.doom9.org/showthread.php?s=&postid=406347#post406347).

But no offense, thanks very much for your hard work and posting your results clearly.

Edit about qpel: Qpel gives you a finer motion estimation and in many cases increases the sharpness but decreases the compressibility, which could explain why it helped only in the low motion clip. Qpel can also have some unpleasing results on sources with a lot of grain, when using low bitrate (which I don't think American Pie 2 was IIRC)

[mevensson]

Originally posted by Selur
Ps.: Could also be nice to see how reduced resolution influences the values ;) [/B]

After your suggestion I tried reduced resolution on the the first clip, but the option seems to be buggy. The resulting file has lots of very bad artifacs all over it. It got PSNR 23.13 and SSIM 4.06 if you want to know.

Have you tried reduced resolution? Does it work for you?

[Tommy Carrot]

Afaik reduced resolution is only supported because it's in the mpeg4 standard, not for using it. :D That option is only useful for very low bitrates (<100 kbps), not for normal encodings.

[Lefungus]

These results may be interesting in a frame-per-frame graph of all values. As you just kept the average score, almost all differences between options are smoothed out.
SSIM is under-optimal in this situation as motion isn't taken into account. But you may still have the graphs (csv files) generated by the various tests. With this, we may see that one scene is much better handled with some option activated.
I've done such extensive tests this summer on an old devapi4 build. Eventually, i found that default options were the best. And qpel is more a matter of taste.
Anyway thanks for the tests !

[mevensson]

I've only got the .csv files for American Pie 2 and Matrix left. I've put them here http://dual.private/xvid/ if you would like to watch them.

[MfA]

Originally posted by KpeX
I completely agree. What was the last time you were watching a movie and thought to yourself, "gee the psnr seems a bit low". To me the only quantitative measure of quality would be first pass filesize

Quantitative measure? ;)

Anyway ... say I have 2 trellis coding modes, the difference being the selection of lambda (which is a black art, so open to experimentation). Is the one with the smallest filesize at fixed quant then the best?

For "compressibility" to be a quality measure it should give some indication ... so you should be able to answer that question yes or no. The problem is of course that no yes or no answer is possible, because "compressibility" is only a measure of coding efficiency under very limited circumstances. For measuring the effects of trellis search or b-frames for instance it is really pretty much useless (for testing motion estimation and qpel it's reasonable though, although of course if during actual rate limited coding your eyes disagree with "compressibility" your eyes are in the right ... as with any objective measure, subjective measures are always better).

[KpeX]

Originally posted by MfA
Quantitative measure? ;)


Yes, from a practical standpoint, assuming a two-pass encoding scenario and attempting to hit a desired number of CDs, the lower the first pass size is (at quant 2), the higher quality the second pass will be with a constant desired size.

[MfA]

It is trivial to make trellis give smaller first passes ... but you wouldnt be happy how it would end up looking.

Also how exactly do I determine the optimal b-frame ratio with your quality measure?

[Mug Funky]

i think i read around here somewhere that trellis hurts PSNR with the MPEG matrix, but helps with H-263.

however, the bitrates were different between the two, so i'm not sure how that difference would translate in 2-pass mode, where the target size is constant.

[Selur]

That option is only useful for very low bitrates (<100 kbps), not for normal encodings.

Hmm,.. why is that so as far as I understood it, the feature should also be usefull for higher datarates,..

To make it short I thought it's ment to lower the resolution horizontal and/or vertical if needed and so to lose some details to avoid blocking.

(this should help even with normal 1CD DVD Rip datarates, shouldn't it ? )


Cu Selur

Ps.: didn't test the feature in Xvid atm, since I got some pc hardware problems.

[Asrial]

What is PSNR and SSIM?

Where can I find more info about what they do and how I can get those results myself?

[Selur]

see: http://www.everwicked.com/content/QualityMetrics/

Cu Selur

[MfA]

PSNR is Peak Signal To Noise ratio, SSIM (http://www.cns.nyu.edu/~zwang/) is the Structural SIMilarity index.

Both are filters in avisynth.