Meaning of PSNR and SSIM scores.

[Chengbin]

I recall a topic about using metrics as a use of measuring efficiency. I felt like playing around with metrics today.

I did a comparison, 8x8dct vs no 8x8dct. Of course the same settings for both except that, using 2 pass at the same bitrate.

I didn't finish the 8x8dct encode yet, but I got the scores for the non 8x8dct encode.

SSIM Mean Y:0.985
PSNR Mean Y:44.197 U:48.890 V:48.670 Avg:45.226 Global:44.522 kb/s:678.98

How do I interpret this data? I believe you look at the average score for PSNR?

I know higher is better, but an increase of x, what does that mean? Can you translate it into percents?

BTW, how do I get the log for these data produced by x264? I had to encode something else after the test to prevent the window from closing.

To the mods, please don't tell me to search. I did search the forum. I didn't find much, and I didn't want to dig up an old thread and hijack it.

Thanks a lot.

[J_Darnley]

Originally Posted by Chengbin

BTW, how do I get the log for these data produced by x264? I had to encode something else after the test to prevent the window from closing.

Run cmd then run x264 in it, or if you want to use a batch file, use pause.

[dstln]

http://en.wikipedia.org/wiki/SSIM

http://en.wikipedia.org/wiki/PSNR

[Dark Shikari]

ssim: ((1-oldssim)/(1-newssim) - 1)*100 = % improvement
psnr: (new - old) / 0.05 = % improvement

[Chengbin]

Originally Posted by J_Darnley

Run cmd then run x264 in it, or if you want to use a batch file, use pause.

That doesn't get you the log files. I still have to hand type it. I use a GUI anyway, so that's out of the question.

[J_Darnley]

Originally Posted by Chengbin

That doesn't get you the log files. I still have to hand type it. I use a GUI anyway, so that's out of the question.

Oh jeez, right click -> mark, make your selection, press enter or use the 2> file redirection.

[kemuri-_9]

Originally Posted by Chengbin

That doesn't get you the log files. I still have to hand type it. I use a GUI anyway, so that's out of the question.

pipe stderr to file.
the cli version of this feat would be 2>file.txt
how guis do this differs look it up with whatever one you're using

[kidjan]

Originally Posted by Chengbin

I recall a topic about using metrics as a use of measuring efficiency. I felt like playing around with metrics today.

I did a comparison, 8x8dct vs no 8x8dct. Of course the same settings for both except that, using 2 pass at the same bitrate.

I didn't finish the 8x8dct encode yet, but I got the scores for the non 8x8dct encode.

SSIM Mean Y:0.985
PSNR Mean Y:44.197 U:48.890 V:48.670 Avg:45.226 Global:44.522 kb/s:678.98

How do I interpret this data? I believe you look at the average score for PSNR?

I know higher is better, but an increase of x, what does that mean? Can you translate it into percents?

SSIM stands for Structural Similarity Index, and it's an algorithm that assesses how "similar" two images are. If you know one image is a reference image, you can use this score to assess how "similar" the encoded image is. The resulting scores range between -1.0 (completely dissimilar) to 1.0 (identical). For the nitty-gritty, wikipedia is a good start, although the wiki article has some slight inaccuracies that should be clear if you read Z. Wang's paper.

PSNR is another measurement that can be used to compute image similarity, although it functions completely different from SSIM. SSIM measures how "similar" various metrics of the images are, while PSNR is essentially a measurement of "error" between the two images. Higher PSNR is better.

In my own anecdotal experience, a SSIM score of 0.985 is quite good. I have a hard time telling the difference between encoded material and an original with SSIM scores above ~.98, although that may vary depending on the SSIM implementation. I'm not sure how x264 implements SSIM (i.e. equal weighting for all categories, is it MSSIM, is the reported score a mean/lowest observed value/median among all frames, etc.) so 0.98 may not be apples to apples with the SSIM implementation I wrote for some other stuff.

In general, SSIM correlates better with mean opinion scores than PSNR, so I tend to trust those results more.

[Chengbin]

Originally Posted by J_Darnley

Oh jeez, right click -> mark, make your selection, press enter or use the 2> file redirection.

OMG I can't believe I didn't find this feature!!! Thanks.

[Chengbin]

I'm getting a 3.7% improvement for SSIM and 3.28% improvement for PSNR comparing non 8x8dct and 8x8dct. The movie is a clip from Wanted. I was going to test the whole movie, but I accidentally encoded no 8x8dct again, so I ran out of time.

???

I thought 8x8dct was a huge gain???

My eyes couldn't tell much (if any) difference either. It is 680Kbps @ 688x288.

[nm]

Originally Posted by Chengbin

I thought 8x8dct was a huge gain???

3 % is a lot compared to gains from many other settings.

My eyes couldn't tell much (if any) difference either. It is 680Kbps @ 688x288.

Perhaps your encode is transparent with and without 8x8dct. Try a lower bitrate.

Edit: Also, a difference of this magnitude is probably only visible when alternating between matched frames from the two encodes.

[Chengbin]

Originally Posted by nm

3 % is a lot compared to gains from many other settings.


Perhaps your encode is transparent with and without 8x8dct. Try a lower bitrate.

I thought the gain is around 10-20%...

I'll do another test tomorrow. Got something else to do today.

[Snowknight26]

Originally Posted by Chengbin

I thought the gain is around 10-20%...

Why would you think that?

[Chengbin]

Originally Posted by Snowknight26

Why would you think that?

Because I thought high profile is always used, and 8x8dct is ALWAYS recommended, like it is a necessity, and that suggests me that 8x8dct is REALLY helpful.

I didn't think 4% is that helpful. Isn't many settings give more, or around 3-4% improvement?

[Dark Shikari]

A) Because it gives more gain psy-wise than PSNR would suggest.

B) Because it gives more quality gain relative to speed cost than almost any other option.