Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 12th October 2009, 23:52   #1  |  Link
Chengbin
Registered User
 
Join Date: Oct 2007
Posts: 1,059
Meaning of PSNR and SSIM scores.

I recall a topic about using metrics as a use of measuring efficiency. I felt like playing around with metrics today.

I did a comparison, 8x8dct vs no 8x8dct. Of course the same settings for both except that, using 2 pass at the same bitrate.

I didn't finish the 8x8dct encode yet, but I got the scores for the non 8x8dct encode.

SSIM Mean Y:0.985
PSNR Mean Y:44.197 U:48.890 V:48.670 Avg:45.226 Global:44.522 kb/s:678.98

How do I interpret this data? I believe you look at the average score for PSNR?

I know higher is better, but an increase of x, what does that mean? Can you translate it into percents?

BTW, how do I get the log for these data produced by x264? I had to encode something else after the test to prevent the window from closing.

To the mods, please don't tell me to search. I did search the forum. I didn't find much, and I didn't want to dig up an old thread and hijack it.

Thanks a lot.
Chengbin is offline   Reply With Quote
Old 13th October 2009, 00:39   #2  |  Link
J_Darnley
Registered User
 
J_Darnley's Avatar
 
Join Date: May 2006
Posts: 959
Quote:
Originally Posted by Chengbin View Post
BTW, how do I get the log for these data produced by x264? I had to encode something else after the test to prevent the window from closing.
Run cmd then run x264 in it, or if you want to use a batch file, use pause.
__________________
x264 log explained || x264 deblocking how-to
preset -> tune -> user set options -> fast first pass -> profile -> level
Doom10 - Of course it's better, it's one more.
J_Darnley is offline   Reply With Quote
Old 13th October 2009, 00:48   #3  |  Link
dstln
Person
 
dstln's Avatar
 
Join Date: Jul 2009
Posts: 58
http://en.wikipedia.org/wiki/SSIM

http://en.wikipedia.org/wiki/PSNR
dstln is offline   Reply With Quote
Old 13th October 2009, 00:52   #4  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,689
ssim: ((1-oldssim)/(1-newssim) - 1)*100 = % improvement
psnr: (new - old) / 0.05 = % improvement
Dark Shikari is offline   Reply With Quote
Old 13th October 2009, 01:13   #5  |  Link
Chengbin
Registered User
 
Join Date: Oct 2007
Posts: 1,059
Quote:
Originally Posted by J_Darnley View Post
Run cmd then run x264 in it, or if you want to use a batch file, use pause.
That doesn't get you the log files. I still have to hand type it. I use a GUI anyway, so that's out of the question.
Chengbin is offline   Reply With Quote
Old 13th October 2009, 01:15   #6  |  Link
J_Darnley
Registered User
 
J_Darnley's Avatar
 
Join Date: May 2006
Posts: 959
Quote:
Originally Posted by Chengbin View Post
That doesn't get you the log files. I still have to hand type it. I use a GUI anyway, so that's out of the question.
Oh jeez, right click -> mark, make your selection, press enter or use the 2> file redirection.
__________________
x264 log explained || x264 deblocking how-to
preset -> tune -> user set options -> fast first pass -> profile -> level
Doom10 - Of course it's better, it's one more.
J_Darnley is offline   Reply With Quote
Old 13th October 2009, 01:16   #7  |  Link
kemuri-_9
Compiling Encoder
 
kemuri-_9's Avatar
 
Join Date: Jan 2007
Posts: 1,348
Quote:
Originally Posted by Chengbin View Post
That doesn't get you the log files. I still have to hand type it. I use a GUI anyway, so that's out of the question.
pipe stderr to file.
the cli version of this feat would be 2>file.txt
how guis do this differs look it up with whatever one you're using
__________________
custom x264 builds & patches | F@H | My Specs
kemuri-_9 is offline   Reply With Quote
Old 13th October 2009, 01:17   #8  |  Link
kidjan
Registered User
 
kidjan's Avatar
 
Join Date: Oct 2008
Posts: 39
Quote:
Originally Posted by Chengbin View Post
I recall a topic about using metrics as a use of measuring efficiency. I felt like playing around with metrics today.

I did a comparison, 8x8dct vs no 8x8dct. Of course the same settings for both except that, using 2 pass at the same bitrate.

I didn't finish the 8x8dct encode yet, but I got the scores for the non 8x8dct encode.

SSIM Mean Y:0.985
PSNR Mean Y:44.197 U:48.890 V:48.670 Avg:45.226 Global:44.522 kb/s:678.98

How do I interpret this data? I believe you look at the average score for PSNR?

I know higher is better, but an increase of x, what does that mean? Can you translate it into percents?
SSIM stands for Structural Similarity Index, and it's an algorithm that assesses how "similar" two images are. If you know one image is a reference image, you can use this score to assess how "similar" the encoded image is. The resulting scores range between -1.0 (completely dissimilar) to 1.0 (identical). For the nitty-gritty, wikipedia is a good start, although the wiki article has some slight inaccuracies that should be clear if you read Z. Wang's paper.

PSNR is another measurement that can be used to compute image similarity, although it functions completely different from SSIM. SSIM measures how "similar" various metrics of the images are, while PSNR is essentially a measurement of "error" between the two images. Higher PSNR is better.

In my own anecdotal experience, a SSIM score of 0.985 is quite good. I have a hard time telling the difference between encoded material and an original with SSIM scores above ~.98, although that may vary depending on the SSIM implementation. I'm not sure how x264 implements SSIM (i.e. equal weighting for all categories, is it MSSIM, is the reported score a mean/lowest observed value/median among all frames, etc.) so 0.98 may not be apples to apples with the SSIM implementation I wrote for some other stuff.

In general, SSIM correlates better with mean opinion scores than PSNR, so I tend to trust those results more.

Last edited by kidjan; 13th October 2009 at 01:19.
kidjan is offline   Reply With Quote
Old 13th October 2009, 01:19   #9  |  Link
Chengbin
Registered User
 
Join Date: Oct 2007
Posts: 1,059
Quote:
Originally Posted by J_Darnley View Post
Oh jeez, right click -> mark, make your selection, press enter or use the 2> file redirection.
OMG I can't believe I didn't find this feature!!! Thanks.
Chengbin is offline   Reply With Quote
Old 13th October 2009, 02:36   #10  |  Link
Chengbin
Registered User
 
Join Date: Oct 2007
Posts: 1,059
I'm getting a 3.7% improvement for SSIM and 3.28% improvement for PSNR comparing non 8x8dct and 8x8dct. The movie is a clip from Wanted. I was going to test the whole movie, but I accidentally encoded no 8x8dct again, so I ran out of time.

???

I thought 8x8dct was a huge gain???

My eyes couldn't tell much (if any) difference either. It is 680Kbps @ 688x288.

Last edited by Chengbin; 13th October 2009 at 02:40.
Chengbin is offline   Reply With Quote
Old 13th October 2009, 02:49   #11  |  Link
nm
Registered User
 
Join Date: Mar 2005
Location: Finland
Posts: 2,643
Quote:
Originally Posted by Chengbin View Post
I thought 8x8dct was a huge gain???
3 % is a lot compared to gains from many other settings.

Quote:
My eyes couldn't tell much (if any) difference either. It is 680Kbps @ 688x288.
Perhaps your encode is transparent with and without 8x8dct. Try a lower bitrate.

Edit: Also, a difference of this magnitude is probably only visible when alternating between matched frames from the two encodes.

Last edited by nm; 13th October 2009 at 03:15.
nm is offline   Reply With Quote
Old 13th October 2009, 02:54   #12  |  Link
Chengbin
Registered User
 
Join Date: Oct 2007
Posts: 1,059
Quote:
Originally Posted by nm View Post
3 % is a lot compared to gains from many other settings.


Perhaps your encode is transparent with and without 8x8dct. Try a lower bitrate.
I thought the gain is around 10-20%...

I'll do another test tomorrow. Got something else to do today.
Chengbin is offline   Reply With Quote
Old 13th October 2009, 03:39   #13  |  Link
Snowknight26
Registered User
 
Join Date: Aug 2007
Posts: 1,394
Quote:
Originally Posted by Chengbin View Post
I thought the gain is around 10-20%...
Why would you think that?
Snowknight26 is offline   Reply With Quote
Old 13th October 2009, 03:43   #14  |  Link
Chengbin
Registered User
 
Join Date: Oct 2007
Posts: 1,059
Quote:
Originally Posted by Snowknight26 View Post
Why would you think that?
Because I thought high profile is always used, and 8x8dct is ALWAYS recommended, like it is a necessity, and that suggests me that 8x8dct is REALLY helpful.

I didn't think 4% is that helpful. Isn't many settings give more, or around 3-4% improvement?

Last edited by Chengbin; 13th October 2009 at 03:51.
Chengbin is offline   Reply With Quote
Old 13th October 2009, 03:52   #15  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,689
A) Because it gives more gain psy-wise than PSNR would suggest.

B) Because it gives more quality gain relative to speed cost than almost any other option.
Dark Shikari is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 01:35.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.