Higher SSIM or Higher PSNR?

[cagali]

The clip is a short 30s 1440x1080i HDTV live action footage.
I encoded it as MBAFF, with x264 8-bit and 10-bit.

The 8-bit encoded clip has 0.2db higher PSNR (+0.6%),
while the 10-bit one has 0.0017 higher SSIM (+0.17%).

Which one do you think is better based on the metrics?
Or ultimately we should only trust our eyes?

It is well-known that for animated source, x264 10-bit is much better than 8-bit.
However for grainy, noisy, live action footage, on the web I cannot find solid support of either format.

Is there any other good video/image comparison method?
Thank you.

8-bit metrics:
SSIM Mean Y:0.9520307 (13.190db)
PSNR Mean Y:37.594 U:45.070 V:43.784 Avg:38.912 Global:38.619 kb/s:3018.10

10-bit metrics:
SSIM Mean Y:0.9537058 (13.345db)
PSNR Mean Y:37.367 U:45.012 V:43.568 Avg:38.692 Global:38.411 kb/s:3021.14

8-bit log:
[5:20:17 AM] x264 [info]: profile High, level 4.1
[5:21:28 AM] x264 [info]: frame I:3 Avg QP:26.57 size: 89307 PSNR Mean Y:40.69 U:46.83 V:45.96 Avg:41.88 Global:41.75
[5:21:28 AM] x264 [info]: frame P:215 Avg QP:30.47 size: 23743 PSNR Mean Y:37.91 U:45.05 V:43.80 Avg:39.20 Global:38.92
[5:21:28 AM] x264 [info]: frame B:401 Avg QP:33.03 size: 6033 PSNR Mean Y:37.40 U:45.07 V:43.76 Avg:38.73 Global:38.45
[5:21:28 AM] x264 [info]: consecutive B-frames: 1.3% 8.1% 86.8% 3.9%
[5:21:28 AM] x264 [info]: mb I I16..4: 24.0% 64.8% 11.2%
[5:21:28 AM] x264 [info]: mb P I16..4: 9.4% 12.5% 0.6% P16..4: 43.8% 6.8% 7.4% 0.1% 0.0% skip:19.4%
[5:21:28 AM] x264 [info]: mb B I16..4: 0.8% 0.6% 0.0% B16..8: 36.6% 4.1% 0.7% direct: 1.6% skip:55.5% L0:35.2% L1:59.2% BI: 5.6%
[5:21:28 AM] x264 [info]: field mbs: intra: 45.4% inter:51.4% skip:25.9%
[5:21:28 AM] x264 [info]: 8x8 transform intra:55.0% inter:83.9%
[5:21:28 AM] x264 [info]: direct mvs spatial:58.6% temporal:41.4%
[5:21:28 AM] x264 [info]: coded y,uvDC,uvAC intra: 34.2% 26.6% 3.8% inter: 9.0% 4.4% 0.7%
[5:21:28 AM] x264 [info]: i16 v,h,dc,p: 38% 22% 21% 19%
[5:21:28 AM] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 9% 37% 5% 7% 8% 6% 6% 7%
[5:21:28 AM] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 23% 18% 9% 6% 10% 11% 9% 7% 7%
[5:21:28 AM] x264 [info]: i8c dc,h,v,p: 61% 19% 17% 3%
[5:21:28 AM] x264 [info]: ref P L0: 64.5% 24.4% 6.3% 2.8% 0.7% 0.6% 0.6% 0.3%
[5:21:28 AM] x264 [info]: ref B L0: 73.5% 23.3% 2.4% 0.4% 0.2% 0.2%
[5:21:28 AM] x264 [info]: ref B L1: 77.5% 22.5%
[5:21:28 AM] x264 [info]: SSIM Mean Y:0.9520307 (13.190db)
[5:21:28 AM] x264 [info]: PSNR Mean Y:37.594 U:45.070 V:43.784 Avg:38.912 Global:38.619 kb/s:3018.10


10-bit log:
[2:55:39 PM] x264 [info]: profile High 10, level 4.1, 4:2:0 10-bit
[2:57:33 PM] x264 [info]: frame I:3 Avg QP:38.31 size: 87716 PSNR Mean Y:40.67 U:46.64 V:45.66 Avg:41.83 Global:41.65
[2:57:33 PM] x264 [info]: frame P:217 Avg QP:42.45 size: 23213 PSNR Mean Y:37.61 U:44.98 V:43.58 Avg:38.92 Global:38.67
[2:57:33 PM] x264 [info]: frame B:399 Avg QP:44.13 size: 6264 PSNR Mean Y:37.21 U:45.02 V:43.55 Avg:38.55 Global:38.26
[2:57:33 PM] x264 [info]: consecutive B-frames: 1.5% 8.7% 87.2% 2.6%
[2:57:33 PM] x264 [info]: mb I I16..4: 20.4% 67.7% 11.9%
[2:57:33 PM] x264 [info]: mb P I16..4: 7.3% 14.0% 0.6% P16..4: 42.7% 6.7% 7.4% 0.1% 0.0% skip:21.3%
[2:57:33 PM] x264 [info]: mb B I16..4: 0.8% 0.7% 0.0% B16..8: 36.9% 4.1% 0.7% direct: 2.1% skip:54.7% L0:34.5% L1:59.9% BI: 5.6%
[2:57:33 PM] x264 [info]: field mbs: intra: 46.0% inter:52.2% skip:24.4%
[2:57:33 PM] x264 [info]: 8x8 transform intra:62.6% inter:86.0%
[2:57:33 PM] x264 [info]: direct mvs spatial:52.4% temporal:47.6%
[2:57:33 PM] x264 [info]: coded y,uvDC,uvAC intra: 37.5% 27.1% 3.8% inter: 9.0% 4.4% 0.6%
[2:57:33 PM] x264 [info]: i16 v,h,dc,p: 36% 22% 22% 20%
[2:57:33 PM] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 14% 7% 39% 5% 7% 8% 7% 6% 7%
[2:57:33 PM] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 24% 18% 9% 6% 10% 11% 9% 7% 7%
[2:57:33 PM] x264 [info]: i8c dc,h,v,p: 69% 14% 13% 3%
[2:57:33 PM] x264 [info]: ref P L0: 65.6% 24.4% 5.7% 2.3% 0.6% 0.5% 0.5% 0.3%
[2:57:33 PM] x264 [info]: ref B L0: 73.9% 23.3% 2.1% 0.3% 0.2% 0.2%
[2:57:33 PM] x264 [info]: ref B L1: 77.5% 22.5%
[2:57:33 PM] x264 [info]: SSIM Mean Y:0.9537058 (13.345db)
[2:57:33 PM] x264 [info]: PSNR Mean Y:37.367 U:45.012 V:43.568 Avg:38.692 Global:38.411 kb/s:3021.14

[Blue_MiSfit]

I don't trust metrics at all, except for very objective tests and only when the encoders in question have been set to optimize for it.

In any case, it's not machines watching your video but rather subjective human eyes

[cagali]

I don't trust metrics at all, except for very objective tests and only when the encoders in question have been set to optimize for it.

In any case, it's not machines watching your video but rather subjective human eyes

Yea, I agree that we cannot only rely metrics, or one metric.

I played both clips and thought one of them might be slightly better. But I did not state that as it might create bias.

I am now looking at ms-ssim, a more sophisticated version of ssim. The implementation that I am using is a plugin for ImageJ.

Any comments are welcomed. Thank you.

[foxyshadis]

If the noise is more than ±1 everywhere across the picture then there's no point in 10bit. In normal encoding, 10bit is for smoother gradients and nothing else, and those are only common with CG or after heavy filtering.

(For lossless and near-lossless, 10bit is extremely useful for keeping extra detail so you can further filter the images later, while minimizing space, if the source is higher than 8bit and you have a 16bit workflow.)

[cagali]

If the noise is more than ±1 everywhere across the picture then there's no point in 10bit. In normal encoding, 10bit is for smoother gradients and nothing else, and those are only common with CG or after heavy filtering.

(For lossless and near-lossless, 10bit is extremely useful for keeping extra detail so you can further filter the images later, while minimizing space, if the source is higher than 8bit and you have a 16bit workflow.)

Agree. The source I stated above is 1080i, however in some other tests using 1080p, 10-bit gives slightly better PSNR and SSIM than 8-bit, and only downside is playback compatibility and encoding speed.

(Your signature is cool, as it is getting real in my city.)

[Boulder]

But bitrate-wise, is it not better to do a 10-bit encode if you have the means to decode it? Or are the crf values not the same between 8- and 10-bit encodes?

[foxyshadis]

But bitrate-wise, is it not better to do a 10-bit encode if you have the means to decode it? Or are the crf values not the same between 8- and 10-bit encodes?

If you're going to flip every option for a bit of quality, you might as well flip that one too. There's a speed hit with 10bit encoding, too.