New x264 RDO is strange ... !!!

[Sagittaire]

Test Main Profil use this CLI

x264.exe --bframe 2 --ref X --mixed-refs --filter 0:0 --bitrate 447 --pass 1 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.30 --analyse "all" --weightb --me "umh" --subme 6/7 --progress -o NUL Encodage.avs
x264.exe --bframe 2 --ref X --mixed-refs --filter 0:0 --bitrate 447 --pass 2 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.30 --analyse "all" --weightb --me "umh" --subme 6/7 --progress -o x264MP-447.mp4 Encodage.avs


Test High Profil use this CLI

x264.exe --bframe 2 --8x8dct --ref X --mixed-refs --filter 0:0 --bitrate 447 --pass 1 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.30 --analyse "all" --weightb --me "umh" --subme 6/7 --progress -o NUL Encodage.avs
x264.exe --bframe 2 --8x8dct --ref X --mixed-refs --filter 0:0 --bitrate 447 --pass 2 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.30 --analyse "all" --weightb --me "umh" --subme 6/7 --progress -o x264MP-447.mp4 Encodage.avs


and here the strange result

RDO subme=7 Main Profil
ref=2 PSNR=40.406
ref=4 PSNR=40.361
ref=6 PSNR=40.316
ref=8 PSNR=40.295

RDO subme=6 Main Profil
ref=2 PSNR=40.564
ref=4 PSNR=40.609
ref=6 PSNR=40.624
ref=8 PSNR=40.629

RDO subme=7 High Profil
ref=2 PSNR=40.842
ref=4 PSNR=40.884
ref=6 PSNR=40.887
ref=8 PSNR=40.894

RDO subme=6 High Profil
ref=2 PSNR=40.800
ref=4 PSNR=40.836
ref=6 PSNR=40.848
ref=8 PSNR=40.853

1) The subme 7 seem to be optimized only for high profil
2) Strange comportement with new RDO & multi-ref for Main Profil : higher ref implicate lower PSNR

[akupenguin]

I said it wasn't optimized for nor even tested with --mixed-refs. I didn't expect it to reduce quality, but ...
And yes, all of my tests were high profile. The difference here is probably that we decide dct size after partition size, which means that (without subme=7) we test several modes with 4x4dct and only one with 8x8dct. So subpel RD is a little bit more likely to improve the 8x8 match.

[bond]

i now compared subme 5-7 myself with the following commandline:

x264 --pass 2 --bitrate 770 --bframe 3 --b-pyramid --ref 5 --mixed-refs --filter -5:-5 --trellis 2 --8x8dct --analyse all --weightb --progress -o test_r361_m5_trellis2.mp4 matrix_LR_720x288_dgdecode.avs

speed:
m5: 3.05 fps
m6: 2.30 fps (- 25%)
m7: 1.64 fps (- 46%)

psnr:
m5: Avg:44.216 Global:43.917
m6: Avg:44.334 Global:44.047
m7: Avg:44.347 Global:44.061

so the psnr values look normal

visual quality:
visually it also was as expected. m6 was slightly better than m5 but not that much. m7 is imho better than m6

[Sagittaire]

@ bond

me6 is optimized for Main Profil and not me7
me7 seem optimized (if your visual impression is good) for High Profil and not me6

try to compare directly me6 and me7 encoding
- if PSNR/SSIM are very high (something like 50 dB/90 or more) then me6 and me7 will be in practice transparent for eyes and your visual impression is probaly false
- if metric are low then your visual is perhaps good : Make SSIM test. Perhaps in this case that SSIM will confirm your visual impression with better result for me7

[bond]

me6 is optimized for Main Profil and not me7why that?

try to compare directly me6 and me7 encoding
- if PSNR/SSIM are very high (something like 50 dB/90 or more) then me6 and me7 will be in practice transparent for eyes and your visual impression is probaly false
- if metric are low then your visual is perhaps good : Make SSIM test. Perhaps in this case that SSIM will confirm your visual impression with better result for me7 i posted the psnr values too in my post

[Sagittaire]

me6 is optimized for Main Profil and not me7

I said it wasn't optimized for nor even tested with --mixed-refs. I didn't expect it to reduce quality, but ...
And yes, all of my tests were high profile. The difference here is probably that we decide dct size after partition size, which means that (without subme=7) we test several modes with 4x4dct and only one with 8x8dct. So subpel RD is a little bit more likely to improve the 8x8 match.

Main profil + mixed-refs + high value for ref + me7 done bad result for OPSNR


Personnaly I can't see real difference between me6 and me7. There are perhaps little local difference but IMO it's impossible to make real overall comparison and I make these test:

- Comparison between source and encoding me6 HP
SSIM = 81.29

- Comparison between source and encoding me7 HP
SSIM = 81.37

- Direct comparison between me6 and me7
SSIM = 89.17

Direct comparison say "In this case there are not overall visibles differences between me6 and me7" because direct SSIM comparison between is very high.

If I want the same SSIM in source-encoding comparison I must use crf = 12 and q12 is very low quant and very high quality : Your visual impression is probaly "placebo impression" ...

RDO is only search in PSNR sense. RDO choose always the best PSNR decision. They are not HVS decision in RDO ... and me7 is very little better than me6 for PSNR.

[bond]

i now tested for subme 7 the patches:
p8rd.3 vs p8rd.9 (+ with and without b-rdo)

v9 (without b-rdo) is definitely better than v3
it shows more details in most cases (in faces, on the clothes...), only in very few cases things looked slightly sharper with v3 (one time in smiths face, also the detailed walls in the lobby shootout look slightly sharper with v3)
i have the feeling v9 is more crispy/grainy than v3
flat walls dont seem to have changed
the flying pieces of the wall scene still has problems with m7 (m6 is also not so good on that compared to m5)

speed:
m7.3: 1.64 fps
m7.9: 1.57 fps
m7.9+b-rdo: 1.41 fps

psnr:
m7.3: Avg:44.347 Global:44.061
m7.9: Avg:44.357 Global:44.068
m7.9+b-rdo: Avg:44.432 Global:44.148

when looking at b-rdo (tested with v9) i have to say i dont think it really helps a lot, i would more say the quality stays ~ the same. sometimes no b-rdo being better some time b-rdo. mybe overall b-rdo is better but i cant say this for sure
i funnily saw that b-rdo seems to have problems with keeping the details in brown hair!? also on the "flying wall pieces" scene it didnt perform any better than no b-rdo. flat walls also dont change



i still feel the need to point out that imho there is no really that big difference between m5-7 (compared to the speed decrease)
i mean m5 already provides very nice results (being even better in a few scenes than m6. eg on the flying wall pieces scenes it also performs overall better than m7 imho, but that seems to be a hard scene for rdo), so if you are in a hurry with a slow pc you might want to choose m5. m6 is propably a good compromise, as it also performs better than m7 in some scenes