x264 r968 & CRF encoding changes

[LiFe]

Heya Folk,

The changes for r968 state:
Move adaptive quantization to before ratecontrol, eliminate qcomp bias

This change improves VBV accuracy and improves bit distribution in CRF and 2pass.

Instead of being applied after ratecontrol, AQ becomes part of the complexity measure that ratecontrol uses.

This allows for modularity for changes to AQ; a new AQ algorithm can be introduced simply by introducing a new aq_mode and a corresponding if in adaptive_quant_frame.

This also allows quantizer field smoothing, since quantizers are calculated beofrehand rather during encoding.

Since there is no more reason for it, aq_mode 1 is removed. The new mode 1 is in a sense a merger of the old modes 1 and 2.

WARNING: This change redefines CRF when using AQ, so output bitrate for a given CRF may be significantly different from before this change!

I use crf 22 for all my encodings and I was wondering if someone (darkshikari) could explain the expected changes to crf for me, and if anyone had done any tests?

Thanks!

[Sagekilla]

Simple: Use an older build (967) and encode using your normal settings, then compare with an encode using the newer build. I'll do some testing later on when I have time.

[Dark Shikari]

Overall the bitrates should stay the same, but some sources may change drastically. Personally, I think the new values are a "more consistent" measure of quality. One example of this is the classic "parkrun.yuv" 720p video; bitrate dropped from 40 megabits to about 26 megabits at CRF18, putting the P-frame quantizers at over 26! However, it still looks almost completely transparent in single-frame comparisons, which means clearly there was way too much bitrate being given before.

[muaddib.dk]

My initial test doesn't look too promising. Had an old CRF encode on harddrive done with r928, so tried and re-do the encode with r968. Same avs, same x264 cmdline.
The picture quality is (by my eyes) in-distuingishable (sp?) between the 2 encodes, though very slight differences can be detected.

cmd-line and x264 stats:

x264.exe --level 4.1 --crf 18 --threads auto --bframes 3 --b-pyramid --bime --weightb --b-rdo --me umh --ref 5 --mixed-refs --subme 7 --trellis 1 --analyse all --8x8dct --aq-strength 1.1 --deadzone-inter 8 --deadzone-intra 4 --no-fast-pskip --progress --output serenity.264 serenity.avs

R928:
x264 [info]: slice I:2785 Avg QP:13.58 size: 86987
x264 [info]: slice P:97336 Avg QP:15.55 size: 39642
x264 [info]: slice B:71100 Avg QP:19.14 size: 10869
x264 [info]: SSIM Mean Y:0.9855848
x264 [info]: PSNR Mean Y:45.613 U:49.894 V:51.459 Avg:46.287 Global:45.518 kb/s:5459.62
encoded 171221 frames, 5.89 fps, 5459.72 kb/s

R968:
x264 [info]: slice I:2785 Avg QP:12.56 size:102739
x264 [info]: slice P:97338 Avg QP:14.12 size: 55432
x264 [info]: slice B:71104 Avg QP:17.36 size: 17673
x264 [info]: SSIM Mean Y:0.9876417
x264 [info]: PSNR Mean Y:46.610 U:50.722 V:52.195 Avg:47.256 Global:46.095 kb/s:7772.38
encoded 171227 frames, 6.60 fps, 7772.48 kb/s


the difference in frame count I have so far contributed to the video being demuxed with evodemux for R928 encode, and eac3to for the R968 encode.

My conclusion so far is that the new ways introduced by 968 has bumped my encode up by roughly 2300kbit for the same quality. Haven't had time to test other sources yet, so can't tell if this is one of those sources that will change drastically or if it's a general issue.

[Dark Shikari]

My conclusion so far is that the new ways introduced by 968 has bumped my encode up by roughly 2300kbit for the same qualityAnd your conclusion would be wrong, because if they're indistinguishable, that means you've reached the threshold of visual losslessness, so you can't compare anymore.

[qyqgpower]

psy-rd would bump bitrate a lot at same crf value, so make sure you are using same psy-rd settings with these two encodes.

[Ranguvar]

psy-rd would bump bitrate a lot at same crf value, so make sure you are using same psy-rd settings with these two encodes.

I believe it won't anymore after this update.

[Sharktooth]

why not?

[Ranguvar]

Sorry, thought the update also took into account Psy-RDO, but it's just AQ it now accounts for. Still, if anyone wants to test :)

[JarrettH]

I bought The Fall on DVD and tried some encoding with it.

r967 - 2.3gb (2,100 kbps video, AC3 5.1)
r968 - 2.6gb (2,700 kbps video, AC3 5.1)
r973 - 2.17gb (2,100 kbps video, AC3 5.1)
r977 - same!

[Quark.Fusion]

My initial test doesn't look too promising. Had an old CRF encode on harddrive done with r928, so tried and re-do the encode with r968. Same avs, same x264 cmdline.
The picture quality is (by my eyes) in-distuingishable (sp?) between the 2 encodes, though very slight differences can be detected.

cmd-line and x264 stats:

x264.exe --level 4.1 --crf 18 --threads auto --bframes 3 --b-pyramid --bime --weightb --b-rdo --me umh --ref 5 --mixed-refs --subme 7 --trellis 1 --analyse all --8x8dct --aq-strength 1.1 --deadzone-inter 8 --deadzone-intra 4 --no-fast-pskip --progress --output serenity.264 serenity.avs

R928:
x264 [info]: slice I:2785 Avg QP:13.58 size: 86987
x264 [info]: slice P:97336 Avg QP:15.55 size: 39642
x264 [info]: slice B:71100 Avg QP:19.14 size: 10869
x264 [info]: SSIM Mean Y:0.9855848
x264 [info]: PSNR Mean Y:45.613 U:49.894 V:51.459 Avg:46.287 Global:45.518 kb/s:5459.62
encoded 171221 frames, 5.89 fps, 5459.72 kb/s

R968:
x264 [info]: slice I:2785 Avg QP:12.56 size:102739
x264 [info]: slice P:97338 Avg QP:14.12 size: 55432
x264 [info]: slice B:71104 Avg QP:17.36 size: 17673
x264 [info]: SSIM Mean Y:0.9876417
x264 [info]: PSNR Mean Y:46.610 U:50.722 V:52.195 Avg:47.256 Global:46.095 kb/s:7772.38
encoded 171227 frames, 6.60 fps, 7772.48 kb/s


the difference in frame count I have so far contributed to the video being demuxed with evodemux for R928 encode, and eac3to for the R968 encode.

My conclusion so far is that the new ways introduced by 968 has bumped my encode up by roughly 2300kbit for the same quality. Haven't had time to test other sources yet, so can't tell if this is one of those sources that will change drastically or if it's a general issue.

Look for AvgQP and SSIM in your log — quality is different, maybe you need to raise CRF value by 1 or 1.5.

[LoRd_MuldeR]

I bought The Fall on DVD and tried some encoding with it.

r967 - 2.3gb (2,100 kbps video, AC3 5.1)
r968 - 2.6gb (2,700 kbps video, AC3 5.1)

So big difference! More than 25% higher

25% bigger file at same CRF doesn't say anything! Metrics (PSNR, SSIM, etc.) don't help much either...

The question is:
Is the increase in quality worth the 25% increase in filesize subjectively :confused:

[Selur]

@kemuri-_9: looking at the kbps it's more than 25% ((2700-2100)/2100)
-
I do agree with LordMulder, the subjective quality is what's counts when it's about crf or constant quantizer,...

[ZombiePimp]

IMO the 25% is not worth it. I've been happy with CRF18, but now the file sizes are just too big, with no benefit. It's like CRF19 is the new CRF18, or maybe even CRF19.5.

[LoRd_MuldeR]

IMO the 25% is not worth it. I've been happy with CRF18, but now the file sizes are just too big, with no benefit. It's like CRF19 is the new CRF18, or maybe even CRF19.5.

If CRF 18 is now bigger, but looks no different, then you already are at the point where your encode is transparent (visually lossless).
It was said that the "CRF meanings" did change! So if CRF 19 or 19.5 now behaves similar to CRF 18 in older revisions, we simply have to learn that.
That's it. I see no problem...

[ZombiePimp]

I agree, not really a problem. It's just going to be a surprise for all the CRF 18 users that don't keep up with the changes.

[LiFe]

Buggerit, CRF 22 for me was just transparent and gave the smallest file size possible. Took me a lot of testing and calculations to find it, now I've gotta do it again!

r-older - 753 kbit/s
r968 - 1203 kbit/s

Double the bit rate is a huge difference. Buggerit. Millennium hand a shrimp.

[refulgentis]

Buggerit, CRF 22 for me was just transparent and gave the smallest file size possible. Took me a lot of testing and calculations to find it, now I've gotta do it again!

r-older - 753 kbit/s
r968 - 1203 kbit/s

Double the bit rate is a huge difference. Buggerit. Millennium hand a shrimp.
Let me know what you find out...I used ~22 too (I used 23, but I should have been using 23 :P)

EDIT: I'm seeing ~24 as the sweet spot, probably could go a bit lower but with disk space the way it is nowadays, there's no point in compromising quality

[foxyshadis]

Find the new CRF that gives you about the same bitrate as your old CRF. Then you don't even have to visually check anything, just make a lot of 30-second encodes and spot-check the one that matches.

[JarrettH]

r973 - 2.17gb (2,100 kbps video, AC3 5.1)
r977 - same!

Lol think I'll just wait till this plethora of changes are done
Oh all these were done with the HQ preset (1 pass CQ)

[LiFe]

Hmm, comparing crf encodes with r979 and r967 and a live action clip from Kenny (no steady cam) and CRF's from 25 to 20 all have SSIM and PSNR's with less than 1% difference.

All the bitrates dropped by almost 10%.

r967 - 2.3gb (2,100 kbps video, AC3 5.1)
r968 - 2.6gb (2,700 kbps video, AC3 5.1)
r973 - 2.17gb (2,100 kbps video, AC3 5.1)
r977 - same!

I find it very bizarre that 968 caused such a bit rate blow out, but something since then has brought it back in line with before the drastic 968 VBV changes....


Encode SSIM PSNR Avg PSNR Global Bitrate
20 New 0.9829700 47.201 44.81 2,867.28
21 New 0.9808333 46.629 44.204 2,439.70
22 New 0.9786104 46.075 43.628 2,060.77
23 New 0.9763189 45.543 43.073 1,729.38
24 New 0.9739465 45.028 42.54 1,441.90
25 New 0.9715173 44.521 42.02 1,195.46

20 Old 0.9827860 46.695 44.827 3,158.01
21 Old 0.9808124 46.177 44.279 2,696.94
22 Old 0.9786387 45.657 43.729 2,281.75
23 Old 0.9763377 45.147 43.189 1,910.60
24 Old 0.9740294 44.657 42.669 1,594.55
25 Old 0.9717060 44.189 42.166 1,318.96

[Quark.Fusion]

I think you should measure % from 1-SSIM

[LiFe]

I'm gonna stay with crf 22. Slight increase in quality over the old crf 22, but the bitrate increase is much more acceptable with the r97x series.

ie.
Original Encode: 753 kbit/s;
r968 Encode: 1203 kbit/s
r979 Encode: 878 kbit/s
r979 w new B Frame decision: 868 kbit/s

Fades look much better.

[uladzislau]

Hi. I don't know from which build exactly, but right now with 979 and up I'm getting encoded file size ~15-20% bigger then before. And this is with exactly same settings and old b-frame fast method. Switching to the newer b-frame methods gives files sizes ~25% bigger.

Example:
Futurama episode 1-1
Pre 979 build b-frame 1 crf 18 = 136MB
979 build and up b-frame 1 crf 18 = 200MB
979 build and up b-frame 2 crf 18 = 220MB

Now let's try lower crf values:
979 build and up b-frame 2 crf 19 = 186MB
979 build and up b-frame 2 crf 20 = 168MB

And I don't see any quality differences really. Any advises? Thanks.

[Sharktooth]

@Life: metrics do not measure quality... how many times shall i say that?
expecially when psy-rd/psy-trellis are enabled, metrics become USELESS.

@uladzislau: it you dont see any difference it means you reached transparency so you can continue rising the CRF until you see a difference.

[Dark Eiri]

n00b question: using CRF 22 in Extra Q. and Fast profiles on MeGUI will achieve the exact same results in quality, only the filesize will be altered, right?

[nm]

Probably not exactly the same quality, but pretty close.

[Dark Eiri]

OK then. I asked because I was encoding a concert and decided to do this test with Extra Q. and Fast presets. The Fast encoded 5x faster and the quality was visually the same, and the filesize wasn't that bigger (~5 MB bigger for a 3min 10sec video). I'll keep testing to see if the difference is considerable, I don't like to trade quality for speed, but in this case it was pretty much win/win.

Thank you for answering!

[LoRd_MuldeR]

n00b question: using CRF 22 in Extra Q. and Fast profiles on MeGUI will achieve the exact same results in quality, only the filesize will be altered, right?

AFAIK if you change your settings, then the CRF meaning will also be changed.

Hence the same CRF value will produce different quality with "fast" and "slow" settings.
No idea how much the difference will be though...

[akupenguin]

Most of x264's speed-vs-quality options are tuned to preserve RD lambda, not quality, so they tend to both reduce bitrate and improve quality. The difference between fastest and slowest settings is maybe 1 point of CRF, in addition to the change in bitrate.

AQ, Psy-RD, and CQM are not included in the above statement as they aren't speed-vs-quality options. These can have much larger effects on perceptual quality (for better or worse depending on whether you screw them up ;))