x264: bit rate on second pass too low - any way to force "minimum bitrate"?

[Lyris]

I'm encoding a documentary to go out to Blu-ray, which features very smooth, denoised, low motion video footage (interviews with still camera), with scenes from a grainy film intercut.

I specified 14mbps as the bit rate, 40mbps as the max (as per the BD spec) and a VBV buffer of 30mbps.

set brate=14000
set speed=slow
set tune=film

x264 --bitrate %brate% --preset %speed% --tune %tune% --bluray-compat
--vbv-maxrate 38000 --vbv-bufsize 30000 --level 4.1 --keyint 24 --open-gop
--slices 4 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1
-o "%of%" "%if%"


On the first pass, x264 gives a fairly constant bit rate. The result looks good.

On the second pass, during some of the low motion video scenes, it applies a bit rate which is too low - even as low as 1.5mbps at times. The resulting video, while outstandingly impressive for that low a bit rate, has visible artefacts. And since I'm going to optical disc and not download, I'd rather have the better quality.

I got around this by just using the first pass encode. Alternatively I had success using --zones and forcing a bitrate multiplier on the affected areas, but this was pretty time consuming.

I did try setting a max quantizer (--qpmax), but that seems unpredictable; I started getting buffer underflows.

I realise x264 has no minimum bitrate setting. Could a ratecontrol expert fill me in on why that is? And possibly, give a suggestion on how to get around the issue of it applying too low a bitrate without resorting to manual adjustment?

For comparison, I used Cinema Craft SP3 to do the PAL DVD version, and it nailed exactly what I wanted: 8-9mbps for the movie scenes, and 3-4mbps for the video inserts. (The visual quality is consistent). The bitrate graph looked like an up-down-up-down wave. Is there any way to get similar behaviour out of x264?

And also, out of curiosity - why would it be doing this when A) the bit rate was clearly specified as 14mbps, and the rate it used for these scenes - 1/10th of that - was inadequate? It seems to have been fooled by the low amount of motion?

[DeeGee]

If I'm understanding correctly a new commit in x264 sources, your problem might be already fixed. You just need to find a new enough x264 build to test if it helps.
https://github.com/DarkShikari/x264-devel/commit/b966ce11b154dbd5f6e2fbbdb1bab6c2b4610454

[edit] And as it seems, I didn't understand it correctly... I guess I'll go back to lurking mode :)

[LoRd_MuldeR]

If I'm understanding correctly a new commit in x264 sources, your problem might be already fixed. You just need to find a new enough x264 build to test if it helps.
https://github.com/DarkShikari/x264-devel/commit/b966ce11b154dbd5f6e2fbbdb1bab6c2b4610454

This commit is about VBV, so it's not specific to 2-Pass encoding.

Also all that VBV does (if enabled) is ensuring that the VBV buffer won't underflow, i.e. it effectively limits the maximum bitrate.

Though it has to be noted that VBV allows (short) bitrate spikes above VBV Maxrate...

BTW: I don't think VBV is the problem here. If VBV effects quality, you would rather see this in "high complexity" (high bitrate) scenes.

I'm encoding a documentary to go out to Blu-ray, which features very smooth, denoised, low motion video footage (interviews with still camera), with scenes from a grainy film intercut.

I specified 14mbps as the bit rate, 40mbps as the max (as per the BD spec) and a VBV buffer of 30mbps.

[code]On the first pass, x264 gives a fairly constant bit rate. The result looks good.
On the second pass, during some of the low motion video scenes, it applies a bit rate which is too low - even as low as 1.5mbps at times.

Well, this sounds exactly like what one does expect from a 2-Pass encode.

The first pass is basically 1-Pass ABR encode, so no big surprise that the resulting bitrate is relatively constant.

In the second pass, 2-Pass rate-control is applied to distribute the available bits over the complete video. And of course "low complexity" scenes will get fewer bits than "high complexity" scenes. If you think too few bits are assigned to the "low complexity" scenes, you'll have to tweak x264's 2-Pass RC algorithm.

I think "--qcomp" might be one parameter you could try to tweak here...

I did try setting a max quantizer (--qpmax), but that seems unpredictable; I started getting buffer underflows.

If you restrict "--qpmax" too much with VBV enabled, at some point the VBV algorithm won't be able to do its job anymore.

Imagine the VBV algorithm realizes that we need a higher quantizer ("stronger" compression) to avoid an underflow at a certain position, but your settings simply don't allow an even higher quantizer. VBV won't be able to react to the impending underflow. The result will be an actual underflow...

[pandy]

denoised

then add some noise to bias bitrate.

[Dark Shikari]

If I'm understanding correctly a new commit in x264 sources, your problem might be already fixed. You just need to find a new enough x264 build to test if it helps.
https://github.com/DarkShikari/x264-devel/commit/b966ce11b154dbd5f6e2fbbdb1bab6c2b4610454This commit most likely has nothing to do with the problem.
I did try setting a max quantizer (--qpmax), but that seems unpredictable; I started getting buffer underflows.Setting QP-max will cause buffer underflows because it prevents VBV from doing its job.

If you have a small sample of the video that triggers the problem, please post it and a set of encoding settings that consistently produce the problem (along with pointing me towards whatever frame numbers have the issue).

If you really need a constant bitrate video at the expense of quality, try using CBR mode.

Let me clear up some misconceptions here, because the previous post was extremely misleading and largely incorrect:

Though it has to be noted that VBV allows (short) bitrate spikes above VBV Maxrate...Only if you incorrectly define VBV maxrate.BTW: I don't think VBV is the problem here. If VBV effects quality, you would rather see this in "high complexity" (high bitrate) scenes.This is not necessarily true.The first pass is basically 1-Pass ABR encode, so no big surprise that the resulting bitrate is relatively constant.This is not necessarily true. 1-pass bit distribution may be suboptimal, but it is not "relatively constant".In the second pass, 2-Pass rate-control is applied to distribute the available bits over the complete video. And of course "low complexity" scenes will get fewer bits than "high complexity" scenes. If you think too few bits are assigned to the "low complexity" scenes, you'll have to tweak x264's 2-Pass RC algorithm.The part of x264's RC algorithm that affects bit distribution, macroblock-tree, affects both passes identically. It has absolutely nothing to do with 1-pass or 2-pass.I think "--qcomp" might be one parameter you could try to tweak here...This affects both passes identically.

[G_M_C]

Isnt it an idea to use one-pass crf mode instead of 2-pass bitrate mode ?

[Dark Shikari]

Isnt it an idea to use one-pass cq mode instead of 2-pass bitrate mode ?CRF and 2-pass use the exact same bit distribution algorithm. If you do a CRF encode and a 2-pass encode at the same bitrate, the results will be nearly identical.

Hypothetically, there could be a bug in the 2-pass code that didn't show up in CRF, but outside of a hypothetical bug workaround, this is probably not a good suggestion.

[G_M_C]

CRF and 2-pass use the exact same bit distribution algorithm. If you do a CRF encode and a 2-pass encode at the same bitrate, the results will be nearly identical.

Hypothetically, there could be a bug in the 2-pass code that didn't show up in CRF, but outside of a hypothetical bug workaround, this is probably not a good suggestion.

Ok, I thought crf is comparable to a constant quality mode.

My idea is that if you set an high enough quality (low enough setting of --crf, say --crf 12), based on TS' example, the high detail and the denoised scenes would come out with comparable quality (both crf 12); while still maintaining BD specs, if other options arent changed. This is what TS' actually wants isnt it ?

[Dark Shikari]

Ok, I thought crf is comparable to a constant quality mode.It is. 2-pass is also constant quality. You just ask x264 for a bitrate instead of a quality.

Again, they use the same bit distribution algorithm.

[G_M_C]

Hmmm, i see. Thx for the clarification DS !

[Lyris]

then add some noise to bias bitrate.

Hmm, I shouldn't have to change the source on a technical or aesthetic level to fool the rate control algorithm. (I don't mean that to sound ungrateful, BTW).

If you have a small sample of the video that triggers the problem, please post it and a set of encoding settings that consistently produce the problem (along with pointing me towards whatever frame numbers have the issue).
Thanks for looking into this, DS - PM incoming.

[Sagittaire]

I make actually test on x264 Rate Control and optimized profil for Blu Ray encoding. You can try that and make little feedback please:

x264.exe --slow-firstpass --bframe 1 --b-pyramid "strict" --b-adapt 2 --weightp 1 --ref 4 --direct auto --deblock -1:-3 --bitrate 14000 --pass 1 --stats "test.log" --no-mbtree --qcomp 0.25 --ipratio 1.00 --pbratio 1.00 --min-keyint 1 --partitions "all" --me "hex" --subme 7 --no-fast-pskip --no-dct-decimate --deadzone-inter 2 --deadzone-intra 2 --trellis 1 --psy-rd 1.0:0.25 --aq-mode 1 --aq-strength 0.5 --bluray-compat --vbv-maxrate 40000 --vbv-bufsize 30000 --vbv-init 0.9 --level 4.1 --keyint 24 --open-gop --slices 4 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --psnr --ssim -o test.264 Lossless.avs

x264.exe --bframe 1 --b-pyramid "strict" --weightp 1 --ref 4 --direct auto --deblock -1:-3 --bitrate 14000 --pass 3 --stats "test.log" --no-mbtree --qcomp 0.25 --ipratio 1.00 --pbratio 1.00 --min-keyint 1 --partitions "all" --me "umh" --subme 10 --no-fast-pskip --no-dct-decimate --deadzone-inter 2 --deadzone-intra 2 --trellis 2 --psy-rd 1.0:0.25 --aq-mode 1 --aq-strength 0.5 --bluray-compat --vbv-maxrate 40000 --vbv-bufsize 30000 --vbv-init 0.9 --level 4.1 --keyint 24 --open-gop --slices 4 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --psnr --ssim -o test.264 Lossless.avs

[jethro]

Min bitrate is a heavy-handed tool to enforce a minimal level of quality. Maybe it is possible to set minimal byte size for individual macroblocks to prevent them being quantized to oblivion?

[Lyris]

Min bitrate is a heavy-handed tool to enforce a minimal level of quality.
I understand that there may be arguments against it in that it's not a "neat" solution, but the fact is, it works very well to avoid losing quality in CCE when I'm using it to produce DVDs. Keep in mind the goal is to fill both (or one) layer of an optical disc, any savings beyond that are nearly irrelevant when it comes to physical media. If there's a potentially wasteful but more consistent choice, that's the one that makes most sense in this area :)

Haven't had any luck with other encoding parameters yet. It looks like I might need to go through and manually assign a higher bit rate to these scenes in order to fit this on the disc.

[Sagittaire]

even with really low qcomp ... ???
With qcomp = 0 you have really constant bitrate.
Try my profil with lower qcomp.

[Lyris]

I did try some of your settings. That resulted in the opposite situation: the grainy film clips degraded in quality, whereas the interviews looked better (in line with what DS told me to expect).

What actually helped in the end was using the --psnr setting. Probably not ideal, but the closest thing I could find which worked in time for submission.

[Sagittaire]

I did try some of your settings. That resulted in the opposite situation: the grainy film clips degraded in quality, whereas the interviews looked better (in line with what DS told me to expect).

What actually helped in the end was using the --psnr setting. Probably not ideal, but the closest thing I could find which worked in time for submission.

Well try to find the best value of qcomp for your eyes.
Anyway in your case I think that the average must be higher to have acceptable quality in complete area.
--pnsr command line don't mean that the profil is optimised for psnr but just that psnr is calculate.

[LoRd_MuldeR]

If he indeed means that using "--psnr" solved his problem, then this would be a typical placebo. Adding "--psnr" does not change the output at all.

If, however, he means that he is using "--tune psnr" (which optimizes the output for "--psnr"), then it's not very surprising that he is seeing a significant difference.

That's because "--tune psnr" essentially disables all the Psy optimizations of x264! But completely disabling the Psy optimizations usually is a very bad choice :eek:

(Instead I would keep the Psy optimizations enabled and instead try to tweak the Psy parameters a little, e.g. "--psy-rd" or "--aq-strength")

[Lyris]

I didn't know that - I was confusing it with the --tune psnr you mentioned. Thanks for the clarification.

In that case, it'd have to be the extra zones (with bitrate multipliers) I specified which made the difference.

[nibus]

You might also try disabling mbtree (--no-mbtree). This will give more bitrate to those denoised portions and limit qcomp swinging the bitrate so much.

[benwaggoner]

I understand that there may be arguments against it in that it's not a "neat" solution, but the fact is, it works very well to avoid losing quality in CCE when I'm using it to produce DVDs.
I argue that min bitrate was a workaround for design limitations in older MPEG-2 encoders. An encoder should be smart enough to lower QP for PSNR-easy portions of video when neccessary for a constant visual quality.

Sufficiencly good psychovisual optimizations should eliminate the need for a minimum bitrate.