vbv rate limiting optimizations

[MarkZ]

So, I've been using vbv rather strictly for my encodes to prevent the max bitrate from exceeding the hardware capabilities of my player. My player maxes out with a buffer/rate setting of 1500/2500 for 720x480 encodes. I'd like to further improve the quality of these encodes if possible.

Two potential avenues seem to be to alter the ipratio/pbratio, and to modify the qcomp setting (note: I'm using CRF here...).

With some light experimentation, I've decided to increase qcomp to 0.7 or higher with the idea that since vbv is clamping the max bitrate I can reduce the variance of the bitrate to reduce the amount of frames that exceed the vbv threshold. I'd like to verify this by finding software that will display a bitrate histogram, but I haven't found this yet. Suggestions?

I'm thinking about making similar ipratio/pbratio changes to perhaps complement my tight vbv settings. The idea would be to increase quality slightly in p- and b-frames to compensate for the reduction in quality of i-frames that results from vbv rate limiting. Does this make sense? What values would you try?

[akupenguin]

I've decided to increase qcomp to 0.7
Does not make sense. For any scene that is vbv limited both before and after your qcomp change, you haven't greatly altered the bit distribution. For any scene that is vbv limited before and not vbv limited after, you reduced quality more than necessary. For any scene of less than average complexity, you increased quality. In all 3 cases, feel free to adjust qcomp if the result actually looks better to you, but this is completely unrelated to vbv.

I'm thinking about making similar ipratio/pbratio changes
Does not make sense. If you have to take bits away from I-frames in order to fit in the vbv, then x264's vbv code will do so. If you don't have to, then you shouldn't.

[MarkZ]

Does not make sense. For any scene that is vbv limited both before and after your qcomp change, you haven't greatly altered the bit distribution. For any scene that is vbv limited before and not vbv limited after, you reduced quality more than necessary. For any scene of less than average complexity, you increased quality. In all 3 cases, feel free to adjust qcomp if the result actually looks better to you, but this is completely unrelated to vbv.

The bolded part was my rationale. Although the vbv limits diminish quality during complex scenes, the hope was that I could use those extra bits to help improve quality during less complex scenes. My understanding is that this is precisely what qcomp was designed to do. I reasoned that a vbv-limited encode is an ideal situation for this since vbv is essentially reducing the amount of bitrate swing.

Does not make sense. If you have to take bits away from I-frames in order to fit in the vbv, then x264's vbv code will do so. If you don't have to, then you shouldn't.

Not sure I understand. Does this mean that x264 is not basing p-frame quality on the resultant vbv-limited I-frame quality, but rather the I-frame quality as if it wasn't vbv limited?

[DarkZeros]

x264 always tries to reach the best image quality with your settings.
You shouldn't force a lower I/P or I/B ratio to fix a VBV issue.

Just set the VBV max value. x264 will recalc all the frames to fit in that maximun and mantain all the quality.

[akupenguin]

The bolded part was my rationale. Although the vbv limits diminish quality during complex scenes, the hope was that I could use those extra bits to help improve quality during less complex scenes.
If you want to spend more bits in total, reduce the value of crf. That will improve the quality of everything equally, except the scenes that already ran into the vbv limit. Anything else would be an attempt at psy tuning of bit distribution, which you should do only if you think you've identified a problem in the default crf model.

Does this mean that x264 is not basing p-frame quality on the resultant vbv-limited I-frame quality, but rather the I-frame quality as if it wasn't vbv limited?
If an I-frame ever has worse quality than the following P-frames, then you haven't actually saved any bits on that scene (compared to leaving the I-frame at full quality), because any details that were omitted from the I-frame just have to be coded in the P-frames before their quality is up to snuff. For more complicated reasons, you even want I-frames a little higher quality than P-frames, not just equal (hence the default ipratio).
x264 first runs unrestricted crf, then looks for vbv violations and takes away quality equally from everything that can reallocate bits toward that peak. This range will usually be about the length of the vbv buffer (0.6 seconds in your case).