1-pass bitrate mode: bitrate is off by 25%

[Warperus]

Recently I tried to compare few codecs in a field of game movies coding. x264 showed great results in different modes, and even in 1-pass bitrate-wise mode it showed quite the good picture. But when I looked closer I found out why: bitrate is not tolerated as it should be, average bitrate is higher by 20-25%.

I have used command line like this:
x264 -o x264_1k_unrate.mp4 --preset veryslow --bitrate 1000 Renders\x264_dp_5000.mp4

--ratetol option doesn't help (--ratetol 0 produces the same).
I have not found anything that could explain this behaviour. Is it a bug or something?

The source producing this effect is uploaded here:
http://www.mediafire.com/?3dzgmzn3nzy

MediaInfo output:
--------------------------
General
Complete name : C:\x264_1k_unrate.mp4
Format : MPEG-4
Format profile : JVT
Codec ID : avc1
File size : 23.3 MiB
Duration : 2mn 38s
Overall bit rate : 1 235 Kbps
Encoded date : UTC 2010-04-21 19:05:44
Tagged date : UTC 2010-04-21 19:05:44

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.0
Format settings, CABAC : Yes
Format settings, ReFrames : 16 frames
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 2mn 38s
Bit rate mode : Variable
Bit rate : 1 233 Kbps
Nominal bit rate : 1 000 Kbps
Maximum bit rate : 9 957 Kbps
Width : 720 pixels
Height : 576 pixels
Display aspect ratio : 5:4
Frame rate mode : Constant
Frame rate : 25.000 fps
Standard : PAL
Resolution : 8 bits
Colorimetry : 4:2:0
Scan type : Progressive
Bits/(Pixel*Frame) : 0.119
Stream size : 23.2 MiB (100%)
Writing library : x264 core 93 r1542 5b86182
Encoding settings : cabac=1 / ref=16 / deblock=1:0:0 / analyse=0x3:0x133 / me=umh / subme=10 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=12 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / constrained_intra=0 / bframes=8 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / wpredb=1 / wpredp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=abr / mbtree=1 / bitrate=1000 / ratetol=1.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00
Encoded date : UTC 2010-04-21 19:05:44
Tagged date : UTC 2010-04-21 19:09:41

[Dark Shikari]

This is normal. The end of your video is extremely complex, far more than the start of the video. Since you're using 1-pass bitrate mode, x264 doesn't know about the end of the video when encoding the start of the video. When it hits the really complex end of the video, it has two choices: drop the quality to horrifically low levels or go a bit over the target bitrate. It does the latter, adapting downwards over time such that if the video was a lot longer, it would eventually reach 1000kbps.

tl;dr don't use 1-pass ABR.

[Blue_MiSfit]

This leads me to a related question.

Consider a similar source (high complexity at the end causing these issues with 1 pass ABR). If we instead use CRF encoding with VBV, can the following be assumed?

1) x264 will do the encode per normal using CRF for "Average complexity" regions, so long as the ABR is well below the VBV limitations
2) If we run into a high complexity region, the bitrate will be limited to comply with the VBV restrictions
3) If this high complexity region is followed by another low complexity region, x264 goes back to "normal" behavior, using only as much bitrate as is necessary to satisfy CRF rate control

So, from a big picture, x264 will produce a CRF encode, with only the regions of HIGH complexity being limited by VBV. Correct?

Thus, the CRF encode would only ever be as large as a CBR encode, but probably smaller. Correct? In other words, CRF + VBV is a way to tell the encoder "Hey, encode this file with constant quality x, but don't be larger than y", correct?

Or... am I just getting very confused? I'm basically trying to determine the ideal way to implement encoding where I only can run a single pass, want the quality to be consistent, and have SOME flexibility in terms of final ABR, but want to keep things "within reason" - define that however you will.. :p

~MiSfit

[Dark Shikari]

This leads me to a related question.

Consider a similar source (high complexity at the end causing these issues with 1 pass ABR). If we instead use CRF encoding with VBV, can the following be assumed?

1) x264 will do the encode per normal using CRF for "Average complexity" regions, so long as the ABR is well below the VBV limitations
2) If we run into a high complexity region, the bitrate will be limited to comply with the VBV restrictions
3) If this high complexity region is followed by another low complexity region, x264 goes back to "normal" behavior, using only as much bitrate as is necessary to satisfy CRF rate control

So, from a big picture, x264 will produce a CRF encode, with only the regions of HIGH complexity being limited by VBV. Correct?

Thus, the CRF encode would only ever be as large as a CBR encode, but probably smaller. Correct?Yes. In other words, CRF + VBV is a way to tell the encoder "Hey, encode this file with constant quality x, but don't be larger than y", correct?No, it's limiting local bitrate, not global bitrate.

[Blue_MiSfit]

Hrmm... maybe I'm not expressing myself clearly.

I understand that VBV only limits local bitrate. However, based on this:

B_M: Thus, the CRF encode would only ever be as large as a CBR encode, but probably smaller. Correct?
D S: Yes.

I could say (in the case of CRF+VBV) the output will never be larger than x kbps, where x == vbv-maxrate == vbv-bufsize. Correct??

Example: Assume the source is EXTREMELY complex throughout. Something that would constantly thrash against High @ 4.1 VBV:

--crf 14 --vbv-bufsize 62500 --vbv-maxrate 62500 will never be larger than --bitrate 62500 --vbv-maxrate 62500 --vbv-bufsize 62500. Correct?

If so, then one can predict the size of example #1 as being no larger than 62500kbps (in the extreme case), and probably much smaller. Correct?

If you're curious why I'm pursuing this line of questioning, I want to have the benefits of CRF, while limiting insanely high bitrates, and do it all in one pass. In other words, I don't want any captured file to be larger than 100mbps, but less is quite OK, provided the quality is perceptually lossless (virtually guaranteed by CRF 14)

Perhaps I'm misguided in trying to tell x264 "Hey, encode this file with constant quality x, but don't be larger than y" via this method, but it seems like it should work in theory?

If not, I'm just being stupid.

~MiSfit

[Dark Shikari]

Hrmm... maybe I'm not expressing myself clearly.

I understand that VBV only limits local bitrate. However, based on this:

B_M: Thus, the CRF encode would only ever be as large as a CBR encode, but probably smaller. Correct?
D S: Yes.

I could say (in the case of CRF+VBV) the output will never be larger than x kbps, where x == vbv-maxrate == vbv-bufsize. Correct??No idea where bufsize came out of there, but other than that, sure, that's correct. That's just not a very efficient way of doing that.

[Blue_MiSfit]

Indeed. My head is stuck in old work mode, where maxrate==bufsize always. Actually, assuming High @ 4.1, can I really assume more than 62500kbps for bufsize will work? Or do the standards only define maxrate and leave bufsize up to the decoder?

Anyway, is there a better way of doing what I want with x264 in it's present state? My crazy idea seems more efficient than simply doing a CBR encode...

~MiSfit

[Dark Shikari]

Indeed. My head is stuck in old work mode, where maxrate==bufsize always. Actually, assuming High @ 4.1, can I really assume more than 62500kbps for bufsize will work? Or do the standards only define maxrate and leave bufsize up to the decoder?

Anyway, is there a better way of doing what I want with x264 in it's present state? My crazy idea seems more efficient than simply doing a CBR encode...2-pass CRF. On the second pass, set bitrate equal to MIN( firstpass bitrate, the max bitrate you want to allow ).

[Blue_MiSfit]

But, I must do it all in one pass!! :devil: hint hint HD-SDI capture

[creamyhorror]

A related question: is it fair to say that reducing qcomp will reduce the variance of the final bitrate under CRF? Furthermore, that it results in a reduced level of fine detail/grain preserved? And, that it increases mb-tree's strength and discards more information from high-motion MBs?

Just asking to confirm my understanding, and to find out what situations qcomp could usefully be adjusted in. Sometimes I wish for a bit less variation in bitrates, and it would be nice to have a general though partial means of achieving that. (I've had to denoise certain troublesome old sources in the past in order to achieve a fair bitrate; it's more targeted but also more troublesome.)

[Warperus]

Dark Shikari
Thank you for answer. I don't want to use 1-pass ABR in real life, but that doesn't change the fact this mode is nearly unpredictable in some situations with default settings.
For example, some video service hosts my video without recompression if bitrate is lower than 2mbps (AFAIK youtube made things like this about a year ago). In this case I want to find a way to limit local bitrate. Otherwise it will be recompressed on video service side - the thing I'm trying to avoid. Predictable low quality of some scenes is ok as far as I know what's going on.

Unfortunately, vbv-maxrate by itself doesn't limit max local bitrate. I've processed my source with --vbv-maxrate 1000 --vbv-bufsize 30000 and it produced that same effect of average 1200 bitrate. And only --vbv-maxrate 1000 --vbvbufsize 1000 limited local bitrate in the final part even though default min-qp limited bitrate in first part even further so the final bitrate was like 770kbit/s.

Is it ever possible to do CBR in x264?

[Blue_MiSfit]

Yes, CBR is quite possible. Set --bitrate and --vbv-maxrate to the same value, and set --vbv-bufsize to the buffer size of your decoder, or equal to --bitrate if you're unsure.

If you need the extra padding (i.e. frame level CBR), then you can turn on --nal-hrd cbr. This will pad the output so you get a precise bitrate out of x264.

This is very much not ideal for compression though :)

~MiSfit

[Dark Shikari]

Dark Shikari
Thank you for answer. I don't want to use 1-pass ABR in real life, but that doesn't change the fact this mode is nearly unpredictable in some situations with default settings.
For example, some video service hosts my video without recompression if bitrate is lower than 2mbps (AFAIK youtube made things like this about a year ago). In this case I want to find a way to limit local bitrate. Otherwise it will be recompressed on video service side - the thing I'm trying to avoid. Predictable low quality of some scenes is ok as far as I know what's going on.I'm pretty sure Youtube doesn't allow passthrough on any videos anymore.Unfortunately, vbv-maxrate by itself doesn't limit max local bitrate. I've processed my source with --vbv-maxrate 1000 --vbv-bufsize 30000 and it produced that same effect of average 1200 bitrate.Yes, because the VBV buffer starts [/b]filled[/b]. Thus, if it is less full at the end than at the start, your average bitrate will exceed 1000 even though the VBV was never violated. Solution: use a smaller bufsize.

Better solution: just use 2-pass mode and don't worry too much about VBV.