Bitrate over and undershooting despite nal-hrd cbr [Archive]

FranceBB

5th April 2022, 15:33

Hi there,
a client is asking us to deliver files at 500 Mbit/s CBR with 8 slices, no motion compensation (i.e ref 1, keyint 1), profile High 4:2:2 10bit planar, Level 5.2 with audio mono PCM 24bit 48'000Hz and container mxf with timecode set to a fixed value they asked, namely 20:45:00:16.

I complied to the request with:

x264.exe "\\mibctvan000\Ingest\MEDIA\temp\AVS Script.avs" --preset medium --profile high422 --level 5.2 --keyint 1 --no-cabac --slices 8 --bitrate 500000 --vbv-maxrate 500000 --vbv-bufsize 500000 --deblock -1:-1 --overscan show --colormatrix bt2020nc --range tv --log-level info --thread-input --transfer arib-std-b67 --colorprim bt2020 --videoformat component --nal-hrd cbr --output-csp i422 --output-depth 10 --output "\\mibctvan000\Ingest\MEDIA\temp\raw_video.h264"

and then I used FFMpeg and BBC BMX to mux the audio so that the encoded file is:

General
Complete name : \\mibcuvnex001\Edit\File.mxf
Format : MXF
Format version : 1.3
Format profile : OP-1a
Format settings : Closed / Complete
File size : 482 GiB
Duration : 2 h 13 min
Overall bit rate : 519 Mb/s
Encoded date : 2022-03-09 18:16:54.908
Writing application : BBC bmx 0.1.3.0.2
Writing library : libMXF (Win32) 1.0.3.0.2

Video
ID : 1001
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High 4:2:2 Intra@L5.2
Format settings, CABAC : No
Format settings, GOP : N=1
Format settings, wrapping mode : Frame
Codec ID : 0D01030102106001-0401020201323001
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 500 Mb/s
Width : 3 840 pixels
Height : 2 160 pixels
Display aspect ratio : 16:9
Frame rate : 50.000 FPS
Color space : YUV
Chroma subsampling : 4:2:2
Bit depth : 10 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 1.206
Stream size : 465 GiB (96%)
Title : V1
Writing library : x264 core 164 r3079 d9a19f0
Encoding settings : cabac=0 / ref=1 / deblock=1:-1:-1 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=30 / lookahead_threads=5 / sliced_threads=0 / slices=8 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=0 / weightp=0 / keyint=1 / keyint_min=1 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=cbr / mbtree=0 / bitrate=500000 / ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=81 / qpstep=4 / vbv_maxrate=500000 / vbv_bufsize=500000 / nal_hrd=cbr / filler=1 / ip_ratio=1.40 / aq=1:1.00
Color range : Limited
Color primaries : BT.2020
Transfer characteristics : HLG
Matrix coefficients : BT.2020 non-constant

Audio #1
ID : 2001
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 24 bits
Stream size : 2.14 GiB (0%)
Title : A1
Locked : Yes

Audio #2
ID : 2002
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 24 bits
Stream size : 2.14 GiB (0%)
Title : A2
Locked : Yes

Audio #3
ID : 2003
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 24 bits
Stream size : 2.14 GiB (0%)
Title : A3
Locked : Yes

Audio #4
ID : 2004-1
Format : Dolby E
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Muxing mode : SMPTE ST 337
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 1 291 kb/s
Channel(s) : 6 channels
Channel layout : L C Ls X R LFE Rs X
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 20 bits
Delay relative to video : -20 h 45 min
Stream size : 1.20 GiB (0%)
Title : Dolby E world
Original frame rate : 25.000
Locked : Yes

Audio #5
ID : 2004-2
Format : Dolby E
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Muxing mode : SMPTE ST 337
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 505 kb/s
Channel(s) : 2 channels
Channel layout : X X X L X X X R
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 20 bits
Delay relative to video : -20 h 45 min
Stream size : 481 MiB (0%)
Title : Dolby E world
Original frame rate : 25.000
Locked : Yes

Audio #6
ID : 2005
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 24 bits
Stream size : 2.14 GiB (0%)
Title : A5
Locked : Yes

Audio #7
ID : 2006
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 24 bits
Stream size : 2.14 GiB (0%)
Title : A6
Locked : Yes

Audio #8
ID : 2007
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 24 bits
Stream size : 2.14 GiB (0%)
Title : A7
Locked : Yes

Audio #9
ID : 2008
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 24 bits
Stream size : 2.14 GiB (0%)
Title : A8
Locked : Yes

Other #1
ID : 901-Material
Type : Time code
Format : MXF TC
Frame rate : 50.000 FPS
Time code of first frame : 20:45:00:16
Time code settings : Material Package
Time code, striped : Yes
Title : TC1

Other #2
ID : 901-Source
Type : Time code
Format : MXF TC
Frame rate : 50.000 FPS
Time code of first frame : 20:45:00:16
Time code settings : Source Package
Time code, striped : Yes
Title : TC1

Unfortunately, though, the file got rejected.
The official reason reported by the studio I sent the file to is "Play Exception - Read too large for buffer" and "Determine Buffer Alignment Error in DID Reader".

Although the message is a bit cryptic, it looks like it's a buffer issue and indeed when I took a look at the file with BitrateViewer, it was anything but CBR:

https://i.imgur.com/Cy2DjsQ.png

The peak of 718 Mbit/s is FAR FAR higher than the 500 Mbit/s required, therefore the poor hardware playback port must have been struggling to cache it in memory and decode it properly, hence creating the issue.

Is there an option I'm missing in x264 to sort of force it to have a much stricter bitrate cap with as little oscillations as possible?

I'm currently relying on

--bitrate 500000 --vbv-maxrate 500000 --vbv-bufsize 500000 --nal-hrd cbr

to output a CBR; too bad it oscillates a bit too much :(

Any suggestions? (I can provide a short sample if needed).

jpsdr

5th April 2022, 18:17

Warning, what i'll say is to be confirmed/checked.
If i recall properly what someone said to me once, a long time ago, a cbr at X Mb can have peaks higher than X, it's not out of spec, it's because the buffer allow it, and it's its purpose.
Maybe... Only the bitrate is not enough to make a proper encode, you have to know also the buffer. If my memory and understanding is correct, maybe it's not surprising to have peaks at 718 Mb with everything (bitrate, buffer, etc...) at 500 Mb.

MasterNobody

5th April 2022, 19:53

https://i.imgur.com/Cy2DjsQ.png

The peak of 718 Mbit/s is FAR FAR higher than the 500 Mbit/s required, therefore the poor hardware playback port must have been struggling to cache it in memory and decode it properly, hence creating the issue.
Not sure which application you used to look at bitrate but as far I guess it show instant bitrate (bitrate of single frame) and not VBV buffer filling as Elecard Buffer Analyzer. Most probably what you really want is single frame VBV, and for that you should specify: vbv-bufsize = vbv-maxrate / fps
Or always use: --vbv-bufsize 1 and x264 will fix it with warning.

Theoretical maximum instant bitrate with your VBV settings and fps=50 can be almost 500*50=25000 Mbit/s (1 frame fill full VBV buffer and next 49 frames take almost 0 Mbit/s) and still be correct VBV-wise. x264 probably wouldn't generate such streams but 2*vbv-maxrate should be possible.

Selur

6th April 2022, 19:07

Side note: Shouldn't one use '--qcomp 0' when aiming for cbr encoding?

FranceBB

6th April 2022, 19:25

Side note: Shouldn't one use '--qcomp 0' when aiming for cbr encoding?

Uhmmm I think you're probably right!
By default it was --qcomp 0.60 but I guess I'm gonna try to encode the same file with --qcomp 0 and see what happens. If that solves the bitrate oscillation problem, you're gonna get a virtual badge of honour xD

rwill

6th April 2022, 20:37

Uhmmm I think you're probably right!
By default it was --qcomp 0.60 but I guess I'm gonna try to encode the same file with --qcomp 0 and see what happens. If that solves the bitrate oscillation problem, you're gonna get a virtual badge of honour xD

If I remember correctly nothing but the VBV clamper has any huge influence on final rate in CBR. It either lowers or higher 'some' QP so over the lookahead distance VBV will exit 'ok'.

Otherwise, what MasterNobody wrote.

FranceBB

6th April 2022, 22:38

If I remember correctly nothing but the VBV clamper has any huge influence on final rate in CBR. It either lowers or higher 'some' QP so over the lookahead distance VBV will exit 'ok'.

Otherwise, what MasterNobody wrote.

Ok, I'm gonna try with --vbv-bufsize 10000 too as it's 500,000/50 and I'll come back to you again.

kolak

6th April 2022, 23:08

It's normal that "those" viewers show peaks.
I see same thing with h264. Many spikes above specified CBR value yet when you put it through eg. Tektronix it's all fine.
BitrateViewer is by not means any reference tool.

rwill

7th April 2022, 06:29

The more I think about it the more I guess this use case could be hacked into x264 efficiently with another pseudo AVC INTRA class.

Blue_MiSfit

8th April 2022, 08:15

Seems like a good use case for single frame VBV :)

FranceBB

8th April 2022, 13:37

Ok, so, second attempt done with: --vbv-bufsize 10000 as it's 500,000 (bitrate) : 50 (framerate).

and sure enough, the bitrate didn't oscillate! :D

https://i.imgur.com/ojyRhJS.png

General
Complete name : \\mibcuvnex001\Edit\TEST WORKFLOW SKY UHD HDR HLG\4SH01011_UHD_HDR_HLG v2.mxf
Format : MXF
Format version : 1.3
Format profile : OP-1a
Format settings : Closed / Complete
File size : 482 GiB
Duration : 2 h 13 min
Overall bit rate : 519 Mb/s
Encoded date : 2022-04-07 14:56:27.684
Writing application : BBC bmx 0.1.3.0.2
Writing library : libMXF (Win32) 1.0.3.0.2

Video
ID : 1001
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High 4:2:2 Intra@L5.2
Format settings, CABAC : No
Format settings, GOP : N=1
Format settings, wrapping mode : Frame
Codec ID : 0D01030102106001-0401020201323001
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 500 Mb/s
Width : 3 840 pixels
Height : 2 160 pixels
Display aspect ratio : 16:9
Frame rate : 50.000 FPS
Color space : YUV
Chroma subsampling : 4:2:2
Bit depth : 10 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 1.206
Stream size : 465 GiB (96%)
Title : V1
Writing library : x264 core 164 r3079 d9a19f0
Encoding settings : cabac=0 / ref=1 / deblock=1:-1:-1 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=67 / lookahead_threads=11 / sliced_threads=0 / slices=8 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=0 / weightp=0 / keyint=1 / keyint_min=1 / scenecut=40 / intra_refresh=0 / rc_lookahead=1 / rc=cbr / mbtree=0 / bitrate=500000 / ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=81 / qpstep=4 / vbv_maxrate=500000 / vbv_bufsize=10000 / nal_hrd=cbr / filler=1 / ip_ratio=1.40 / aq=1:1.00
Color range : Limited
Color primaries : BT.2020
Transfer characteristics : HLG
Matrix coefficients : BT.2020 non-constant

Audio #1
ID : 2001
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 24 bits
Stream size : 2.14 GiB (0%)
Title : A1
Locked : Yes

Audio #2
ID : 2002
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 24 bits
Stream size : 2.14 GiB (0%)
Title : A2
Locked : Yes

Audio #3
ID : 2003
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 24 bits
Stream size : 2.14 GiB (0%)
Title : A3
Locked : Yes

Audio #4
ID : 2004-1
Format : Dolby E
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Muxing mode : SMPTE ST 337
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 1 291 kb/s
Channel(s) : 6 channels
Channel layout : L C Ls X R LFE Rs X
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 20 bits
Delay relative to video : -20 h 45 min
Stream size : 1.20 GiB (0%)
Title : Dolby E world
Original frame rate : 25.000
Locked : Yes

Audio #5
ID : 2004-2
Format : Dolby E
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Muxing mode : SMPTE ST 337
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 505 kb/s
Channel(s) : 2 channels
Channel layout : X X X L X X X R
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 20 bits
Delay relative to video : -20 h 45 min
Stream size : 481 MiB (0%)
Title : Dolby E world
Original frame rate : 25.000
Locked : Yes

Audio #6
ID : 2005
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 24 bits
Stream size : 2.14 GiB (0%)
Title : A5
Locked : Yes

Audio #7
ID : 2006
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 24 bits
Stream size : 2.14 GiB (0%)
Title : A6
Locked : Yes

Audio #8
ID : 2007
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 24 bits
Stream size : 2.14 GiB (0%)
Title : A7
Locked : Yes

Audio #9
ID : 2008
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (BWF)
Codec ID : 0D01030102060100
Duration : 2 h 13 min
Bit rate mode : Constant
Bit rate : 2 304 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Frame rate : 50.000 FPS (960 SPF)
Bit depth : 24 bits
Stream size : 2.14 GiB (0%)
Title : A8
Locked : Yes

Other #1
ID : 901-Material
Type : Time code
Format : MXF TC
Frame rate : 50.000 FPS
Time code of first frame : 20:45:00:16
Time code settings : Material Package
Time code, striped : Yes
Title : TC1

Other #2
ID : 901-Source
Type : Time code
Format : MXF TC
Frame rate : 50.000 FPS
Time code of first frame : 20:45:00:16
Time code settings : Source Package
Time code, striped : Yes
Title : TC1

I'm sending them the new file, thank you so much!! :D

benwaggoner

9th April 2022, 00:58

You can probably get away with a higher --vbv-bufsize then 10000. It's possible that low a value is impacting quality.

FranceBB

9th April 2022, 08:55

You can probably get away with a higher --vbv-bufsize then 10000. It's possible that low a value is impacting quality.

Probably, but bitrate / fps gives me a nice steady 500 Mbit/s result which should be the dream of any hardware playback device, regardless of its buffering capability. Anyway, I'm totally up to suggestions about quality improvement, so as long as the bitrate stays the same, if you have any suggestions, go for it. :)

jpsdr

9th April 2022, 11:54

I think you have a part of your answer in your post. You need to know buffer capability of HW playback device (i don't know if such a thing is possible), maybe it's the only way to optimise quality.

FranceBB

9th April 2022, 12:49

I think you have a part of your answer in your post. You need to know buffer capability of HW playback device (i don't know if such a thing is possible), maybe it's the only way to optimise quality.

Yeah, exactly.
Some playback ports are better than others, but when it comes down to hardware, it's always a lottery. :|
Anyway, I have good news as the operators of the studio I sent the file to repeated the test and did a full QC on it and it played all the way 'till the end. :D

Thanks everyone for the help, as always.
That's why I love Doom9.

EDIT: I've got the result: QC Pass, everything was fine. Thank you, guys!! :D Open Source rocks!

https://c.tenor.com/ukjUvE17ZcQAAAAC/party-minions.gif