rbsp_alignemnt_zero_bit

[leoenc]

We're getting some x264 streams rejected by the decoder, claiming that they have rbsp_alignment_zero_bit = 1, where only rbsp_alignment_zero_bit = 0 is allowed.

Anyone knows how to fix this?

Thanks

[Guest]

What decoder are you talking about? Can you post a sample source stream that invokes this problem?

[leoenc]

Thanks for your reply, neuron2

The decoder is a Tektronix Cerify.

Would the command line be enough?

x264.exe --pass 1 --stats "test.stats" --preset medium --bitrate 1500 --profile high --level 4.1 --keyint 25 --open-gop --ref 3 --mixed-refs --b-pyramid strict --bframes 3 --vbv-bufsize 1500 --vbv-maxrate 6000 --thread-input --threads auto --interlaced --tff --sar 64:45 --nal-hrd cbr --videoformat component --colorprim bt709 --transfer bt709 --colormatrix bt709 --pic-struct --output x264.264 source.avs

x264.exe --pass 2 --stats "test.stats" --preset medium --bitrate 1500 --profile high --level 4.1 --keyint 25 --open-gop --ref 3 --mixed-refs --b-pyramid strict --bframes 3 --vbv-bufsize 1500 --vbv-maxrate 6000 --thread-input --threads auto --interlaced --tff --sar 64:45 --nal-hrd cbr --videoformat component --colorprim bt709 --transfer bt709 --colormatrix bt709 --pic-struct --output x264.264 source.avs

[Guest]

I ask again: Can you post a sample source stream that invokes this problem?

[leoenc]

I will prepare a sample asap

[LoRd_MuldeR]

Found some info, that might be helpful here:
http://mailman.videolan.org/pipermail/x264-devel/2008-November/005169.html

[leoenc]

Source:
http://www.hamburgpromedia.com/hpm-downloads/MXF-Testfiles/IMX_XDCAM_50Mbit_SD.mxf

And output:
http://www.sendspace.com/file/eewy8a

[Guest]

Sorry but I can't process MXF. Can you demux the raw AVC and link that?

[LoRd_MuldeR]

From what I see the "output" is TS. Since x264 doesn't output TS, how did you create that?

[leoenc]

Sure:
http://www.sendspace.com/file/jw381e
(raw MPEG-2, not AVC)

[leoenc]

@LoRd_MuldeR, with a Manzanita multiplexer.

[Guest]

(raw MPEG-2, not AVC) You're confusing me now. Your original post says you have some AVC streams that are confusing the decoder. So why are you giving us an MPEG2 stream?

Please give us the AVC stream that is not properly decoded. Or please better explain your issue to resolve the confusion. Thank you.

[nevcairiel]

I thought it was pretty clear.
He has this MPEG-2 stream which he transcodes to H.264 using the command lines mentioned earlier.

He provided both the input MPEG-2 and the output H.264 (in TS).

This was his output H.264 stream:
http://www.sendspace.com/file/eewy8a
as linked a few posts above.

[LoRd_MuldeR]

So I think we have three options here:

(1) His H.264 decoder is parsing the stream in the wrong way (rbsp_alignemnt_zero_bit can not be non-zero, see link above)

(2) x264 creates invalid bitstream, which I think is unlikely - but you never know ;)

(3) His TS muxer somehow screwed up the stream or at least produced something his decoder/demuxer doesn't handle right

[nevcairiel]

If its truely syntactically impossible for this to be non-zero, it would seem odd that a validation would check this, unless its an assert that somehow checks its own function for correctness instead of the bitstream.

[leoenc]

Sorry about the confusion, nevcairiel got it right, the AVC stream which is causing the error is here: http://www.sendspace.com/file/eewy8a.

@LoRd_MuldeR, I agree. I'll check with the Manzanita team if option 3 is a possibility.

[LoRd_MuldeR]

For testing purposes: Can you pump the "raw" AVS, exactly as it pops out of x264, into the "Tektronix Cerify" decoder directly and avoid the TS muxing+demuxing altogether?

(If that does work, you know where the problem is probably located)

[leoenc]

Just tried that. The raw x264 stream passes the test.
It is strange, as the stream isn't altered in a way by the muxer. I have demuxed it and compared it to the pre-muxed .264 and they are identical bit for bit.

[leoenc]

Strike my last message, I was looking at the wrong place.
The raw x264 stream DOESN'T pass the test too.

[akupenguin]

As nevcairiel said, this should be syntactically impossible. rbsp_stop_one_bit is defined to be the last 1-bit in a RBSP, and rbsp_alignment_zero_bit are the trailing 0-bits (if any). rbsp_stop_one_bit determines which bit a slice ends at, not the other way around.

[drmpeg]

It could just be a bug in the verifier. After all, it's only software. @leoenc, could you post the output of the verifier here? Specifically, which NAL (or file byte offset) it's complaining about.

Ron

[leoenc]

Here is the report for the first second:

00:00:00:00
frame 1
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position
0x32b6 (dec. 12982), bit 0. Only allowable value
is 0 (0x0).

00:00:00:01
frame 2
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position
0x3ce0 (dec. 15584), bit 0. Only allowable value
is 0 (0x0).

00:00:00:03
frame 4
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position
0x4945 (dec. 18757), bit 0. Only allowable value
is 0 (0x0).

00:00:00:05
frame 6
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position
0x68fb (dec. 26875), bit 0. Only allowable value
is 0 (0x0).

00:00:00:06
frame 7
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position
0x5386 (dec. 21382), bit 0. Only allowable value
is 0 (0x0).

00:00:00:09
frame 10
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position
0xe354 (dec. 58196), bit 0. Only allowable value
is 0 (0x0).

00:00:00:13
frame 14
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position
0x17bb9 (dec. 97209), bit 0. Only allowable
value is 0 (0x0).

00:00:00:14
frame 15
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position
0x16d03 (dec. 93443), bit 0. Only allowable
value is 0 (0x0).

00:00:00:16
frame 17
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position
0x1acac (dec. 109740), bit 0. Only allowable
value is 0 (0x0).

00:00:00:17
frame 18
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position
0x203ce (dec. 132046), bit 0. Only allowable
value is 0 (0x0).

00:00:00:22
frame 23
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position
0x27e9b (dec. 163483), bit 0. Only allowable
value is 0 (0x0).

00:00:00:23
frame 24
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position
0x2cd62 (dec. 183650), bit 0. Only allowable
value is 0 (0x0).

00:00:00:24
frame 25
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position
0x2bea6 (dec. 179878), bit 0. Only allowable
value is 0 (0x0).

[drmpeg]

It looks like it's complaining about the case where rbsp_stop_one_bit is already byte aligned.

Ron

[leoenc]

But is it right?
I have also tested a Mainconcept stream from the same source, and it passed the test.

Does anyone know of a way to scan those alignment/trailing bits and see if there are indeed 1's instead of 0's in them?

[leoenc]

@drmpeg,

Where the rbsp_stop_one_bit is already byte aligned: The rbsp_stop_one_bit is the last bit of the aligned byte so there are no rbsp_alignment_zero_bits for the decoder to check? (which would appear to be a decoder issue which we can raise with Tektronix)

Or that at this point there is an rbsp_stop_one_bit and an rbsp_alignment_zero_bit that is set to 1? (which would appear to be syntactically wrong)

[drmpeg]

@drmpeg,

Where the rbsp_stop_one_bit is already byte aligned: The rbsp_stop_one_bit is the last bit of the aligned byte so there are no rbsp_alignment_zero_bits for the decoder to check? (which would appear to be a decoder issue which we can raise with Tektronix)

Or that at this point there is an rbsp_stop_one_bit and an rbsp_alignment_zero_bit that is set to 1? (which would appear to be syntactically wrong)

That's the gist of it. The trick will be to prove exactly which case is true. Here's a hex dump to show the issue.

http://www.w6rz.net/rbsp.png

The 0x81 byte is located at offset 0x3ce0, the last byte in one of the frames which the Tek analyzer complained about. The question is whether the 1 at bit 0 is the real rbsp_stop_one_bit (and the NAL is already byte aligned) or the 1 at bit 7 is the real rbsp_stop_one_bit (and bit 0 is an rbsp_alignment_zero_bit that is erroneously set to 1).

I guess the manly way to tackle this would be to instrument a decoder and see if it uses all the bitstream tokens up to bit 0, or if it exits the frame on bit 7. I'm guessing this is what the Tek analyzer may be doing itself.

Another way would be to change all of the erroneous rbsp_alignment_zero_bit locations back to 0 (with a hex editor) and see if it changes the decoding of the bitstream. I've already tried this on a few frames, but it didn't make any difference. However, the content is letterboxed and the black bars are probably not ideal for showing any problems.

Could you code some non-letterboxed content with the same x264 settings and see if the Tek analyzer complains? If it does, it may be a better bitstream to try this approach on.

Ron

[leoenc]

Thanks Ron, it does complain also on a non-letterboxed content.

Here's the sample:
http://www.sendspace.com/file/i6odjr

[drmpeg]

Thanks Ron, it does complain also on a non-letterboxed content.

Here's the sample:
http://www.sendspace.com/file/i6odjr

Can you post the Tek analyzer report for this stream too?

Ron

[leoenc]

Here it is for the first 100 frames:

00:00:00:00 frame 1
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0xb00f (dec. 45071), bit 0. Only allowable value is 0 (0x0).
00:00:00:01 frame 2
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0xbd78 (dec. 48504), bit 0. Only allowable value is 0 (0x0).
00:00:00:03 frame 4
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0xc27e (dec. 49790), bit 0. Only allowable value is 0 (0x0).
00:00:00:05 frame 6
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0xcd90 (dec. 52624), bit 0. Only allowable value is 0 (0x0).
00:00:00:06 frame 7
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0xcf71 (dec. 53105), bit 0. Only allowable value is 0 (0x0).
00:00:00:08 frame 9
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0xcbe5 (dec. 52197), bit 0. Only allowable value is 0 (0x0).
00:00:00:09 frame 10
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0xdd1f (dec. 56607), bit 0. Only allowable value is 0 (0x0).
00:00:00:13 frame 14
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x15285 (dec. 86661), bit 0. Only allowable value is 0 (0x0).
00:00:00:14 frame 15
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x16f9e (dec. 94110), bit 0. Only allowable value is 0 (0x0).
00:00:00:16 frame 17
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x14227 (dec. 82471), bit 0. Only allowable value is 0 (0x0).
00:00:00:17 frame 18
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x1c9dc (dec. 117212), bit 0. Only allowable value is 0 (0x0).
00:00:00:22 frame 23
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x25cd0 (dec. 154832), bit 0. Only allowable value is 0 (0x0).
00:00:00:23 frame 24
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x278b7 (dec. 161975), bit 0. Only allowable value is 0 (0x0).
00:00:00:24 frame 25
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x2520e (dec. 152078), bit 0. Only allowable value is 0 (0x0).
00:00:01:02 frame 28
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x3c0fb (dec. 246011), bit 0. Only allowable value is 0 (0x0).
00:00:01:03 frame 29
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x3c409 (dec. 246793), bit 0. Only allowable value is 0 (0x0).
00:00:01:04 frame 30
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x3b910 (dec. 243984), bit 0. Only allowable value is 0 (0x0).
00:00:01:06 frame 32
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x3e520 (dec. 255264), bit 0. Only allowable value is 0 (0x0).
00:00:01:10 frame 36
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x414a0 (dec. 267424), bit 0. Only allowable value is 0 (0x0).
00:00:01:10 frame 36
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 4 bits at time 1.420s.The overflow occurred during the decoding of frame 37, bitstream position 0x40f4e (dec. 266062). The total HRD buffer occupancy is 1500004 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:01:11 frame 37
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x417ac (dec. 268204), bit 0. Only allowable value is 0 (0x0).
00:00:01:12 frame 38
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x40b31 (dec. 265009), bit 0. Only allowable value is 0 (0x0).
00:00:01:13 frame 39
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x43b7d (dec. 277373), bit 0. Only allowable value is 0 (0x0).
00:00:01:14 frame 40
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 1 bits at time 1.500s.The overflow occurred during the decoding of frame 41, bitstream position 0x44a9c (dec. 281244). The total HRD buffer occupancy is 1500001 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:01:15 frame 41
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 0 bits at time 1.540s.The overflow occurred during the decoding of frame 42, bitstream position 0x467e8 (dec. 288744). The total HRD buffer occupancy is 1500000 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:01:16 frame 42
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x43812 (dec. 276498), bit 0. Only allowable value is 0 (0x0).
00:00:01:16 frame 42
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 3 bits at time 1.460s.The overflow occurred during the decoding of frame 39, bitstream position 0x417ad (dec. 268205). The total HRD buffer occupancy is 1500003 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:01:18 frame 44
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 4 bits at time 1.660s.The overflow occurred during the decoding of frame 45, bitstream position 0x4bfcb (dec. 311243). The total HRD buffer occupancy is 1500004 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:01:19 frame 45
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x4e192 (dec. 319890), bit 0. Only allowable value is 0 (0x0).
00:00:01:19 frame 45
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 3 bits at time 1.700s.The overflow occurred during the decoding of frame 46, bitstream position 0x4dd17 (dec. 318743). The total HRD buffer occupancy is 1500003 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:01:20 frame 46
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x4ad77 (dec. 306551), bit 0. Only allowable value is 0 (0x0).
00:00:01:21 frame 47
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x53074 (dec. 340084), bit 0. Only allowable value is 0 (0x0).
00:00:01:24 frame 50
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x52c14 (dec. 338964), bit 0. Only allowable value is 0 (0x0).
00:00:01:24 frame 50
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 2 bits at time 1.740s.The overflow occurred during the decoding of frame 47, bitstream position 0x4fa63 (dec. 326243). The total HRD buffer occupancy is 1500002 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:02:00 frame 51
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x69663 (dec. 431715), bit 0. Only allowable value is 0 (0x0).
00:00:02:00 frame 51
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 5 bits at time 1.900s.The overflow occurred during the decoding of frame 51, bitstream position 0x56f92 (dec. 356242). The total HRD buffer occupancy is 1500005 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:02:02 frame 53
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x6c759 (dec. 444249), bit 0. Only allowable value is 0 (0x0).
00:00:02:05 frame 56
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x6fe65 (dec. 458341), bit 0. Only allowable value is 0 (0x0).
00:00:02:07 frame 58
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x708e4 (dec. 461028), bit 0. Only allowable value is 0 (0x0).
00:00:02:08 frame 59
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x6f8c3 (dec. 456899), bit 0. Only allowable value is 0 (0x0).
00:00:02:09 frame 60
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x74021 (dec. 475169), bit 0. Only allowable value is 0 (0x0).
00:00:02:10 frame 61
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x744f2 (dec. 476402), bit 0. Only allowable value is 0 (0x0).
00:00:02:10 frame 61
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 0 bits at time 2.540s.The overflow occurred during the decoding of frame 62, bitstream position 0x74022 (dec. 475170). The total HRD buffer occupancy is 1500000 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:02:12 frame 63
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 1 bits at time 2.500s.The overflow occurred during the decoding of frame 60, bitstream position 0x708e5 (dec. 461029). The total HRD buffer occupancy is 1500001 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:02:13 frame 64
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x76fca (dec. 487370), bit 0. Only allowable value is 0 (0x0).
00:00:02:15 frame 66
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x778ae (dec. 489646), bit 0. Only allowable value is 0 (0x0).
00:00:02:18 frame 69
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x7a952 (dec. 502098), bit 0. Only allowable value is 0 (0x0).
00:00:02:19 frame 70
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 3 bits at time 2.700s.The overflow occurred during the decoding of frame 71, bitstream position 0x7b97f (dec. 506239). The total HRD buffer occupancy is 1500003 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:02:20 frame 71
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 4 bits at time 2.660s.The overflow occurred during the decoding of frame 68, bitstream position 0x778af (dec. 489647). The total HRD buffer occupancy is 1500004 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:02:23 frame 74
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x833a9 (dec. 537513), bit 0. Only allowable value is 0 (0x0).
00:00:02:24 frame 75
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x800e6 (dec. 524518), bit 0. Only allowable value is 0 (0x0).
00:00:02:24 frame 75
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 2 bits at time 2.740s.The overflow occurred during the decoding of frame 72, bitstream position 0x7d6cb (dec. 513739). The total HRD buffer occupancy is 1500002 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:03:00 frame 76
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 5 bits at time 2.900s.The overflow occurred during the decoding of frame 76, bitstream position 0x84bfa (dec. 543738). The total HRD buffer occupancy is 1500005 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:03:01 frame 77
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x9a9ea (dec. 633322), bit 0. Only allowable value is 0 (0x0).
00:00:03:02 frame 78
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x9b0fa (dec. 635130), bit 0. Only allowable value is 0 (0x0).
00:00:03:03 frame 79
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x9b672 (dec. 636530), bit 0. Only allowable value is 0 (0x0).
00:00:03:06 frame 82
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0x9eb9a (dec. 650138), bit 0. Only allowable value is 0 (0x0).
00:00:03:09 frame 85
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0xa1a73 (dec. 662131), bit 0. Only allowable value is 0 (0x0).
00:00:03:11 frame 87
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0xa24c2 (dec. 664770), bit 0. Only allowable value is 0 (0x0).
00:00:03:14 frame 90
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0xa56aa (dec. 677546), bit 0. Only allowable value is 0 (0x0).
00:00:03:16 frame 92
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0xa4be7 (dec. 674791), bit 0. Only allowable value is 0 (0x0).
00:00:03:19 frame 95
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 3 bits at time 3.700s.The overflow occurred during the decoding of frame 96, bitstream position 0xa95e7 (dec. 693735). The total HRD buffer occupancy is 1500003 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.
00:00:03:20 frame 96
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0xa7f58 (dec. 687960), bit 0. Only allowable value is 0 (0x0).
00:00:03:22 frame 98
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0xaf316 (dec. 717590), bit 0. Only allowable value is 0 (0x0).
00:00:03:24 frame 100
Incorrect rbsp_alignment_zero_bit
rbsp_alignment_zero_bit = 1 (0x1), at position 0xadc43 (dec. 711747), bit 0. Only allowable value is 0 (0x0).
00:00:03:24 frame 100
HRD overflow
Hypothetical Reference Decoder SchedSelIdx = 0 model has overflowed by 2 bits at time 3.740s.The overflow occurred during the decoding of frame 97, bitstream position 0xab333 (dec. 701235). The total HRD buffer occupancy is 1500002 bits and the maximum limit is 1500000 bits. HRD parameters were derived from the sequence_header.

[drmpeg]

It's a bug in x264. I went ahead and turned on the trace in the JM decoder, and it always exits the frame on the 1 bit before the reported errors.

However, it's pretty much a non-issue since it doesn't affect decoding.

Ron

[leoenc]

Thanks, Ron.

The problem is that quite a lot of broadcasters rely on Cerify's reports, even if this issue has no effect on actual decoding.
Since x264 is probably the best H.264 encoder out there, it would be a shame to use an inferior encoder just because of this minor issue.

[LoRd_MuldeR]

Thanks, Ron.

The problem is that quite a lot of broadcasters rely on Cerify's reports, even if this issue has no effect on actual decoding.
Since x264 is probably the best H.264 encoder out there, it would be a shame to use an inferior encoder just because of this minor issue.

You may want to discuss the issue on the x264 mailing list then:
http://mailman.videolan.org/listinfo/x264-devel

[jpsdr]

If according drmpeg it's a bug in x264, dev should at least be notified.

[LoRd_MuldeR]

If according drmpeg it's a bug in x264, dev should at least be notified.

Well, might not be a bug but a "feature", after all ;)

See here:
* http://mailman.videolan.org/pipermail/x264-devel/2009-March/005756.html
* http://mailman.videolan.org/pipermail/x264-devel/2009-March/005761.html

If I get this right, the "fix" might be as easy as commenting out one line. The discussion is from 2009, but that line still exists in x264_cabac_encode_flush().

[leoenc]

Yes, I can confirm that.
Deleting that line solves the issue.

[Selur]

If like Ludrao mentioned (http://mailman.videolan.org/pipermail/x264-devel/2009-March/005761.html) this is really a violation of the standard, this is a bug and not a 'feature' ;)

[LoRd_MuldeR]

Those are just padding bits, used to ensure the proper byte-alignment. According to the specs "The next bit consists of a single rbsp_stop_one_bit equal to 1" and "When the rbsp_stop_one_bit is not the last bit of a byte-aligned byte, one or more bsp_alignment_zero_bit is present to result in byte alignment". According to the specs, those padding bits are "equal to 0". But as they are just padding and will be ignored (skipped over) anyway, it doesn't matter much.

The H.264 verification tool probably is right to complain about the non-zero padding bits, because such tools are supposed to be nit-picky. On the other hand, real H.264 decoders won't care. Bitstreams created by x264 probably have been like that since at least 2008. And still they play fine, right? Also, we can assume the developers have added the "watermark" for a reason, so I doubt it will be removed in the official code ;)

[MasterNobody]

LoRd_MuldeR
By making not full quote of specs you are missing the point what rbsp_trailing_bits() element is (which values are correct and which not).
2) rbsp_trailing_bits() are present after the SODB as follows:
i) The first (most significant, left-most) bits of the final RBSP byte contains the remaining bits of the SODB (if any).
ii) The next bit consists of a single rbsp_stop_one_bit equal to 1.
iii) When the rbsp_stop_one_bit is not the last bit of a byte-aligned byte, one or more rbsp_alignment_zero_bit is present to result in byte alignment.
And according to this definition it is technically not possible to make rbsp_alignment_zero_bit equal 1 (as already was pointed to in this thread before) because if you see bit equal 1 after rbsp_stop_one_bit that mean that the bit you thought was rbsp_stop_one_bit was really not it but part of SODB.
The only thing you can technically make incorrect in rbsp_trailing_bits() is to not have rbsp_stop_one_bit at all i.e. zero byte.

[LoRd_MuldeR]

And according to this definition it is technically not possible to make rbsp_alignment_zero_bit equal 1 (as already was pointed to in this thread before) because if you see bit equal 1 after rbsp_stop_one_bit that mean that the bit you thought was rbsp_stop_one_bit was really not it but part of SODB.

I see. At least if there really is no other way of determining how many bits there are left of the SODB. But if that is the case, then I wonder, how can the bitstream created by x264 be decoded correctly at all? :scared:

(Also it seems somehow the H.264 verification tool does know the SODB ends before the "additional" 1-bit. Otherwise it couldn't complain about the 1-bit in the rbsp_trailing_bits, right?)

[MasterNobody]

I see. At least if there really is no other way of determining how many bits there are left of the SODB. But if that is the case, then I wonder, how can the bitstream created by x264 be decoded correctly at all? :scared:
Why shouldn't it?

[drmpeg]

@MasterNobody. You're beating a dead horse. This issue is fully understood, and you're backtracking over a canard that's already been exposed to be not true.

What should be discussed at this point is whether a more clever way (that doesn't break the H.264 specification and make bitstream verifiers complain) to uniquely identify x264 streams can be found.

Ron

[LoRd_MuldeR]

Why shouldn't it?

Well, you said the end of the SODB is determined by the last 1-bit (rbsp_stop_one_bit) and all bits after that are 0-bits by definition (rbsp_alignment_zero_bit). However we know that x264 sometimes (according some "watermark" pattern) adds an additional 1-bit after the actual rbsp_stop_one_bit, i.e. at a position where only rbsp_alignment_zero_bit's would be expected - making that "watermark" bit look as if it was the rbsp_stop_one_bit. But it isn't.

Now, if there really was no other way to determine the end of the SODB, expect for searching the very last 1-bit (rbsp_alignment_zero_bit), this would mean two things:

(1) When the x264 watermark bit happens to be "1", the decoder would assume that this watermark bit is the rbsp_stop_one_bit, rather than the "real" stop bit. In other words, the decoder could not see the "real" stop bit, because it gets confused by the additional watermark bit, which - according to the specification - now has become the stop bit. Consequently the decoder would assume the SODB ends at a later position than it actually does. It thus would read some "superfluous" bits. Doesn't sound very "safe" to me. How is it made sure that this cannot lead to decoding issues? :confused:

(2) The H.264 verifier also would assume the watermark bit (if present) is the rbsp_stop_one_bit. Consequently it could never have thrown the warning, which has lead to this thread. But it does! So, after all, it seems the H.264 verifier somehow knows where the SODB really ends, it recognizes the "real" rbsp_stop_one_bit as such. And it notices that there is a one bit - the x264 watermark bit - where only zero bits should be present. How is that possible then? :confused:

[akupenguin]

Consequently the decoder would assume the SODB ends at a later position than it actually does. It thus would read some "superfluous" bits.
That's just the nature of CABAC: A CABAC decoder treats the whole SODB as a single real number, and the result of any given decoded decision depends on whether its numeric value is within certain ranges. Any number in the right range works, and appending extra digits won't move it out of the right range, for the same reason that coding more decisions doesn't change the value of previous decisions.

CAVLC determines where a slice ends based on whether there are any bits left in the SODB. CABAC doesn't, because the equivalent information would be whether there are any decisions left, but a decision can be less than 1 bit, so even bit-accurate knowledge of where the SODB ends wouldn't help. Thus, the standard never defines CABAC as depending on the number of bits in the SODB, and instead explicitly codes the end of the slice as a CABAC decision (a highly compressed decision, of course, since it's very predictable that most macroblocks aren't the end of a slice.)

[kabelbrand]

Whether bug or feature, if there is no good reason for this "watermark" or whatever it is it should be changed to play nice with verifiers since these make or break usage in professional environments. Even if it is just a note or warning people tend to be overly careful and will not allow it if the report is not clean.
At least this is my experience with Blu-ray verifiers - if a device has problems with the playback of a finished product months after release someone will point at a totally unrelated verifier note (e.g. slice count in fake interlaced) and you have to explain what this is about and calm everyone down.

[LoRd_MuldeR]

That's just the nature of CABAC: A CABAC decoder treats the whole SODB as a single real number, and the result of any given decoded decision depends on whether its numeric value is within certain ranges. Any number in the right range works, and appending extra digits won't move it out of the right range, for the same reason that coding more decisions doesn't change the value of previous decisions.

CAVLC determines where a slice ends based on whether there are any bits left in the SODB. CABAC doesn't, because the equivalent information would be whether there are any decisions left, but a decision can be less than 1 bit, so even bit-accurate knowledge of where the SODB ends wouldn't help. Thus, the standard never defines CABAC as depending on the number of bits in the SODB, and instead explicitly codes the end of the slice as a CABAC decision (a highly compressed decision, of course, since it's very predictable that most macroblocks aren't the end of a slice.)

Thanks for the info. This makes it clear then!

[kieranrk]

Whether bug or feature, if there is no good reason for this "watermark" or whatever it is it should be changed to play nice with verifiers since these make or break usage in professional environments. Even if it is just a note or warning people tend to be overly careful and will not allow it if the report is not clean.
At least this is my experience with Blu-ray verifiers - if a device has problems with the playback of a finished product months after release someone will point at a totally unrelated verifier note (e.g. slice count in fake interlaced) and you have to explain what this is about and calm everyone down.

As explained many times this doesn't violate the spec. In your world we would work round every analyzer bug.

[LoRd_MuldeR]

As explained many times this doesn't violate the spec. In your world we would work round every analyzer bug.

I think that's not so clear. As akupenguin has explained, because of the way CABAC (and arithmetic coding in general) works, appending more digits does not effect previously encoded symbols/decisions. Also, because there is a separate "end of slice" decision, the end of the slice is does not depend on the number of bits in the SODB. That's why it still decodes properly after appending the "superfluous" bits.

Still: If did follow the specifications closely, there obviously wouldn't be anything left to be encoded after the "end of slice" decision indicated that the slice does end here. So if we compare the "expected" output to the output from x264, we see that there are some additional/unexpected bits. Regardless of whether those effect decoding or not - the specification does not intend them to be there.

So the H.264 verification complains about an "anomaly" in the bitstream, because it is its job to be nit-picky ;)

[kieranrk]

So if we compare the "expected" output to the output from x264, we see that there are some additional/unexpected bits. Regardless of whether those effect decoding or not - the specification does not intend them to be there.

There is no such thing as an "expected" output nor does the specification "intend [the bits] to be there" or not there. The encoder can write whatever it likes so long as it conforms to the specification. Just because this is unusual, doesn't mean it is not compliant.

[drmpeg]

The encoder can write whatever it likes so long as it conforms to the specification.

Correct. So we look at the H.264 specification, and in section 7.4.2.11, it states:


7.4.2.11 RBSP trailing bits semantics

rbsp_stop_one_bit shall be equal to 1.

rbsp_alignment_zero_bit shall be equal to 0.


Whenever you see the word "shall" in a specification, that's a hard requirement. H.264 bitstream verifiers like the Tektronix Cerify are most certainly going to test that that hard requirement is met.

x264 fails to meet the "rbsp_alignment_zero_bit shall be equal to 0" requirement. Therefore, x264 is not fully compliant to the H.264 specification.

On the other hand, it's a pretty damn clever hack because it uses bits that don't affect the decoding process. Even better, it doesn't require a bitstream analyzer to check. All you need is a hex editor to see the signature.

So it's a philosophical issue. Should x264 go in the pedantic direction and take the hack out (and possibly develop an even more clever watermark)? Or should it just be left alone (as a known issue and hopefully folks get directed to this thread as they rediscover this in the future)?

Personally, I don't have a dog in the fight.

Ron

[sneaker_ger]

But that hard requirement is met, as explained earlier. Look at page 65 for example:
the rbsp_stop_one_bit, which is the last (least significant, right-most) bit equal to 1,

So by definition the error message "Incorrect rbsp_alignment_zero_bit rbsp_alignment_zero_bit = 1 (0x1), at position" is wrong. The verifier is looking at the "(least significant, right-most) bit equal to 1", which is the rbsp_stop_one_bit by definition of the spec - not the rbsp_alignment_zero_bit.

As far as I understand it, if you wanted to prove a spec violation you'd have to find a part of the spec explicitly saying that no garbage data between the last CABAC decision and the rbsp_stop_one_bit is allowed. I kind of share LoRd_MuldeR's reservations expressed in post #47, i.e. a spec says what bits have to be there and that stuffing in extra bits (well, not really "extra" here because the alignment has to be fulfilled anyways) does not have to be explicitly prohibited to make them illegal, because that would be impractical and just unnecessarily blow every spec out of proportion and therefore it's hard to find any explicit rules concerning this specific question. When implementing a spec best practice is to try to follow the writers intend, not to find "clever" ways to interpret it.