Hex editing VBV and Bitrate flags for HD DVD

[hwjohn]

First, I just wanted to say thanks to the forums. I have not had a need to post until now because of all the great info (been reading doom9 for several years).

I'm working on a HD DVD calibration disc with another guy on the AV Science forums www.avsforum.com. This is an open source type thing that will be free to the community when we are done.

I have been using QuEnc and HCenc with some success to encode the 1920x1080 clips that Avisynth is generating, but it looks like both programs stamp the SD values for VBV and bitrate into the stream. I realize that is is by design because they weren't necessarily created for HD DVD. The patterns are simple enough that even the SD buffer will not underflow, but I would like to have the correct values in the bitstream so it is technically correct. I don't mind hex editing it, but I can't find any conclusive information that explains how the sequence headers are laid out so that I can find the flags and change them.

Any idea on how I can change these flags?

Thanks in advance,
Casey

[hank315]

I'm working on a HD DVD calibration disc with another guy on the AV Science forums www.avsforum.com. This is an open source type thing that will be free to the community when we are done.
Just read the thread on avs, cool, might be useful and free is great...

I have been using QuEnc and HCenc with some success to encode the 1920x1080 clips that Avisynth is generating, but it looks like both programs stamp the SD values for VBV and bitrate into the stream. I realize that is is by design because they weren't necessarily created for HD DVD. The patterns are simple enough that even the SD buffer will not underflow, but I would like to have the correct values in the bitstream so it is technically correct. I don't mind hex editing it, but I can't find any conclusive information that explains how the sequence headers are laid out so that I can find the flags and change them.

Any idea on how I can change these flags?

Yes, they are in the sequence header.
Download the MPEG2 2000 standard from
http://www.itu.int/rec/T-REC-H.262/en
and look at page 23, table 6.2.2.1 which describes the layout of the sequence header.

There's a sequence header at the beginning of the file, most of the time also at the start of each GOP.
Startcode 00 00 01 B3, there's no endcode, bits are as in table 6.2.2.1 padded with zero bits until the next startcode, all startcodes start at a byte boundary.

For HCenc there's no need to change these values, they are written correct for SD and HD.

[hwjohn]

Just read the thread on avs, cool, might be useful and free is great...


Yes, they are in the sequence header.
Download the MPEG2 2000 standard from
http://www.itu.int/rec/T-REC-H.262/en
and look at page 23, table 6.2.2.1 which describes the layout of the sequence header.

There's a sequence header at the beginning of the file, most of the time also at the start of each GOP.
Startcode 00 00 01 B3, there's no endcode, bits are as in table 6.2.2.1 padded with zero bits until the next startcode, all startcodes start at a byte boundary.

For HCenc there's no need to change these values, they are written correct for SD and HD.

Thanks Hank. Since I have your attention, let me ask you something about HCenc.

1) Does HCenc base the flags on the input resolution? It does this automagically, so I don't have to intentionally set anything, right?

2) I originally wanted to use HCenc to encode all the clips because it seems to give better quality and it has an option to set the BT.709/601 flags, but I had a problem getting it to stick to the average bitrate. Sometimes it would overshoot (for example setting 4000 average would yield 5100 average) and sometimes it would grossly undershoot (4000 average would be 900). Is there any reason for this?

[hank315]

1) Does HCenc base the flags on the input resolution? It does this automagically, so I don't have to intentionally set anything, right?
Yes, it's based on the input resolution and is set automatically, can be overruled with the *MPEGLEVEL command.

2) I originally wanted to use HCenc to encode all the clips because it seems to give better quality and it has an option to set the BT.709/601 flags, but I had a problem getting it to stick to the average bitrate. Sometimes it would overshoot (for example setting 4000 average would yield 5100 average) and sometimes it would grossly undershoot (4000 average would be 900). Is there any reason for this?
Depends on the source, if the clip is very short the bitrate mechanism won't work optimal.
Also, for example, if you want to encode a colorbar clip at high bitrate it will certainly have quantisation saturation, there's no real info in such a clip, all DCT coefficients are zero except the DC coef.
That's for an I-frame, in P and B-frames the MBs will all be skipped with zero motion vectors, giving a compression of nearly 100%.
LAVC encoders like QuEnc and Mencoder will generate a larger file because they don't skip MBs in B-frames.

[hwjohn]

Yes, it's based on the input resolution and is set automatically, can be overruled with the *MPEGLEVEL command.

I took a look and sure enough, the AUTO setting is doing it correctly (not that I doubted you). The bitrate comes out to 24500 and the buffer comes out to 9781248. Where does that bitrate come from? From my understanding, the max bitrate for HD DVD is 29400.

Depends on the source, if the clip is very short the bitrate mechanism won't work optimal.

The shortest clips are about 8 minutes, which must be too short since they give the most problems. Are there any tricks I can use to overcome this?

Also, for example, if you want to encode a colorbar clip at high bitrate it will certainly have quantisation saturation, there's no real info in such a clip, all DCT coefficients are zero except the DC coef.

That shouldn't cause any problems, should it? Shouldn't it just make it more compressible? (sorry I'm an idiot:))

That's for an I-frame, in P and B-frames the MBs will all be skipped with zero motion vectors, giving a compression of nearly 100%.
LAVC encoders like QuEnc and Mencoder will generate a larger file because they don't skip MBs in B-frames.

Now that I look, there is a significant reduction in size compared to QuEnc. As Borat would say, "Very nice..."

[dragongodz]

LAVC encoders like QuEnc and Mencoder will generate a larger file because they don't skip MBs in B-frames.

sorry but this statement needs correcting. they CAN do skipped MB's in B-frames. it depends on the settings used. for example do not use hi-quality or extreme settings and skipped MB's are produced no worries.

a small sample done with QuEnc for anyone that may doubt it.
http://d.turboupload.com/d/2154896/col-q.zip.html

[hwjohn]

@ dragongodz,
Thanks for the update on QuEnc. I think it is a great encoder as well. I was using High Quality, so that was the reason that the file sizes were larger.

@hank315
I didn't realize that the bitrate is divided by 400, so it is really 24500*400 = 9800000 bits/s = 9.8 Mbps. Is that correct? Again, I was under the impression that the max bitrate should be set to 29.4 Mbps for HD DVD. I hope you don't mind, but I asked the same question over at AVS from dr1934, who has been helping me there as well. Thanks!

[hank315]

sorry but this statement needs correcting. they CAN do skipped MB's in B-frames. it depends on the settings used. for example do not use hi-quality or extreme settings and skipped MB's are produced no worries.

a small sample done with QuEnc for anyone that may doubt it.
http://d.turboupload.com/d/2154896/col-q.zip.html
Uh... my wrong, you're right, mostly I use hi-quality settings so I never noticed it.

I didn't realize that the bitrate is divided by 400, so it is really 24500*400 = 9800000 bits/s = 9.8 Mbps. Is that correct? Again, I was under the impression that the max bitrate should be set to 29.4 Mbps for HD DVD. I hope you don't mind, but I asked the same question over at AVS from dr1934, who has been helping me there as well. Thanks!
The bitrate is stored in multiples of 400 bits so 24500 is really 9.8 Mb/s.
The max. bitrate for HD DVD is 29.4 Mb/s but it can be set lower.

That shouldn't cause any problems, should it? Shouldn't it just make it more compressible?
No problems, it will compress *very* good, the file dragongodz posted is only 430 Kb/s.

[hwjohn]

Uh... my wrong, you're right, mostly I use hi-quality settings so I never noticed it.


The bitrate is stored in multiples of 400 bits so 24500 is really 9.8 Mb/s.
The max. bitrate for HD DVD is 29.4 Mb/s but it can be set lower.


No problems, it will compress *very* good, the file dragongodz posted is only 430 Kb/s.

Thanks for the help Hank. We are also attempting to make an AVCHD disk for Blu-ray players, but that is proving more difficult. Do you know of any way to get MP4 video with no audio track muxed and into a .mov container? (sorry I know it is off topic)

[hwjohn]

hank315,
I was wrong over at AVS, HCenc is encoding the bitrate properly. I misunderstood how it worked, my apologies. We are having another problem with it though. If you get a chance, check out the AVS thread, it explains everything.

Thanks a lot for all the help, and happy Thanksgiving.