NAL HRD has been committed!

[shon3i]

I'll investigate tonight, but I had a buffer underflow in Scenarist today in a clip of 3 mins (it stops after 6 seconds of mux) with an avg bitrate of 25 Mbps (which is quite 'standard') and only one audio (DD at 640Kbps). It could be the vbv-bufsize set to 30.000 so tonight I'll try with a more conservative value.
What exact settings you use for buffer and max rate? I discovered in past (on earlier HRD patches aslo) that if use buffer different that max rate, scenarist discover potential buffer uder/overflow and stop muxing. I found if i set VBV 30000/30000 or 15000/15000 or anything same for buffer and maxrate scenarist mux flawlessy. IIRC CBP removal delay should be always 0.9, i don't know is that same as x264 VBV-Int option, which i try to adjust, to achieve that delay, but without success, i even get VBV errors durring encode :). I read somewhere in scenarist documentation that delay should be always near 0.9. Other encoders (Cinevision, MC, Elecard) keep that on 0.7-1.0 nevemind i set for buffer/maxrate, while with x264 can be anything, for example if i set 30000/40000 it will be around 0.5-0.6, if buffer is higer will be over 1.0 and around 1.80

[mp3dom]

I have used 30.000 for vbv-bufsize and 40.000 for vbv-maxrate which, if I'm not mistaken, are also the standard values. I haven't tried the CBP delay, nice suggestion, I'm trying it!

[kieranrk]

Can you upload the encoded clip somewhere (or send it to me privately) so I can see if it's a real buffer underflow or Scenarist being wrong.

[mp3dom]

You have a PM. :)

[shon3i]

I have used 30.000 for vbv-bufsize and 40.000 for vbv-maxrate which, if I'm not mistaken, are also the standard values.Exactly, but removal delay will be 0.6 in that case which is maybe not recommended.

Can you upload the encoded clip somewhere (or send it to me privately) so I can see if it's a real buffer underflow or Scenarist being wrong. You not find any underflows in stream itself, its problem in VBV delay that must be some constant value. I will post some examples, of different encoders

[mp3dom]

Anyway it's kinda strange that I've encoded full Elephants Dream with even higher settings (same vbv_bufsize/vbv_maxrate but avg bitrate raised from 25Mbps to 30Mbps) without any problem (and it's a 10min clip where the clip that fail to mux is only 3 minutes and scenarist fail after 2 seconds!)

[shon3i]

Anyway it's kinda strange that I've encoded full Elephants Dream with even higher settings (same vbv_bufsize/vbv_maxrate but avg bitrate raised from 25Mbps to 30Mbps) without any problem (and it's a 10min clip where the clip that fail to mux is only 3 minutes and scenarist fail after 2 seconds!)
What exactly settings? you use?

[mp3dom]

keyint at 24, min keyint at 2, bpyramid, 4 reference, 3 bframes, 4 slices, 30 mbps of bitrate, aq 1, all partitions, weight prediction at 2 and so trellis

[shon3i]

I mean VBV :)

[mp3dom]

Ah, sorry ^^; I'm using always 40K and 30K for maxrate and buffersize respectively.

[shon3i]

Ah, sorry ^^; I'm using always 40K and 30K for maxrate and buffersize respectively.
Can you try with 30K for both max rate and buffer, you will not loose any quality anyway. Just for test. Because CBP removal will be then 0.9

[mp3dom]

Tested with both vbv values at 30K and Scenarist always throws a buffer underflow... :(

[kieranrk]

Well the file you sent me is valid. The H.264 spec only specifies a cap for the initial_cbp_removal_delay in VBR mode.

[mp3dom]

So is Scenarist wrong or there's really a underflow? It complains at about 2 secs.

[kieranrk]

So is Scenarist wrong or there's really a underflow? It complains at about 2 secs.

There's definitely no underflow.

[mp3dom]

From a certain point of view I'm glad, from the other I'm a bit sad because I can't find a way to let Scenarist accept that footage encoded with x264. Anyway, thank you very much for the patience

[shon3i]

mp3dom can you send me source and encoded clip, for testing on PM?

[mp3dom]

The original clip is too huge for my adsl line (we are speaking of 2.7GiB of a YV12 video already compressed in lossless way, also is a working project so I can not redistribute it). I can send the portion that I've already sent to kieranrk and it's about 5 sec of video but I doubt that it's possible to reproduce the underflow (or the problem) with few seconds...

[shon3i]

Ok, can send then 5 sec of source and encoded video which fail to mux?

[shon3i]

Btw i found rule about VBV Buffer in Cinevision manual.

Bitrate buffer specifies the size of the virtual buffer verifier in bytes or bits. This value
should be adjusted to bitrate (Constant Bitrate) or to maximum bitrate (Variable Bitrate), to
avoid DTS/PTS underflows during muxing.

So if stream itself not contain any underflow, scenarist can detect possible underflow that not exist, but can result incorrect playback on SAP.

[renqian]

can't download the Binary .anyone can help me? :)

[mp3dom]

So if stream itself not contain any underflow, scenarist can detect possible underflow that not exist, but can result incorrect playback on SAP.

Ok but it seems there's no a simple way to avoid this because even with 30.000 for both bufsize and maxrate Scenarist gave me a buffer underflow... :confused:

[jpsdr]

Btw i found rule about VBV Buffer in Cinevision manual.

Bitrate buffer specifies the size of the virtual buffer verifier in bytes or bits. This value
should be adjusted to bitrate (Constant Bitrate) or to maximum bitrate (Variable Bitrate), to
avoid DTS/PTS underflows during muxing.



Does it mean that there is something to update/fix in the patch or in x264 ?

[shon3i]

Ok but it seems there's no a simple way to avoid this because even with 30.000 for both bufsize and maxrate Scenarist gave me a buffer underflow... :confused:
I don't know what you do but with your sample i sucessfuliy mux.

Here is mine cmd i used during encoding

x264.exe --preset slow --tune film --pass 2 --bitrate 30000 --level 4.1 --ref 4 --keyint 24 --min-ke
yint 2 --vbv-maxrate 30000 --vbv-bufsize 30000 --slices 4 --aud
--nal-hrd vbr --sar 1:1 --no-fast-pskip --no-dct-decimate --pic-struct --colorpr
im "bt709" --transfer "bt709" --colormatrix "bt709" --b-pyramid strict -o "AoD-Ut.264" "AoD-Ut.avi"

Does it mean that there is something to update/fix in the patch or in x264 ? No this is mean to keep possible near VBV-Bufsize and VBV-Maxrate to target bitrate. There is no reason to set VBV to 30/40 which is max for BD standard if your target bitrate is 10mbps for example. You can safetly use lower VBV valuses such 15/15

[mp3dom]

The only difference between my CMD and yours is that I've used an average bitrate of 25 Mbps rather than 30. I think the main difference is that you have an avg bitrate of 30 in a static 5sec clip which is somewhat too short for reproduce the problem (i think). I've the 3 min. clip and probably in the same static scene the avg bitrate is lower causing the underflow. It's not your fault, since I can't upload the full clip at all... :(

[jpsdr]

Is the --pic-struct also necessary for BR ?

[shon3i]

The only difference between my CMD and yours is that I've used an average bitrate of 25 Mbps rather than 30. I think the main difference is that you have an avg bitrate of 30 in a static 5sec clip which is somewhat too short for reproduce the problem (i think). I've the 3 min. clip and probably in the same static scene the avg bitrate is lower causing the underflow. It's not your fault, since I can't upload the full clip at all...

Ok i will now test with 25mbps maybe i have luck to reproduce problem :) That's why i told you to keep VBV's near to bitrate. Maybe you can try with VBV 25/25.

Is the --pic-struct also necessary for BR ? Only when interlaced/pulldown used, but some verifiers aslo notice that pic-struct should be used always. Anyway nobody have problem. Current HDR patch use when need.

[mp3dom]

Tried to encode the same file with same parameters (for buffer and maxrate) with CineVision and it mux without problems :confused:

[shon3i]

Tried to encode the same file with same parameters (for buffer and maxrate) with CineVision and it mux without problems :confused:
Well you can't watch like this, because encoders have completly different ratecontrols and decisions. If you look HRD model via some buffer analyser you see that graph is totaly different. Anyway i have same problem with some retail discs in past, few of these have realy unusual VBV (29/38) and very distant from target bitrate.

[mp3dom]

Yeah, I know. Anyway it doesn't work even with 25/25/25 (bitrate/maxrate/bufsize). I think I'll encode this video with something else... Good it's 3 min long but think what happends if it appear after a full film (I try to imagine after 5 days of encode, Scenarist refuse to mux)

Edit: With your exact settings it pass that point and continue to mux, however it fails 10 seconds after.

[shon3i]

Yeah, I know. Anyway it doesn't work even with 25/25/25 (bitrate/maxrate/bufsize). I think I'll encode this video with something else... Good it's 3 min long but think what happends if it appear after a full film (I try to imagine after 5 days of encode, Scenarist refuse to mux)

Edit: With your exact settings it pass that point and continue to mux, however it fails 10 seconds after.
Can you try with higher buffer than max-rate for example 25/25/30 (bitrate/maxrate/bufsize) or add to x264 cmd --vbv-int 0.4 to raise buffer/maxrate ratio

[mp3dom]

Tried both and nothing new (buffer underflow)

Edit: It could be that I've resolved. I'm testing a bit more...

Edit2: Resolved! Lowering the qcomp parameter does the trick for me.

[shon3i]

Tried both and nothing new (buffer underflow)

Edit: It could be that I've resolved. I'm testing a bit more...

Edit2: Resolved! Lowering the qcomp parameter does the trick for me.
haha i just want to suggest you that, or something that will change ratecontrol decisions :)

Btw how much you lower it? 0.5?

[Dark Shikari]

haha i just want to suggest you that, or something that will change ratecontrol decisions :)It's generally highly inadvisable to lower qcomp. If it "fixes" a problem, it's by luck, not something you should touch.

[mp3dom]

I've set to 0.5. Anyway as Dark Shikari said, probably it's a rare clip where the underflow occours. I've encoded some other clips without any underflow issue so better to leave qcomp as its default.

[shon3i]

I agree absolutely, but sometimes need to tweak it a little. Anyway That why MC/Cinevision have more two options for VBV/HRD which serve to prevent such possible events (VBV/HRD buffer fullness).

Btw when i already mention Mainconcept encoder, in his option there is several ability to write HRD info: Type 1, Type 2, Type 1+SEI and Type 2 without SEI. Aslo some other options are NAL, VCL, Both NAL and VCL is that can be usefull anyway. I know that Blu-Ray need Type 2.

[jpsdr]

Can this story of 'type 2' explain the problem mp3dom is having with his little clip ?
In that case, does it mean an update of the patch may be needed ?

[kieranrk]

Can this story of 'type 2' explain the problem mp3dom is having with his little clip ?
In that case, does it mean an update of the patch may be needed ?

There is no problem with the clip as far as I can see. I tried it on a copy of Muigenerator and it said that only 3 slices instead of 4 were present. Therefore I have to conclude that muigenerator is broken.

[shon3i]

There is no problem with the clip as far as I can see. I tried it on a copy of Muigenerator and it said that only 3 slices instead of 4 were present. Therefore I have to conclude that muigenerator is broken.
Yes it's realy wierd, but maybe something broken during cut, because i never had problem with slices and muigenerator, only with mp3dom sample.

Anyway, i just playing with Buffer Analyzer which scans whole BD structure (BD Quest Buffer Analyzer), and show three types of buffers, first of raw stream (like elecard), second of stream during mux (if muxer cause), and third of muxed inside transport stream. So as i already say, if stream itself is prefectly clean and without underflows, that not mean, will stay after muxing with other streams. Maybe is need to have some option like HRD buffer fullness to control that.

[kieranrk]

So as i already say, if stream itself is prefectly clean and without underflows, that not mean, will stay after muxing with other streams. Maybe is need to have some option like HRD buffer fullness to control that.

The muxer should handle all of these issues.

[shon3i]

Yes it should, for instance Scenarist automatically stop mux. But what is happend if we that stream mux with TsMuxer? Here i used mp3dom problematic sample, and muxed it with TsMuxer. Here is graph analysis.

http://img138.imageshack.us/img138/557/bufff.th.jpg (http://img138.imageshack.us/my.php?image=bufff.jpg)

Clearly visible that buffer after muxing (video only) goes over all restrictions.

Aslo Neuron's VBVChecker 1.7 with strict checking aslo found underflow with this file.

[mp3dom]

The problem of 3 slices comes up with the 'cut' made. I've encoded it with 4 slices and the full avc stream doesn't show any warning in MUI Generator. Probably the 'cut' has changed something in the header. Apart that clip, all the other clips that I've made are OK and muxed perfectly. Also BDQuest doens't show any warning regarding bitrate allocation (always pass). To me this patch is really fine (the only thing left is the pulldown that it could be really nice for standard definition as primary video, since it would be possible to encode a 480p24 stream and pulldown to 60i to adhere the BD specs)

[Guest]

Aslo Neuron's VBVChecker 1.7 with strict checking aslo found underflow with this file. It's not appropriate to use strict checking unless the encoder implements stuffing.

[kieranrk]

The transport buffer overflows, and note the BDAV-STD (System Target Decoder) at the top; therefore it's a transport issue.

On another note; what software produces that?

[Guest]

the only thing left is the pulldown DGAVCPulldown can apply pulldown.

[kieranrk]

(it would be possible to encode a 480p24 stream and pulldown to 60i to adhere the BD specs)

You're not allowed to do that. The framerate prior to pulldown must be valid.

DGAVCPulldown can apply pulldown.

As far as I know it doesn't correct HRD to match the new frame durations...(and apply appropriate hacks because HRD was not designed with pulldown in mind)

[mp3dom]

DGAVCPulldown can apply pulldown.
Yes, I know and I've tried it but the stream is refused by MUI generator

You're not allowed to do that. The framerate prior to pulldown must be valid.
Uhmm, are you sure? I've made an SD video (the source is 720x480, 23.976p) with CineVision. During importing it says (correctly) 23.98p. I've set the encoder as 'Primary Video', 720x480 and the encoder shows a 23.98->29.98 (pulldown) as final target. The final AVC file is detected as 29.98 fps and DGAVCDec show the incremental counter of 'repeated field flag'. This file is accepted and muxed by Scenarist (it's anyway detected as 29.98)

[kieranrk]

Uhmm, are you sure? I've made an SD video (the source is 720x480, 23.976p) with CineVision. During importing it says (correctly) 23.98p. I've set the encoder as 'Primary Video', 720x480 and the encoder shows a 23.98->29.98 (pulldown) as final target. The final AVC file is detected as 29.98 fps and DGAVCDec show the incremental counter of 'repeated field flag'. This file is accepted and muxed by Scenarist (it's anyway detected as 29.98)

The spec specifically says in "Table 9-29 – Allowed combinations of parameters for 525/60 video format" frame_mbs_only_flag must be zero. (i.e. must be interlaced)

[mp3dom]

Infact it is detected as 'frame' structure by DGAVCdec even if it has field pulldown (Scenarist, If I remember correctly, detect the file as interlaced). I think that some other parameters will tell that it's a soft pulldown or something similar (maybe pic-struct?)

Edit: Here is what DGAVCDec detect:
http://xs.to/thumb-F663_4B992852.jpg (http://xs.to/share-F663_4B992852.html)

And here is what Scenarist detect:
http://xs.to/thumb-BE4F_4B992852.jpg (http://xs.to/share-BE4F_4B992852.html)

[kieranrk]

Scenarist is wrong, DGAVCDec is right.