NAL HRD has been committed!

[shon3i]

The transport buffer overflows, and note the BDAV-STD (System Target Decoder) at the top; therefore it's a transport issue.

On another note; what software produces that?
With tsmuxer as i said, which not make aditional checking during mux for possible undeflows. But sonic scenarist will probably make same, but they are have checker, which automaticlly stop muxing and prevent this as in mp3dom case, and report underflow. Adjusting qpcomp in x264 solove mp3dom problem, this change HRD model which passes muxing.

Also BDQuest doens't show any warning regarding bitrate allocation (always pass)Bitrate!=Buffer Analysis.

[mp3dom]

Bitrate!=Buffer Analysis.
Yeah! You're right, my mistake!

[kieranrk]

With tsmuxer as i said, which not make aditional checking during mux for possible undeflows. But sonic scenarist will probably make same, but they are have checker, which automaticlly stop muxing and prevent this as in mp3dom case, and report underflow. Adjusting qpcomp in x264 solove mp3dom problem, this change HRD model which passes muxing.

Bitrate!=Buffer Analysis.

I mean which program created the graphic.

You haven't grasped what I've said. In that picture the buffer analysis is of the Transport stream buffers (which are part of the STD). It is not the fault of x264 when STD buffers overflow. (There's one exception but that's not allowed in Blu-Ray) The muxer needs to deliver packets at the correct rate.

[shon3i]

I mean which program created the graphic.Aha ;) sorry, with BD Quest Buffer Analyzer :)

It is not the fault of x264 when STD buffers overflow. But what then make that muxer "incorrectly" mux stream, i don't use other streams (video,audio,sub). I use tsmuxer for example because is "non strict", while Sonic Scenarist will tend to do same, but stops when see things goes wrong. There is must some rule for VBV or HRD buffer, something that like min/max (not mean for -vbv-bufsize), some tolerance.

Btw mp3dom, report that Cinvision stream with "same" settings passes mux, i know is not good comparing these encoders with x264 because ratecontrols are different, but i know that Mainconcept have very strict VBV, from ealier days.

[kolak]

Aha ;) sorry, with BD Quest Buffer Analyzer :)

But what then make that muxer "incorrectly" mux stream, i don't use other streams (video,audio,sub). I use tsmuxer for example because is "non strict", while Sonic Scenarist will tend to do same, but stops when see things goes wrong. There is must some rule for VBV or HRD buffer, something that like min/max (not mean for -vbv-bufsize), some tolerance.

Btw mp3dom, report that Cinvision stream with "same" settings passes mux, i know is not good comparing these encoders with x264 because ratecontrols are different, but i know that Mainconcept have very strict VBV, from ealier days.

This is exactly what I was talking about- it looks like x264 can produce compliant streams, but in reality (muxing stage) there are still unpredictable problems.

I've never had problems with Cinevision even if I quite often push all settings to the limits and done already hundreds encodes for BD.

[Dark Shikari]

This is exactly what I was talking about- it looks like x264 can produce compliant streams, but in reality (muxing stage) there are still unpredictable problems.Then why blame x264? If there are problems in the muxing stage, your muxer is broken, plain and simple.

[kolak]

Then why blame x264? If there are problems in the muxing stage, your muxer is broken, plain and simple.

Can be, but muxer works fine with 3 other encoders (each of them is from different company). Muxer also uses some information from AVC headers, so problem can be there.
It's more about making them working together- there are many things which are not specified in BD spec and this may be responsible for problems.

[kieranrk]

there are many things which are not specified in BD spec and this may be responsible for problems.

Feel free to go into more detail...

[Revgen]

Let's be honest here.

Do you think the scenarist programmers are going to fix the problem?

[kolak]

Feel free to go into more detail...

I'm not technical enough, but have seen early stages of HD DVD and used few beta software (Cinemacraft HDe and MemoryTech software for HD DVD authoring). There were issues at the beginning, because BD spec is just a small section of AVC spec, so there is room for different interpretation of unspecified parameters.

[shon3i]

This is exactly what I was talking about- it looks like x264 can produce compliant streams, but in reality (muxing stage) there are still unpredictable problems.

I've never had problems with Cinevision even if I quite often push all settings to the limits and done already hundreds encodes for BD.
Well now i can't say what is precent of this streams encoded with x264 that fail mux in scenarist, because i use various HRD patches, so they are not been so perfect. For now i don't have any issue. So probably percent is something about 1-2% not more But i am still afraid because mp3dom reported this issue. Anyway i see most of retail BD that can't be remuxed with same error, that disc is mostly form small distributors, so i suspect they probably use cheap encoders/muxers.

It's more about making them working together- there are many things which are not specified in BD spec and this may be responsible for problems. I totaly agree with this.

Do you think the scenarist programmers are going to fix the problem? Scenarist don't have problem. Tsmuxer detect same.

[Dark Shikari]

Let's be honest here.

Do you think the scenarist programmers are going to fix the problem?Why do we care? Our eventual goal is to create an open source Blu-ray authoring pipeline, which means Scenarist can die in a fire.

[Revgen]

Why do we care? Our eventual goal is to create an open source Blu-ray authoring pipeline, which means Scenarist can die in a fire.

I wasn't aware of this.

That's good news.

[kolak]

Well now i can't say what is precent of this streams encoded with x264 that fail mux in scenarist, because i use various HRD patches, so they are not been so perfect. For now i don't have any issue. So probably percent is something about 1-2% not more But i am still afraid because mp3dom reported this issue. Anyway i see most of retail BD that can't be remuxed with same error, that disc is mostly form small distributors, so i suspect they probably use cheap encoders/muxers.

I totaly agree with this.

Scenarist don't have problem. Tsmuxer detect same.

What do you use for demuxing? Are you sure that AACS is striped properly?

I would be careful about these re-authoring statements.
BD is not DVD- there are not many authoring software, which can do project ready for replication. Probably 98% (or more) BDs in the shops will be made with Scenarist or Blu-print, which are quite strict.

I use DVD Reuathor and already found few projects (which I made in the first place) not ripping and re-authorign properly.
BD can be only worse- it's very new compared to DVD.

[shon3i]

What do you use for demuxing? Are you sure that AACS is striped properly?Absolutely, tryed with both comercial and free software, BDReauthor, BD Demuxer and some ts demuxer and eac3to/tsmuxer as for free soft. I very sure that demuxed streams are good.

[kolak]

Absolutely, tryed with both comercial and free software, BDReauthor, BD Demuxer and some ts demuxer and eac3to/tsmuxer as for free soft. I very sure that demuxed streams are good.

Neither of these programs is very accurate.
BDReauthor comes from the same company as DVDReauthor and I would not blindly trust in it.
We tried it on simple project and didn't restore it as original. It's very useful in some cases, but not 100% reliable.
It can "fix" AVC header, so out of spec file can be imported into Scenarist, but this is not a proper fix, just trick to pass Scenarist checks-it's very dangerous option.

Do a project, replicate it and try to rip and restore- you may be surprised.

[shon3i]

Do a project, replicate it and try to rip and restore- you may be surprised.
I already have that expirience. But here is something different, i tryed with all possible TS demuxers, raw streams aren't been identical, but all failed on same place, so it's definitly wrong in stream itself.

@Dark Shikari i just want to ask? how can be on muxer side, because all present muxers (not only Blu-Ray) produce sam behavior. Aslo why qpcomp solve the problem, and vbv-int (in my earler case). Does not this make conclusion that something missing. All encoders i tryed for BD encoding are usualy are focus on BD encoding only, they are more restrictive then we know. I want to say that it is very difficult to make general use Encoder, to cover all places

[Dark Shikari]

@Dark Shikari i just want to ask? how can be on muxer side, because all present muxers (not only Blu-Ray) produce sam behavior. Aslo why qpcomp solve the problem, and vbv-int (in my earler case). Does not this make conclusion that something missing.If every day I drive my car to work, and then one day I bring a can of soda with me, and I get into a car accident, does that mean the can of soda is responsible for the crash?All encoders i tryed for BD encoding are usualy are focus on BD encoding only, they are more restrictive then we know. I want to say that it is very difficult to make general use Encoder, to cover all placesAre you sure that the Blu-ray encoders are using the same VBV restrictions as x264? Because I'm pretty sure that you aren't.

Have you checked any with vbv.pl?

[kolak]

If every day I drive my car to work, and then one day I bring a can of soda with me, and I get into a car accident, does that mean the can of soda is responsible for the crash?Are you sure that the Blu-ray encoders are using the same VBV restrictions as x264? Because I'm pretty sure that you aren't.

Have you checked any with vbv.pl?

All pro encoders are very strict in terms of VBV and that's why I've never had any muxing problems and as I said- I quite often push parameters to the max.

[Dark Shikari]

All pro encoders are very strict in terms of VBV and that's why I've never had any muxing problems and as I said- I quite often push parameters to the max.But do you know that they're actually maxing out the VBV parameters? No, you don't, because you haven't checked.

It's quite possible they're intentionally leaving extra leeway beyond what is necessary (e.g. for the sake of the muxer).

[shon3i]

Are you sure that the Blu-ray encoders are using the same VBV restrictions as x264? Because I'm pretty sure that you aren't.They are not in first place offcourse, but beleve me i do anything to make it, even i scan streams with various tools to make it indentical. Only thing that i notice is CPB removal relay is always been different, for example cinevision encode always keep it around 0.9, while x264 very vary. I didn't checke with VBV.pl, because is as other buffer analyzers. Here we need something that know what is Blu-Ray restriction (like BD Quest Buffer Analyzer), that will contorol HRD model that follow VBV model etc, in Blu-Ray range.

Like kolak say, i neve have that expirience with other encoders. I always check stream with analyzer and notice, that buffer came out for example 29999 if i use default HRD/VBV buffersize which is 30000. But that is not porblem, i mean i tryed exatly same settings with x264.

[kieranrk]

I always check stream with analyzer and notice, that buffer came out for example 29999 if i use default HRD/VBV buffersize. But that is not porblem, i mean i tryed exatly same settings with x264.

EDIT: never mind; Blu-Ray fixed the issue I was talking about long before TS did.

[shon3i]

What can be done then? Probably other encoders have some workaround

[kieranrk]

What can be done then? Probably other encoders have some workaround

Fix the muxer.

Do you have any apps that show how many bits per second are entering the transport buffer?

[Dark Shikari]

What can be done then? Probably other encoders have some workaroundWell, if you don't tell us the results of vbv.pl, we can't figure out now can we?

What I want to know is if you set maxrate=40k, what bufsize does vbv.pl come up with? And if you set bufsize=30k, what maxrate does vbv.pl come up with?

[shon3i]

Fix the muxer.
Well i can contact Authors of tsmuxer and sonic support and send problematic stream.

Do you have any apps that show how many bits per second are entering the transport buffer? I think i not, but if you have something on mind (name of program) i can check maybe i have it on work.

Well, if you don't tell us the results of vbv.pl, we can't figure out now can we?

What I want to know is if you set maxrate=40k, what bufsize does vbv.pl come up with? And if you set bufsize=30k, what maxrate does vbv.pl come up with? Ok, no problem i will do everything you need, but tomorow now i go to sleep ;)

[shon3i]

I don't know why i get this message when i try to use vbv.pl

List form of pipe open not implemented at C:\vbv.pl line 32

I have ActivePeal 5.10.1 installed on Windows 7 64bit

cmd that i use is

vbv.pl --bufsize 30000 video.mkv

[kieranrk]

cmd that i use is

vbv.pl --bufsize 30000 video.mkv

Do it on a raw .264 file.

[shon3i]

Do it on a raw .264 file.
Well in help of VBV.pl stay what is supported for infile:

Infile can be:
a text file containing frame sizes in bytes, 1 per line
a text file containing the output of `x264 -v`
matroska (only the first track is analyzed)

Maybe i have some old version of VBV.pl

[kieranrk]

Well in help of VBV.pl stay what is supported for infile:


Yes but when you are muxing Blu-Rays, you are not muxing mkv files. There are differences between muxing into raw .264 and mkv.

[shon3i]

Yes but when you are muxing Blu-Rays, you are not muxing mkv files. There are differences between muxing into raw .264 and mkv.
Agree, but what can i do then ?

[kieranrk]

x264 -v should probably work.

[mp3dom]

Today I've tried to made an encode. I've used the "zones" parameter to change some values (trellis and deadzone). x264 shows a warning saying that VBV parameters could not be modified with HRD activated. Since it's the first time that I've received this warning (also I haven't changed any VBV value, only trellis and deadzone) I'm asking if this is ok (only a warning), or if the zones is trying to change the VBV parameters in any way. Thanks.

[shon3i]

@kieranrk, since i can't find good stream to replicate muxing underflow problem (mp3dom has no access to master anymore), i found some little older documents "Method for preventing buffer underflow during digital transport stream transmission, multiplexing and splicing" where clearly say that HRD/VBV model need to be adjusted and there is min/max. I can send you pdf document in case you don't have, just send me pm.

[kieranrk]

I'm asking if this is ok (only a warning), or if the zones is trying to change the VBV parameters in any way. Thanks.

This is a bug and it's been fixed locally.

[MadMonkey57]

ouch.
I've authored about 100 BD5s with this toolchain (including like 15 with x264-rev1471-nal-hrd-1.32) and so far all was fine. And now, I have a problem.
My toolchain:

- kubuntu 9.10 64 bits
- reading source video (m2ts file): mplayer SVN-r30531 (build from source)
- video encoding: x264-rev1471 + nal-hrd 1.32 (build from source)
- audio reading/encoding: ffmpeg + sox + aften
- tsmuxer 1.10.6
- home made scripts to perform all the work (reading source file, calculating bitrates, encoding audio/video, muxing, creating iso)

All isos ended up 4.3GB and played OK in my Pana BD35, but my last iso is 4.1GB and is rejected by the SAP.

When I play the m2ts within the iso, mplayer says at the very beginning:
[h264 @ 0xd4da00]reference picture missing during reorder 0.6% 0 0
[h264 @ 0xd4da00]reference picture missing during reorder
[h264 @ 0xd4da00]Missing reference picture
[h264 @ 0xd4da00]Missing reference picture
[h264 @ 0xd4da00]reference picture missing during reorder 0.6% 0 0
[h264 @ 0xd4da00]Missing reference picture
[h264 @ 0xd4da00]reference picture missing during reorder 0.6% 0 0
[h264 @ 0xd4da00]Missing reference picture

mediainfo shows:
Video
ID : 4113 (0x1011)
Menu ID : 1 (0x1)
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.0
Format settings, CABAC : Yes
Format settings, ReFrames : 4 frames
Duration : 2h 1mn
Bit rate : 3 816 Kbps
Nominal bit rate : 3 680 Kbps
Width : 1 280 pixels
Height : 720 pixels
Display aspect ratio : 16:9
Frame rate : 23.976 fps
Resolution : 8 bits
Colorimetry : 4:2:0
Scan type : Progressive
Bits/(Pixel*Frame) : 0.173
Stream size : 3.25 GiB (79%)
Writing library : x264 core 89
Encoding settings : cabac=1 / ref=4 / deblock=1:-1:-1 / analyse=0x3:0x133 / me=umh / subme=10 /
psy=1 / psy_rd=1.00:0.15 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 /
fast_pskip=1 / chroma_qp_offset=-3 / threads=12 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 /
constrained_intra=0 / bframes=3 / b_pyramid=0 / b_adapt=2 / b_bias=0 / direct=3 / wpredb=1 /
wpredp=2 / keyint=24 / keyint_min=2 / scenecut=40 / intra_refresh=0 / rc_lookahead=24 / rc=2pass / mbtree=1 /
bitrate=3680 / ratetol=1.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / vbv_maxrate=15000 /
vbv_bufsize=15000 / ip_ratio=1.40 / aq=1:1.00 / pulldown=0 / nal_hrd=vbr
I will investigate deeper but it looks it *could* be an encoding issue.

Does it ring any bell ?

[kolak]

ouch.
I've authored about 100 BD5s with this toolchain (including like 15 with x264-rev1471-nal-hrd-1.32) and so far all was fine. And now, I have a problem.
My toolchain:

- kubuntu 9.10 64 bits
- reading source video (m2ts file): mplayer SVN-r30531 (build from source)
- video encoding: x264-rev1471 + nal-hrd 1.32 (build from source)
- audio reading/encoding: ffmpeg + sox + aften
- tsmuxer 1.10.6
- home made scripts to perform all the work (reading source file, calculating bitrates, encoding audio/video, muxing, creating iso)

All isos ended up 4.3GB and played OK in my Pana BD35, but my last iso is 4.1GB and is rejected by the SAP.

When I play the m2ts within the iso, mplayer says at the very beginning:
[h264 @ 0xd4da00]reference picture missing during reorder 0.6% 0 0
[h264 @ 0xd4da00]reference picture missing during reorder
[h264 @ 0xd4da00]Missing reference picture
[h264 @ 0xd4da00]Missing reference picture
[h264 @ 0xd4da00]reference picture missing during reorder 0.6% 0 0
[h264 @ 0xd4da00]Missing reference picture
[h264 @ 0xd4da00]reference picture missing during reorder 0.6% 0 0
[h264 @ 0xd4da00]Missing reference picture

mediainfo shows:
Video
ID : 4113 (0x1011)
Menu ID : 1 (0x1)
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.0
Format settings, CABAC : Yes
Format settings, ReFrames : 4 frames
Duration : 2h 1mn
Bit rate : 3 816 Kbps
Nominal bit rate : 3 680 Kbps
Width : 1 280 pixels
Height : 720 pixels
Display aspect ratio : 16:9
Frame rate : 23.976 fps
Resolution : 8 bits
Colorimetry : 4:2:0
Scan type : Progressive
Bits/(Pixel*Frame) : 0.173
Stream size : 3.25 GiB (79%)
Writing library : x264 core 89
Encoding settings : cabac=1 / ref=4 / deblock=1:-1:-1 / analyse=0x3:0x133 / me=umh / subme=10 /
psy=1 / psy_rd=1.00:0.15 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 /
fast_pskip=1 / chroma_qp_offset=-3 / threads=12 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 /
constrained_intra=0 / bframes=3 / b_pyramid=0 / b_adapt=2 / b_bias=0 / direct=3 / wpredb=1 /
wpredp=2 / keyint=24 / keyint_min=2 / scenecut=40 / intra_refresh=0 / rc_lookahead=24 / rc=2pass / mbtree=1 /
bitrate=3680 / ratetol=1.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / vbv_maxrate=15000 /
vbv_bufsize=15000 / ip_ratio=1.40 / aq=1:1.00 / pulldown=0 / nal_hrd=vbr
I will investigate deeper but it looks it *could* be an encoding issue.

Does it ring any bell ?

1280x720 at 24p is not officially supported combination on BD.

[mp3dom]

1280x720 is supported at 23.976p/24p/50p/60p

[kieranrk]

Post a raw .264 sample where this occurs. The ts muxer may have used the wrong DTS for whatever reason.

[Sharc]

1280x720 is supported at 23.976p/24p/50p/60p
60p? Isn't it 59.94 according to Blu-ray spec?

[kolak]

1280x720 is supported at 23.976p/24p/50p/60p

Ups- sorry:)

[shon3i]

60p? Isn't it 59.94 according to Blu-ray spec?
Exactly :)

[mp3dom]

Yes, I wrote 60p but I mean 59.94 (60000/1001) :P

[MadMonkey57]

Post a raw .264 sample where this occurs. The ts muxer may have used the wrong DTS for whatever reason.

I've made a couple of x264 encodings. Depending on the video length / bitrate, sometimes I get no error, sometimes I get an "cpb_count 33 invalid" (http://www.megaupload.com/?d=K6NAK8U8) playing with mplayer...

EDIT: Damn it. Upgraded mplayer to latest SVN (mplayer is used to feed x264 and for playing resulting .264 file), ran the exact same encoding as before and I get:
[h264 @ 0xd64fe0]illegal num_reorder_frames 26
[h264 @ 0xd64fe0]illegal num_reorder_frames 26

EDIT 2: same behaviour
- a change in bitrate shows "cpb_count 33 invalid" 3 times now
- another change in bitrate shows no error...

I can post the "illegal num_reorder_frames 26" if necessary.

[kieranrk]

Looks like mplayer is complaining about issues that don't exist in that sherlock holmes clip.

[MadMonkey57]

Some other thing I have noticed:

- I run twice the exact same commands and end up in .264 file that are different (binary comp). Might be normal, I don't know
- ran x264-rev1376 + x264_hrd_pd_interlace.16_r1369 and had one "llegal num_reorder_frames 47" twice

This is really confusing. Maybe it's just mplayer.

Still I don't understand why my BD35 rejects the resulting .264 stream...

[MadMonkey57]

Got it. Serious PEBKAC issue... My subtitle file was corrupted hence muxing problem... Duh!

Sorry guys.

[LoRd_MuldeR]

Some other thing I have noticed:

- I run twice the exact same commands and end up in .264 file that are different (binary comp). Might be normal, I don't know
- ran x264-rev1376 + x264_hrd_pd_interlace.16_r1369 and had one "llegal num_reorder_frames 47" twice

This is really confusing. Maybe it's just mplayer.

First of all x264 should be deterministic (i.e. it should always give the identical output for the identical input), unless you explicitely specify the "--non-deterministic" parameter.

And this usually works for me. For example I regularly do bit-by-bit comparison to test different GCC versions with x264.

Anyway, if MPlayer complains about the stream then this means that libavcodec's H.264 decoder complains. That's the H.264 decoder used in practically all OpenSource applications.

[MadMonkey57]

First of all x264 should be deterministic
Thanks for the clarification. Don't know why I couldn't verify it. Maybe one of those PEBKAC, who knows... I'll try again tomorrow.

Anyway, if MPlayer complains about the stream then this means that libavcodec's H.264 decoder complains. That's the H.264 decoder used in practically all OpenSource applications.
You're right. mplayer's latest SVN won't help here.

[Dark Shikari]

First of all x264 should be deterministic (i.e. it should always give the identical output for the identical input), unless you explicitely specify the "--non-deterministic" parameter.Not with VBV...