open-gop patch for x264 (committed)

[jpsdr]

@shon3i : I've PM'ed you the links for my file. If i remember correctly (and not mix with mp3dom...), you have tools to check Blu-Ray compliancy, so, if it's possible, if you could get the file and check it.
I've also had the small original sample, and if eventualy you want to play with it and see if you can reproduce something...

[mp3dom]

I've the verifier too, but shon3i has more updated version (so, more reliable)
Regarding the commercial encoders, it could be a suggestion to have bufsize=maxrate but anyway they don't force you to do so (personally I don't like it)

[jpsdr]

First, re-encoding the video with buffer=maxbr=15000 mux pass without buffer underflow error.
I've decided, just curious, to install and test bitrateviewer 2.1.1.
......... What was my surprise when i saw the results !!!!
The viewer reported average and peak, i must say in first, that average were always perfectly adequate with the average reported in the log file. So, i consider the information provided by the bitrate viewer reliable.

Here are the peaks values (in kpbs) for encoded files with maxbr=15000 and maxbuff=30000 :
17643, 18226, 18507, 18748, 17913, 16061,14696, 19224, 17180, 22257.
Wohh... !!!! Except for one, that's a way higher than the max bitrate set... Is it normal ??????? Or is there a big problem ???
With a GOP setting of 2s, i don't think these files stay Blu-Ray compliant !
Here are the peaks values (in kpbs) for encoded files with maxbr=15000 and maxbuff=15000 :
15338(*), 14962.
(*) : This was the file wich previously created buffer underflow.
The peak for the (*) file encoded with maxbr=15000 and maxbuff=30000 is 21490.
This is this file i've given the links to DS and kieranrk.
I've restart the encoding of the 10 file encoded with maxbr=15000 and maxbuff=30000 now with maxbr=15000 and maxbuff=15000.
Results not before monday...

[mp3dom]

High peaks doesn't create problem as long as the buffer is filled with data that either don't overflow or underflow. With maxrate=bufsize the 'excursions' of the peaks is more limited over time (hence I don't like it very much).

[jpsdr]

Well, when i limit the bitrate at 15000, i'm a little surprise to see peaks higher than 20000... But if it's normal...

[mp3dom]

It depends by what's the occupancy of the VBV. A high peak could be fully allowed for a short period if the VBV can manage it. I've seen some BD that sometimes have spikes over 60 Mbps (the max allowed is 40 Mbps)

[Trahald]

@jpsdr
I didnt see if you had already done this, but can you make the same encode exact same settings but without open-gop and use a totally unpatched x264 then see if it passes.

Attached is a newer version of the patch. it has changed a little.. the option is " --open-gop coded " if you are doing bluray and " --open-gop display " if you are doing anything else. It does not address the buffer issue (we are still determining if there is one)

[Trahald]

Originally Posted by jpsdr

Well, when i limit the bitrate at 15000, i'm a little surprise to see peaks higher than 20000... But if it's normal...

Yes.. it can peak to 25k and still be valid. its not the same as say clipping a audio stream. peaks are ok as long as the buffer can recover.

[shon3i]

Originally Posted by Trahald

It does not address the buffer issue (we are still determining if there is one)

Open gop have nothing with underflows here, it's alredy known with unpached bulids and before officialy Blu-Ray commit, even with your HRD patch.

There is probably some undocumented rule, that apply when is muxing to TS. Since all comercial encoders handle this.

[Dark Shikari]

Originally Posted by shon3i

Open gop have nothing with underflows here, it's alredy known with unpached bulids and before officialy Blu-Ray commit, even with your HRD patch.

There is probably some undocumented rule, that apply when is muxing to TS. Since all comercial encoders handle this.

Or your muxer is broken.

Stop spreading FUD about "underflows" that don't exist.

Originally Posted by IRC

07:18 <kierank> lol his file muxes fine

[shon3i]

Stop spreading FUD about "underflows" that don't exist.

An you stop FUD about broken muxers because muxers are just fine and newer have problem with other encoders, and always only with x264. All muxers fail at same place, some little restrictive like tsmuxer will mux stream, but clearly show after that BDAV-STD Model model is completly broken

[Dark Shikari]

Originally Posted by shon3i

All muxers fail at same place, some little restrictive like tsmuxer will mux stream, but clearly show after that BDAV-STD Model model is completly broken

Then where is the proof? Show me where x264 violates the NAL-HRD model, and I will fix it. Whining endlessly without showing proof serves no purpose whatsoever.

Nobody has been able to give me a stream produced by a recent x264 that fails validation in any stream analyzer that I own.

[kieranrk]

Originally Posted by shon3i

Open gop have nothing with underflows here, it's alredy known with unpached bulids and before officialy Blu-Ray commit, even with your HRD patch.

There is probably some undocumented rule, that apply when is muxing to TS. Since all comercial encoders handle this.

I have tested the stream jpsdr sent me and it worked fine with muigenerator.

[jpsdr]

Originally Posted by kieranrk

I have tested the stream jpsdr sent me and it worked fine with muigenerator.

Fortunately, otherwise, i'll not even be able to import it in my Blu-Ray project, and so start a mux.

[jpsdr]

Originally Posted by Trahald

@jpsdr
I didnt see if you had already done this, but can you make the same encode exact same settings but without open-gop and use a totally unpatched x264 then see if it passes.

I will, but it will take a little while.
As i said, as problem seems realy on the edge, i don't even know, because of the non-determinist part of x264, if redoing the same thing will not produce a file wich will pass. Indeed, i will also try it, doing a second time the "same" file, to see if it still produces the problem, or magicaly disapered.

[Dark Shikari]

I just spent the past 4 hours rewriting HRD. It's now almost sunrise. So before I completely conk out, try this build. It's extraordinarily beta; I'd be shocked if I didn't break at least something horribly. It's almost surely not correct... but we might as well try it. And it's going to be necessary if we intend to fix it.

Includes opengop, threadpool, and various other things that I've locally committed. The relevant patch.

(Thanks to Manao for help understanding a bit of the batshit insanity behind the HRD math.)

[jpsdr]

Thanks, i'll try, results at best in 8 hours, but most probably tomorrow. Good night...

[shon3i]

Originally Posted by Dark Shikari

(Thanks to Manao for help understanding a bit of the batshit insanity behind the HRD math.)

Many thanks. I hope it's right thing and solution for mux underflow problem. So i been partialy right?

[jpsdr]

Originally Posted by Dark Shikari

Includes opengop, threadpool, and various other things that I've locally committed.

Are you sure ? Because when i've tried it, it said "unrecognise option" on "--open-gop".

.... "unrecognisze option" on "--nal-hrd".
Doing things at down are not always good...
Better take a second look after a good sleep...

[shon3i]

@jpsdr,
both hrd and open gop have sub option

--nal-hrd vbr
--open-gop coded

"coded" is mandatory for blu-ray.