open-gop patch for x264 (committed)

[Trahald]

"ERROR" and then MUI Generation End. as usual when stream is broken/out-of-specs

cmd i used.

how many frames is it.. can you see the minimum # of frames it still fails? i havent been able to duplicate but then ive only used a 1000 frames.

[shon3i]

about 5000 frames is long clip, when i press "Create Files" button in MUI Generator i automaticly get error.

I will do test again with oter clip, to be sure.

[Trahald]

In case you missed it:

[17:34] < Daemon404> wewk: when you get back, ive foudn a bug in opengop. opengop + bframes > 0 + flv = x264 doesnt write it's header (the one with all the used params)
[17:38] < Daemon404> (apparently if the nal-hrd patch is also applied this goes away?)

the flv writer only consider the last sei in an aud (since vanilla x264 only writes one sei that works fine). Hrd adds an sei and open gop adds another one. so only the recovery point is getting written. anyways.. i'll fix it. should have a binary tomorrow or tues.

[Sharc]

OK, and what can open-gop do to improve this pulsing?
No idea. Perhaps someone else may explain?

[moviefan]

Is there a chance that Scenarist is simply wrong with its error about the open-gop stream? DXVA works fine with hrd and open-gop. Of course Blu-ray specs are more restrictive, but still...

[shon3i]

Is there a chance that Scenarist is simply wrong with its error about the open-gop stream? DXVA works fine with hrd and open-gop. Of course Blu-ray specs are more restrictive, but still...
I dont think so, i aslo test with Ateme Kyrion File Encoder (based on encavc 2 engine) which have OpenGOP feature (ateme always had opengop), scenarist passes stream fine, and report in stream properties that open gop is used

[Trahald]

http://www.mediafire.com/?toqfuqwgmzy
gcc: 4.4.2 (x86.generic.Komisar) fprofiled

patches used
x264_hrd_pd_interlace.16_r1369_fix.diff
x264_open_gop_11.diff
x264_adjust_cpb.diff (http://pastebin.com/m55643cd2)

the last patch fixes the flv issue and adjusts hrd for open gop.

This one passes scenarist w/a 5000 frame vid. (although the other worked ok for me to) but try this one.

[shon3i]

Still same :( what version of Sceanarist you have? I have 5.13 which came with MUI Generator version 1.0.3314.20055

EDIT: Without --open-gop switch everything passes, ateme passes. Maybe to send you ateme stream to examine? O somehow to help you?

[Trahald]

I found it.. missing a rbsp_trailing_bits( ) call after the recovery point sei.. will have a binary in a bit

[Trahald]

x264_x86_r1373_gcc_wewk_3.zip (http://www.mediafire.com/?vzchumfvmat)
gcc: 4.4.2 (x86.generic.Komisar) fprofiled

patches used
x264_hrd_pd_interlace.16_r1369_fix.diff
x264_open_gop_11.diff
x264_adjust_cpb_2.diff (http://pastebin.com/m3e630052)

[rack04]

Using this build:

x264_x86_r1373M (http://www.mediafire.com/?mltilymzyij)

Toolchain:
mingwrt-3.16
w32api-3.13
binutils-2.20
gcc-core-4.4.2
zlib-1.2.3
yasm-0.8.0
Patch:
x264_hrd_pd_interlace.16_r1369_fix.diff (http://komisar.gin.by/x.patch/last.used/x264_hrd_pd_interlace.16_r1369.diff)
x264_open_gop_11.diff (http://forum.doom9.org/showthread.php?p=1352134#post1352134)
x264_adjust_cpb_2.diff (http://pastebin.com/m3e630052)
Build:
./configure --extra-cflags="-march=core2"
make fprofiled VIDS="/c/x264/crew_4cif.y4m /c/x264/parkrun.1280x720.yuv"

I created the following sample. I would appreciate if someone with Sceanarist could verify whether or not this stream is Blu-ray compliant.

http://www.mediafire.com/?ylwgwxmzjgx

The command line used to generate the sample:

"C:\Program Files\x264\x264.exe" --profile high --preset placebo --tune animation --crf 18 --level 4.1 --ref 4
--bframes 3 --keyint 24 --min-keyint 1 --vbv-bufsize 30000 --vbv-maxrate 24000 --slices 4 --b-pyramid strict
--sar 1:1 --aud --nal-hrd --open-gop --output "C:\Personal\Videos\hd_dolby_catalyst_lossless-output.mkv"
"C:\Personal\Videos\hd_dolby_catalyst_lossless.avs"

[Trahald]

can you encode it to .264 (raw)? mkv extract removes the aud's (or maybe the muxer... idk) but either way they arent there after demux.

[shon3i]

@Trahald it's work now :) weee, passes everything.

@rack04 do a .264 i will test for you. Anyway be cearful with --vbv-bufsize 30000 --vbv-maxrate 24000 because lower maxrate than buffer in x264 produce vbv-init higher than 1sec which can produce not smooth playback even no under/overflows in stream.

[moviefan]

@rack04 do a .264 i will test for you. Anyway be cearful with --vbv-bufsize 30000 --vbv-maxrate 24000 because lower maxrate than buffer in x264 produce vbv-init higher than 1sec which can produce not smooth playback even no under/overflows in stream.

Does Scenarist complain in that case or do you only notice at playback?

[shon3i]

Scenarist won't mux that stream, it will detect as "possible buffer underflow" and will stop muxing. I can't see real happening with tsmuxer because i have completly menu structure in scenarist.

I don't know is problem in nal hrd or vbv itself, but with other encoders CPB init delay work totaly different.

[moviefan]

So would --vbv-bufsize 25000 --vbv-maxrate 25000 (which is the maxrate for --level 4.0) be safe? I intend to have at maximum two DTS 1536kbps audio streams (if that matters).

[shon3i]

which is the maxrate for --level 4.0maxrate for L4.0 is 24000. so use --vbv-bufsize 24000 --vbv-maxrate 24000, just dont use buffer greather than maxrate. With DTS 1.5 should be fine. Your cmd upper is just fine and it's L4.1 you should only change --vbv-maxrate 24000 to 30000

[moviefan]

Strange since x264 gives a warning if --level 4.0 --vbv-maxrate > 25000 that it should be no more than 25000... That's why I wrote 25000.

[shon3i]

Strange since x264 gives a warning if --level 4.0 --vbv-maxrate > 25000 that it should be no more than 25000... That's why I wrote 25000.
Because x264 follow H264 specification for specified level, and Blu-Ray have different settings.

[rack04]

can you encode it to .264 (raw)? mkv extract removes the aud's (or maybe the muxer... idk) but either way they arent there after demux.

@rack04 do a .264 i will test for you.

Here you go. Thanks.

http://www.mediafire.com/?mmmu4mkvlml

EDIT: Patching the latest git with x264_open_gop_11.diff fails.

[moviefan]

Are there any solutions concerning open-gop + qpfile yet?

[shon3i]

Are there any solutions concerning open-gop + qpfile yet?
what is wrong with this?

[moviefan]

I noticed a crash on Wednesday when using a qpfile to insert IDR keyframes for chapter marks and it turned out on IRC that Trahald had not considered testing on qpfiles. He said he was gonna fix it soon.

[Dark Shikari]

We're slightly overhauling this case; keyframes will now be "K" in the QPfile, and will be handled differently inside of x264. "K" means IDR as usual if opengop is off, and I + recovery point if opengop is on. You'll still be able to force IDR frames, too.

[moviefan]

So I need to replace the -I by -K in the qpfile now, if I want to prepare for chapter marks and encode with open-gop?

[Dark Shikari]

So I need to replace the -I by -K in the qpfile now, if I want to prepare for chapter marks and encode with open-gop?No. IDR frames still work just fine; the point is that a user might want to be able to specify either.

[moviefan]

Hm, but you remember the issue with "cost >= 0" on Wednesday that I pointed out when I was trying to encode with r1376 + hrd + open-gop? Does open-gop need a fix or do I have to change something in the qpfile? (a little confused now)

[Dark Shikari]

Hm, but you remember the issue with "cost >= 0" on Wednesday that I pointed out when I was trying to encode with r1376 + hrd + open-gop? Does open-gop need a fix or do I have to change something in the qpfile? (a little confused now)As I already said quite a few times, that was a problem with OpenGOP: it didn't support forced IDR frames correctly.

[Selur]

Just wondering: Any news on the open-gop patch?

[Daemon404]

Just wondering: Any news on the open-gop patch?

Indeed I would also like to know this.

[kieranrk]

Some tweaks might be necessary to get it to work properly with the HRD patch. Trahald has been busy recently and so hasn't had time to look at it.

[Selur]

Thanks for the info. :)

[Trahald]

busy == lazy/health issues.. im attaching a patch to test. I patched the last rev. by hand to 1510, added a couple lines to address intra-refresh combination, and generated this patch. i really didnt test it. i made a file and it plays and the hrd info looks right (kieranrk will need to look at it).

[kieranrk]

The HRD info looks right to me as well. I'd need to test it some more but it looks fine to me. I suppose people with Blu-Ray verifiers might want to test it to be on the safe side as well.

[shon3i]

If compile ... :)

[kieranrk]

http://dl.dropbox.com/u/2701213/x264/open%20gop/x264.exe

[jpsdr]

I assume, reading this, that open-gop is something supposed to be BR compliant ?
What is the benefit of it ?

[nurbs]

It is described in the first post of this thread.

[jpsdr]

Tested with the last rack 04 build, crashed with 'qpfile' (used to define chapters on blu-ray).
The issue is still present.

[Trahald]

can you post your qpfile and command line. i couldnt duplicate the crash here.

[jpsdr]

Here the command line used, x264 crash during the 1rst pass :

@echo off

SET E_SRC=%6%1.avs
SET E_DST=%3%1.264
SET STAT_FILE=%1.stats
SET TUNING=%4
SET LOG_FILE_1=%1_log_1.txt
SET LOG_FILE_2=%1_log_2.txt

REM Set of max bitrate (ici le bitrate max)
set MAX_BR=24000

REM Set of Buffer (ici le buffer)
set BUF_BR=30000

REM 1ère passe
x264_x86.exe --profile "high" --preset "placebo" --tune %TUNING% --pass 1 --bitrate %2 --stats %STAT_FILE% --level "4.0" --qpmin 0 --qpstep 1 --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --keyint 24 --min-keyint 2 --mvrange 511 --ref 6 --bframe 3 --b-pyramid "strict" --open-gop --subme 9 --trellis 1 --me "umh" --aq-mode 2 --aud --nal-hrd "vbr" --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --qpfile %5 --threads 0 --thread-input --output NUL %E_SRC% 2> %LOG_FILE_1%

REM 2ème passe
x264_x86.exe --profile "high" --preset "placebo" --tune %TUNING% --pass 2 --bitrate %2 --stats %STAT_FILE% --level "4.0" --qpmin 0 --qpstep 1 --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --keyint 24 --min-keyint 2 --mvrange 511 --ref 6 --bframe 3 --b-pyramid "strict" --open-gop --aq-mode 2 --aud --nal-hrd "vbr" --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --qpfile %5 --threads 0 --thread-input --output %E_DST% %E_SRC% 2> %LOG_FILE_2%

Bitrate (%2) : 4526
Tuning : animation

Chapter file :

2045 I -1
17192 I -1
32639 I -1
49540 I -1
63245 I -1
79971 I -1
93840 I -1
109392 I -1
124439 I -1
139582 I -1
155038 I -1
156716 I -1
157436 I -1


Partial result log file :

avs [info]: avisynth 2.6+ detected, forcing conversion to YV12
avs [info]: 1280x720p 1:1 @ 24000/1001 fps (cfr)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile High, level 4.0
[20.7%] 32640/157449 frames, 32.70 fps, 4538.87 kb/s, eta 1:03:37 <= Crash here

[Trahald]

Silly error i missed. only came up with opengop/bpyrstrict/qpfile combo.

anywho.. 1 line and its fixed. Please test.

[jpsdr]

As soon as there will be a build.... (rack04 ??? :rolleyes: ), i'll test it.

[JEEB]

Young men, there's no need to feel down♪～

Clicky (http://x264.fushizen.eu/files/opengop/).

Otherwise unpatched, contains lavf/ffmpeg. 64bit one has ffmpeg's SSE optimizations disabled.

[jpsdr]

I don't know what ffmpeg is using for in x264, as i only use avisynth script with avi lagarith YV12 files for input...
Otherwise, thanks, i'll test it this evening (for me).

[julius666]

I don't know what ffmpeg is using for in x264, as i only use avisynth script with avi lagarith YV12 files for input...
Otherwise, thanks, i'll test it this evening (for me).

If you don't use any filter on your input video in avisynth just opening with avisource, then you don't need avisynth. You could directly feed x264 with your source.
That's what ffmpeg is for.

[Midzuki]

I don't know what ffmpeg is using for in x264, as i only use avisynth script with avi lagarith YV12 files for input...
Otherwise, thanks, i'll test it this evening (for me).

If you don't use any filter on your input video in avisynth just opening with avisource, then you don't need avisynth. You could directly feed x264 with your source.
That's what ffmpeg is for.

Well, me neither doesn't use the ffmpeg-input capabilities of the "new generation" of x264 builds, and to me, their greater filesize becomes "bloatware". :devil: This is the reason why I've asked for "7-zipped" builds — believe me, there still exist many many people who DON'T have a fast Internet connection, which means,

in the "dial-up world",
1MB == 5 minutes of download time (typically)

[shon3i]

verifier's does not show any problems with OpenGOP on short sample. I will encode whole movie to recheck.

Anyway, how far this from merging to GIT?

[Trahald]

It was approved by pengvado (about 100 revisions ago pre nal-hrd) so should be close. Ill wait till you do a longer test then submit to 'da man' for approval.

[Atak_Snajpera]

Does this patch solve "frame pulsing problem" with small --keyint (for example 24) and low bitrate?