open-gop patch for x264 (committed)

[Trahald]

I hacked up a open gop patch. all it really does is change the idr interval higher and force i-frames in at the max-keyint and flag them as recovery points. then clears the reference list of all frames behind the i when the first non-b is ready to be encoded. So for the most part it just interferes with frame decisioning a little and does some reference maintenance. x264 already supported i (small i) frames so it was fairly easy.

in a very few tests i get about 1.5% smaller size. thats with both at b-adapt 1(x264 default). B-adapt 2 with open-gop gives 4% improvement over a closed gop encode at b-adapt 1 and 5% over b-adapt 2 with closed-gop. fades seem to do well bitrate wise also.

This patch really is only useful if you have small gops (ie bluray) . my tests were at min keyint of 12. This patch is not for your production encodes as it is experimental. Do not create any binaries with this patch that may be in automatic upgrading scripts to protect the innocent.

there will be 2 attachments. one is the patch by itself and one with hrd(interlaced) 13 patch on top of it (as hrd has to be adjusted to open gop and also the patches update the same part of encoder.c ). this hrd version does not replace the current hrd patch.

again both are experimental

[Dark Shikari]

I hacked up a open gop patch. all it really does is change the idr interval higher and force i-frames in at the max-keyint and flag them as recovery points. then clears the reference list of all frames behind the i when the first non-b is ready to be encoded. So for the most part it just interferes with frame decisioning a little and does some reference maintenance. x264 already supported i (small i) frames so it was fairly easy.Did you make it so scenecuts are closed-gop but I-frames at max-keyint (not scenecuts) are open gop?

[Trahald]

Did you make it so scenecuts are closed-gop but I-frames at max-keyint (not scenecuts) are open gop?
yessir. my thought was that max keyint is some arbitrary thing, but a scenecut is x264 saying a key frame is needed here (re:quality). i plan to go back in and look for more cases where open gop makes sense.

[techouse]

interesting.....

[shon3i]

Can somebody compile with opengop feature and x264_win_zone_parse_fix_05.diff and x264_hrd_pulldown.13_interlace.diff

[moviefan]

Also AutoVAQ.02 in addition to shon3i's proposal...?

[Trahald]

This is an experimental build with Open GOP for those that want to play w/it.

http://www.mediafire.com/file/4muyzwzzozh/x264_1171_opengop.zip

fprofiled

x264_open_gop_hrd.2.diff
x264_win_zone_parse_fix_05.diff
x264_AutoVAQ.03.diff

[rack04]

This is an experimental build with Open GOP for those that want to play w/it.

http://www.mediafire.com/file/4muyzwzzozh/x264_1171_opengop.zip

fprofiled

x264_open_gop_hrd.2.diff
x264_win_zone_parse_fix_05.diff
x264_AutoVAQ.03.diff

Thanks. Can you make a build with "x264_open_gop_hrd.2.diff " and "x264_hrd_pulldown.13_interlace.diff" for testing Blu-ray sources?

[nurbs]

I guess "x264_open_gop_hrd.2.diff" contains the "x264_hrd_pulldown.13_interlace.diff"

[rack04]

I guess "x264_open_gop_hrd.2.diff" contains the "x264_hrd_pulldown.13_interlace.diff"

According to Trahald:

there will be 2 attachments. one is the patch by itself and one with hrd(interlaced) 13 patch on top of it (as hrd has to be adjusted to open gop and also the patches update the same part of encoder.c ). this hrd version does not replace the current hrd patch.

[shon3i]

@Trahald, thanks for patch and experimental build, results are more than expected


With Open GOP
x264.exe" --pass 2 --bitrate 1000 --stats "drift.stats" --level 4.1 --keyint 24 --min-keyint 1 --ref 4 --mixed-refs --no-fast-pskip --bframes 3 --b-adapt 2 --weightb --direct auto --deblock -1:-1 --subme 9 --trellis 2 --psy-rd 1.0:0.2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 17999 --vbv-maxrate 16000 --me umh --threads 12 --thread-input --aq-mode 0 --sar 1:1 --progress --no-dct-decimate --no-psnr --no-ssim --output "drift.264" "drift.avs" --sar 1:1 --mvrange 511 --aud --nal-hrd --open-gop

avis [info]: 720x528 @ 25.00 fps (986 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast FastShuffle SSEMisalign LZCNT
x264 [info]: profile High, level 4.1

x264 [info]: slice I:56 Avg QP:26.41 size: 16729
x264 [info]: slice P:321 Avg QP:28.84 size: 7113
x264 [info]: slice B:609 Avg QP:31.49 size: 2709
x264 [info]: consecutive B-frames: 4.3% 16.7% 40.1% 38.8%
x264 [info]: mb I I16..4: 30.9% 60.1% 9.1%
x264 [info]: mb P I16..4: 5.3% 5.4% 0.7% P16..4: 55.8% 8.5% 10.5% 0.0% 0.0% skip:13.7%
x264 [info]: mb B I16..4: 0.1% 0.2% 0.0% B16..8: 42.1% 0.9% 1.2% direct: 3.9% skip:51.5% L0:41.9% L1:51.5% BI: 6.6%
x264 [info]: 8x8 transform intra:55.4% inter:77.8%
x264 [info]: direct mvs spatial:89.2% temporal:10.8%
x264 [info]: coded y,uvDC,uvAC intra:54.6% 62.4% 22.4% inter:16.8% 32.3% 4.0%
x264 [info]: ref P L0 75.2% 15.3% 6.7% 2.8%
x264 [info]: ref B L0 86.0% 10.2% 3.9%
x264 [info]: kb/s:987.8
encoded 986 frames, 18.08 fps, 988.32 kb/s

Without Open GOP
x264.exe" --pass 2 --bitrate 1000 --stats "drift.stats" --level 4.1 --keyint 24 --min-keyint 1 --ref 4 --mixed-refs --no-fast-pskip --bframes 3 --b-adapt 2 --weightb --direct auto --deblock -1:-1 --subme 9 --trellis 2 --psy-rd 1.0:0.2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 17999 --vbv-maxrate 16000 --me umh --threads 12 --thread-input --aq-mode 0 --sar 1:1 --progress --no-dct-decimate --no-psnr --no-ssim --output "drift.mkv" "drift.avs" --sar 1:1 --mvrange 511 --aud --nal-hrd

avis [info]: 720x528 @ 25.00 fps (986 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast FastShuffle SSEMisalign LZCNT
x264 [info]: profile High, level 4.1

x264 [info]: slice I:72 Avg QP:27.25 size: 14636
x264 [info]: slice P:364 Avg QP:29.19 size: 6626
x264 [info]: slice B:550 Avg QP:31.74 size: 2569
x264 [info]: consecutive B-frames: 10.4% 15.5% 37.7% 36.3%
x264 [info]: mb I I16..4: 33.6% 58.6% 7.8%
x264 [info]: mb P I16..4: 4.6% 4.7% 0.6% P16..4: 56.6% 8.0% 10.9% 0.0% 0.0% skip:14.6%
x264 [info]: mb B I16..4: 0.1% 0.2% 0.0% B16..8: 41.0% 0.9% 1.2% direct: 3.5% skip:53.0% L0:41.3% L1:52.1% BI: 6.6%
x264 [info]: 8x8 transform intra:54.9% inter:77.6%
x264 [info]: direct mvs spatial:88.9% temporal:11.1%
x264 [info]: coded y,uvDC,uvAC intra:53.1% 63.5% 24.5% inter:16.9% 32.2% 4.1%
x264 [info]: ref P L0 76.2% 15.1% 6.4% 2.3%
x264 [info]: ref B L0 88.2% 9.0% 2.8%
x264 [info]: kb/s:989.6
encoded 986 frames, 20.15 fps, 990.23 kb/s


and I have to ask how this patch is far from the final version :) because work perfectly

[Dark Shikari]

Patch review for open-GOP patch.

1. Use spaces, not tabs. I don't want to have to correct this for you.

2. Use consistent style. x264 puts the bracket on the next line, not the same line as the if statement.

3. What is this?

if( h->param.b_open_gop && h->param.i_keyint_max < 250 ) { //using default as cutoff
h->param.i_keyint_max = 250;
}

This doesn't make any sense.

4. Don't add extra spaces for no reason:

@@ -1472,9 +1479,17 @@ int x264_encoder_encode( x264_t *h,
return 0;
}

+
if( h->fenc->i_type == X264_TYPE_IDR )

5. What is this for?

+ if( frames[1]->i_frame - h->frames.i_last_i >= h->param.i_keyint_max_i && h->param.b_open_gop )
+ {
+ frames[1]->i_type = X264_TYPE_I;
+ frames[1]->i_is_gop_start = 1;
+ return;
+ }
+
if( num_frames == 1 )
{
no_b_frames:
frames[1]->i_type = X264_TYPE_P;
+ if( frames[1]->i_frame - h->frames.i_last_i >= h->param.i_keyint_max_i && h->param.b_open_gop )
+ {
+ frames[1]->i_type = X264_TYPE_I;
+ frames[1]->i_is_gop_start = 2;
+ return;
+ }
if( h->param.i_scenecut_threshold && scenecut( h, &a, frames, 0, 1 ) )
frames[1]->i_type = idr_frame_type;
return;
@@ -516,6 +530,12 @@ no_b_frames:
{
int num_bframes;
int max_bframes = X264_MIN(num_frames-1, h->param.i_bframe);
+ if( frames[1]->i_frame - h->frames.i_last_i >= h->param.i_keyint_max_i && h->param.b_open_gop )
+ {
+ frames[1]->i_type = X264_TYPE_I;
+ frames[1]->i_is_gop_start = 3;
+ return;
+ }
if( h->param.i_scenecut_threshold && scenecut( h, &a, frames, 0, 1 ) )
{
frames[1]->i_type = idr_frame_type;
@@ -526,6 +546,12 @@ no_b_frames:

for( j = 1; j < num_bframes+1; j++ )
{
+ if( frames[j]->i_frame - h->frames.i_last_i >= h->param.i_keyint_max_i && h->param.b_open_gop )
+ {
+ frames[j]->i_type = X264_TYPE_I;
+ frames[j]->i_is_gop_start = 4;
+ return;
+ }
if( h->param.i_scenecut_threshold && scenecut( h, &a, frames, j, j+1 ) )
{
frames[j]->i_type = X264_TYPE_P;
@@ -533,6 +559,12 @@ no_b_frames:
}
frames[j]->i_type = X264_TYPE_B;
}
+ if( frames[num_bframes+1]->i_frame - h->frames.i_last_i >= h->param.i_keyint_max_i && h->param.b_open_gop )
+ {
+ frames[num_bframes+1]->i_type = X264_TYPE_I;
+ frames[num_bframes+1]->i_is_gop_start = 5;
+ return;
+ }
frames[num_bframes+1]->i_type = X264_TYPE_P;
}
else if( h->param.i_bframe_adaptive == X264_B_ADAPT_FAST )
@@ -552,12 +584,22 @@ no_b_frames:
#define INTER_THRESH 300
#define P_SENS_BIAS (50 - h->param.i_bframe_bias)
frames[1]->i_type = X264_TYPE_B;
-
+ if( frames[1]->i_frame - h->frames.i_last_i >= h->param.i_keyint_max_i && h->param.b_open_gop )
+ {
+ frames[1]->i_type = X264_TYPE_I;
+ frames[1]->i_is_gop_start = 6;
+ return;
+ }
for( j = 2; j <= X264_MIN( h->param.i_bframe, num_frames-1 ); j++ )
{
int pthresh = X264_MAX(INTER_THRESH - P_SENS_BIAS * (j-1), INTER_THRESH/10);
int pcost = x264_slicetype_frame_cost( h, &a, frames, 0, j+1, j+1, 1 );
-
+ if( frames[j]->i_frame - h->frames.i_last_i >= h->param.i_keyint_max_i && h->param.b_open_gop )
+ {
+ frames[j]->i_type = X264_TYPE_I;
+ frames[j]->i_is_gop_start = 7;
+ return;
+ }
if( pcost > pthresh*i_mb_count || frames[j+1]->i_intra_mbs[j+1] > i_mb_count/3 )
{
frames[j]->i_type = X264_TYPE_P;
@@ -570,6 +612,12 @@ no_b_frames:
else
{
int max_bframes = X264_MIN(num_frames-1, h->param.i_bframe);
+ if( frames[1]->i_frame - h->frames.i_last_i >= h->param.i_keyint_max_i && h->param.b_open_gop )
+ {
+ frames[1]->i_type = X264_TYPE_I;
+ frames[1]->i_is_gop_start = 8;
+ return;
+ }
if( h->param.i_scenecut_threshold && scenecut( h, &a, frames, 0, 1 ) )
{
frames[1]->i_type = idr_frame_type;
@@ -578,6 +626,12 @@ no_b_frames:

for( j = 1; j < max_bframes+1; j++ )
{
+ if( frames[j]->i_frame - h->frames.i_last_i >= h->param.i_keyint_max_i && h->param.b_open_gop )
+ {
+ frames[j]->i_type = X264_TYPE_I;
+ frames[j]->i_is_gop_start = 9;
+ return;
+ }
if( h->param.i_scenecut_threshold && scenecut( h, &a, frames, j, j+1 ) )
{
frames[j]->i_type = X264_TYPE_P;
@@ -585,6 +639,12 @@ no_b_frames:
}
frames[j]->i_type = X264_TYPE_B;
}
+ if( frames[max_bframes+1]->i_frame - h->frames.i_last_i >= h->param.i_keyint_max_i && h->param.b_open_gop )
+ {
+ frames[max_bframes+1]->i_type = X264_TYPE_I;
+ frames[max_bframes+1]->i_is_gop_start = 10;
+ return;
+ }
frames[max_bframes+1]->i_type = X264_TYPE_P;
}
}

That's loads of code duplication, and I don't see why opengop should have code in slicetype decision at all. Why do we need the same code run 10 times with 10 different values for i_is_gop_start?

6. Why do we have two parameters that do the same thing?

int i_keyint_max; /* Force an IDR keyframe at this interval */
+ int i_keyint_max_i;

[Trahald]

@Dark Shikari
1,2 & 4 . the only cleanup i did was to remove fprintf's. I'll take care of that stuff tho

3 & 6. when open_gop is true then i_keyint_max ends up meaning (max IDR interval) which i push to 250 unless its value is higher. i_keyint_max_i represents the interval of little i frames. So...
i_keyint_max_i = i_keyint_max; // the users input
i_keyint_max = i_keyint_max > 250 ? i_keyint_max : 250;

5. yeah yuck. thats me throwing spaghetti at the wall and seeing what sticks. i'll work on it. the multiple values thing was more for debugging but since its harmless i left it.

[Dark Shikari]

3 & 6. when open_gop is true then i_keyint_max ends up meaning (max IDR interval) which i push to 250 unless its value is higher. i_keyint_max_i represents the interval of little i frames. So...
i_keyint_max_i = i_keyint_max; // the users input
i_keyint_max = i_keyint_max > 250 ? i_keyint_max : 250;I don't think this is really a good solution.

When someone says they want a keyframe interval of 90 frames, they mean they want a keyframe every 90 frames; an open-GOP I-frame is still a keyframe, so that counts. Yes, we'll have to redefine what i_keyint_max does in the documentation to not literally say "IDR."

If, for technical reasons, we still need to force IDR frames every once in a while despite open-GOPs being perfectly fine keyframes, the value should be really large.

[Trahald]

I don't think this is really a good solution.

When someone says they want a keyframe interval of 90 frames, they mean they want a keyframe every 90 frames; an open-GOP I-frame is still a keyframe, so that counts. Yes, we'll have to redefine what i_keyint_max does in the documentation to not literally say "IDR."

If, for technical reasons, we still need to force IDR frames every once in a while despite open-GOPs being perfectly fine keyframes, the value should be really large.

only 1 IDR is needed in a stream. (well there i a limit on poc_msb but its too huge to ever be hit) keeping the value was helpful since x264 makes some non-gop size related decisions based on the value. but i can remove the extra integer and use the users input as max_keyint then either...
1. anyplace there is a max_keyint referenced that relates to scenecut, replace w/integer 400000 when open gop is true
2. remove max_keyint from the scenecut routines (w/or w/out opengop) and do whatever keyint is used for there some other way

[Dark Shikari]

keeping the value was helpful since x264 makes some non-gop size related decisions based on the value.Which ones? Scenecut should use the time since the last keyframe, not the last IDR, for its decision.
1. anyplace there is a max_keyint referenced that relates to scenecut, replace w/integer 400000 when open gop is true
2. remove max_keyint from the scenecut routines (w/or w/out opengop) and do it some other wayAgain, "max keyint" in scenecut routines should be the distance between keyframes, not the distance between IDR frames.

[Trahald]

Which ones? Scenecut should use the time since the last keyframe, not the last IDR, for its decision.All... Problem is I dont want x264 thinking 'i only have 24 frames to work with.' in my early tests default decisioning resulted in open gop being the same size or sometimes bigger than closed gop equivelant. I got the most benefit (ie output closer in size to a stream that had a large key interval , which was my goal) when x264 thought there was a big distance between frames. Again, "max keyint" in scenecut routines should be the distance between keyframes, not the distance between IDR frames.yeah.. i didnt really make my post clear.

in either scenario, max_keyint_i is done away with and there is only max key int
in scenarion 1. its a cosmetic removal as im just hardcoding the larger value. but.. this would code wise remove the extra value. max_keyint would be commented as maximum distance between IDR frames or Recovery points.

OR

scenario 2. here again, there is only one max_keyint . but here we remove any scenecut decisioning that relies on the distance between keyintervals, whatever that interval is. scenecuts in streams at max_keyint 24 would be in the same spots as streams with max_keyint 50, etc.

[Dark Shikari]

All... Problem is I dont want x264 thinking 'i only have 24 frames to work with.'What do you mean? x264 only cares about the number of frames it has to "work with" in two places:

1. B-frame decision, in which this doesn't help, since you're still forced to end the GOP, even if it's on an open GOP.

2. Scenecut decision, in which what you are doing is outright lying to the scenecut algorithm about where the last keyframe actually was. I will not accept a patch that does this for any reason whatsoever. If you think that the scenecut algorithm is suboptimal, fix it separately.in either scenario, max_keyint_i is done away with and there is only max key int
in scenarion 1. its a cosmetic removal as im just hardcoding the larger value.Why have a larger value at all if you're not going to allow it to be adjusted? Why do we have to force IDR frames every once in a while?scenario 2. here again, there is only one max_keyint . but here we remove any scenecut decisioning that relies on the distance between keyintervals, whatever that interval is. scenecuts in streams at max_keyint 24 would be in the same spots as streams with max_keyint 50, etc.This is a bad idea and will reduce encoding efficiency.

What is wrong with my original suggestion of just doing it the sensible way--leaving scenecut as it is, and treating all open GOPs as keyframes? Why do we have to munge the scenecut algorithm like this, independent of what your patch is actually doing?

[Selur]

Again, "max keyint" in scenecut routines should be the distance between keyframes, not the distance between IDR frames.
how about two two parameters 'max IDR interval' and 'may key frame interval' ? (I need IDR frames to cut lossles, don't I?)

[Dark Shikari]

how about two two parameters 'max IDR interval' and 'may key frame interval' ? (I need IDR frames to cut lossles, don't I?)Well, you probably can cut at a recovery point, the cutting software just has to be smarter.

[G_M_C]

If, for technical reasons, we still need to force IDR frames every once in a while despite open-GOPs being perfectly fine keyframes, the value should be really large.


only 1 IDR is needed in a stream.


If add these up, then the vanue DS is speaking of, could be set eqqual to the number of frames in the input-stream ? If you then make first frame of the output-stream an IDR, you wont get another one (if not needed/requested).

[Trahald]

If add these up, then the vanue DS is speaking of, could be set eqqual to the number of frames in the input-stream ? If you then make first frame of the output-stream an IDR, you wont get another one (if not needed/requested).
That would still fall under 'lying' to the scene-cut routine.

[Trahald]

how about two two parameters 'max IDR interval' and 'may key frame interval' ? (I need IDR frames to cut lossles, don't I?)A lot of sample streams i got when making h264info were cut at open gop. all that happens is first few b-frames (any that pts < the pts of the i-frame) are discarded during playback. same as what happens when seeking in an open gop stream. and yeah... just need a smart enough cutter. or maybe dumb enough, ie most probably just look for i frames , not necessarily IDRs.

[Trahald]

@DS
I understand the changes you request for code-sake and I'll try to make them. Only that it probably will have to start with 'git reset --hard' except for maybe the SEI part. since i wrote it in a way that required me to understand the least about frame type decisioning. so might be a while. I really didnt make it with a commit in mind. just that noone else has bothered so i made something that worked (result wise if not code wise) mostly for myself although I dont mind sharing.

Would you explain why x264 even cares at all about max keyframe interval in scene cut decisions. AFAIK that is unique to x264. Im willing to code it (time permitting) without agreeing w/it but it would help if i could wrap my head around the idea behind it.

[LoRd_MuldeR]

Well, you probably can cut at a recovery point, the cutting software just has to be smarter.

So we will have "recovery points" in between IDR frames, that are not IDR frames themselves ???

Can you explain a bit: How does that work? I mean, if we start decoding at a Non-IDR frame, we will require references from previous frames, don't we?

Okay, as far as I understand, a recovery point says "output will be good again in x frames, if you start decoding here". There may be "scrambled" output until frame x is reached.

But how does the encoder know? Does it keep track of all macro-blocks and write a recovery point after each macro-block has been "refreshed" at least once? Or what?

[Dark Shikari]

@DS
I understand the changes you request for code-sake and I'll try to make them. Only that it probably will have to start with 'git reset --hard' except for maybe the SEI part. since i wrote it in a way that required me to understand the least about frame type decisioning. so might be a while. I really didnt make it with a commit in mind. just that noone else has bothered so i made something that worked (result wise if not code wise) mostly for myself although I dont mind sharing.I don't think it's that hard at all; what you've done is almost right. All you have to do is consider recovery points to be keyframes, and you're done. You don't need to understand frametype decision at all; you only need to not change it.Would you explain why x264 even cares at all about max keyframe interval in scene cut decisions. AFAIK that is unique to x264. Im willing to code it (time permitting) without agreeing w/it but it would help if i could wrap my head around the idea behind it.Because if x264 is near the end of the GOP, x264 is more willing to place extra I-frames than if it had just placed one two frames ago.So we will have "recovery points" in between IDR frames, that are not IDR frames themselves ???

Can you explain a bit: How does that work? I mean, if we start decoding at a Non-IDR frame, we will require references from previous frames, don't we?

Okay, as far as I understand, a recovery point says "output will be good again in x frames, if you start decoding here". There may be "scrambled" output until frame x is reached.

But how does the encoder know? Does it keep track of all macro-blocks and write a recovery point after each macro-block has been "refreshed" at least once? Or what?That's called periodic intra refresh, and this patch isn't related to that.

[Manao]

But how does the encoder know? Does it keep track of all macro-blocks and write a recovery point after each macro-block has been "refreshed" at least once? Or what?The encoder can enforce it by preventing frames temporally after the I to reference frames temporally before the I. Don't forget that even if a frame is in the reference buffer (aka DPB), you are allowed not to use it (they'll be flushed out of the DPB in good time, ie, when you'll have encoded enough new reference frames)

[LoRd_MuldeR]

The encoder can enforce it by preventing frames temporally after the I to reference frames temporally before the I.

Hmm, I see. But what is the benefit of placing a "recovery" Non-IDR I-Frame instead of a "true" IDR-Frame, if we don't reference the frames before that I-Frame anyway?

That's called periodic intra refresh, and this patch isn't related to that.

Does x264 support/use periodic intra refresh yet?

[Dark Shikari]

Does x264 support/use periodic intra refresh yet?No; it's really only useful for extremely-low-delay situations where your VBV is too small to allow periodic I-frames.

[Manao]

Hmm, I see. But what is the benefit of placing a "recovery" Non-IDR I-Frame instead of a "true" IDR-Frame, if we don't reference the frames before that I-Frame anyway?You can still open the gop, while an IDR forces to close the gop.

[LoRd_MuldeR]

You can still open the gop, while an IDR forces to close the gop.

Sorry, what exactly does that mean then? :o

I though using references from the previous GOP (that is: before the last I-Frame) inside the current GOP is the definition of an "open" GOP :confused:

If we now decide to not use references from before the I-Frame (to get a valid "recovery point"), wouldn't that result in "closed" GOP's ???

[Manao]

In temporal order : closed gop : PbbbPbbbPI
open gop : PbbbPbbbbIIn coding order :closed gop : PPbbbPbbbI
open gop : PPbbbIbbbbNote how, in coding order (ie, in the bitstream), the b frames temporally before the I frame are actually after the I frame. That's why the gop is called open. If you start decoding at the I, you won't be able to decode the b (you'll have to skip them), but it's no big deal because they are temporally before the I. IDR prevents you to use bframes in the open gop fashion because it flushes the reference buffer (thus, were you to use an IDR in an open gop situation, the bframes wouldn't not be able to reference the preceding P : they wouldn't be bframes anymore).

[LoRd_MuldeR]

Ah, I see :thanks:

[juGGaKNot]

Trahald can you explain in n00b terms how does this help/affect the quality of fades and where is it usefull ( high resolutions only ? )

thnx.

This patch really is only useful if you have small gops (ie bluray) . my tests were at min keyint of 12. This patch is not for your production encodes as it is experimental. Do not create any binaries with this patch that may be in automatic upgrading scripts to protect the innocent.

Hmm a link about gops/gops-keyint relation ?

[Manao]

As Trahald already said, this patch is useful mostly when you use a low max keyint interval. The reason for its usefulness is that, without the patch, x264 considers that key frames are IDR, which are special picture types that prevents the previous picture type to be a B frame. So, even if x264 would have wanted to use a B frame, if the following frame must be a key frame because of max keyint, without the patch the B will be transformed into a P frame, which isn't as good.

Furthermore, open gops will help reduce the problem of intra "pumping" at low bitrates and small gops.

Finally, a gop, in the mpeg2 terminology, is a group of picture starting on an intra frame, and stopping just before the next one. For mpeg2, all intra frames are entry points in the video, so settings the gop period set the entry points period. For h264, not all intra frames are sync points, but somehow the gop remained aliased to the idea of sync point (and not intra frames). So a gop can be seen as starting on an entry point (be it a IDR or a plain I frame) and stopping just before the next one.

With that definition, max-keyint sets the maximum gop size, and min keyint sets the minimum gop size.

[juGGaKNot]

As Trahald already said, this patch is useful mostly when you use a low max keyint interval.

Yes, thnx, most of this i have read ( not in such depth ) but i am asking about where to use them.

Sets the maximum interval between IDR-frames (aka keyframes) in x264's output.

As far as i know when i render uncompressed from vegas every frame is a keyframe, is this wrong ?

http://i286.photobucket.com/albums/ll105/juGGaKNot4cs/temps/keyframes_x264.png

[LoRd_MuldeR]

I don't know what VEGAS does. But x264 in "lossless" mode uses I-Frames (probably IDR and Non-IDR) as well as P-Frames.

Only B-Frames are disabled, because they don't help in the special case of lossless encoding.

If you want to enforce an I-Frame only stream for some reason, you can simply set the value of min_keyint to 1 ;)

[Manao]

It's normal. Uncompressed video doesn't have temporal dependencies between frames, so you can seek instantaneously wherever you want in an uncompressed video. The same would be true if you were encoding with mjpeg (all frames are intra), or ASP/AVC with only intra frames (max-keyint set to 1 with x264 for example).

As for when to use open gop : always. There are no (valid) reason not to use it. It's better, period.

[juGGaKNot]

It's normal. Uncompressed video doesn't have temporal dependencies between frames, so you can seek instantaneously wherever you want in an uncompressed video. The same would be true if you were encoding with mjpeg (all frames are intra), or ASP/AVC with only intra frames (max-keyint set to 1 with x264 for example).

As for when to use open gop : always. There are no (valid) reason not to use it. It's better, period.

K, i will test so maxkeyint 1 ?

Again, "max keyint" in scenecut routines should be the distance between keyframes, not the distance between IDR frames.

[Manao]

K, i will test so maxkeyint 1 ?What for ?

[juGGaKNot]

What for ?

"max keyintn should be the distance between keyframes"

and in uncompressed every frame is keyframe

So i assumed 1

What should i use for uncompressed ?

LE : using --open-gop --keyint 50 --min-keyint 1 for a 1184x666p50 uncompressed sample.

should i use --keyint 500 ? seek time is not that important.

[Dark Shikari]

What should i use for uncompressed ?Don't compress with x264, if you do, it's not uncompressed.

[juGGaKNot]

Don't compress with x264, if you do, it's not uncompressed.

Isn't there a 4:4:4 lossless in x264 ? :)

Anyway i meant the source is always uncompressed and not blu-ray or others so

--open-gop --keyint 50 --min-keyint 1 is optimal for me no ?

what about keyint, the x10 default or the fps ?

[Dark Shikari]

Isn't there a 4:4:4 lossless in x264 ? :)No, there's lossless, but it isn't 4:4:4.

[juGGaKNot]

No, there's lossless, but it isn't 4:4:4.

I was kinda kidding because i was thinking that you were joking with the "don't use x264" stuff but i guess my english is not that decent.

Back to topic, uncompressed, open gop, --open-gop --keyint 50 --min-keyint 1 is optimal for me no ?

[neuron2]

Back to topic, uncompressed, open gop, --open-gop --keyint 50 --min-keyint 1 is optimal for me no ? There's no need for x264.exe if you want uncompressed. I think you don't understand the meaning of what you are saying.

Dark wasn't joking. AVC is a *compression* standard.

[LoRd_MuldeR]

In other words:
Video that was compressed in a lossless way (which x264 can do) is not uncompressed anymore. It's losslessly compressed, but compressed.
x264 cannot output uncompressed video, because it's a H.264 encoder. And H.264 is compressed by definition. It allows both, lossy and lossless compression.
If you want uncompressed video, you must use somthing like "raw" YUV or RGB data...

[juGGaKNot]

K, my english is bad

I am just saying that my input is uncompressed ( because it is linked to keyframe interval ) so i should use --open-gop --keyint 50 --min-keyint 1 right ? etc etc

I do not want to encode lossless with x264.

[LoRd_MuldeR]

The way how your input was encoded doesn't matter for x264's frame-type decision!

You should keep the default "--keyint" and "--min-keyint" values, unless you have a reason to change them ;)

Rule of thumb:
The minimum keyframe interval should be equal to FPS, the maximum keyframe interval should be equal to FPS * 10.

So it would be 25 and 250 for 25 fps content - the defaults.

(In case you need to use a very small key-frame interval for some reason, the open-gop patch may be useful)

[juGGaKNot]

(In case you need to use a very small key-frame interval for some reason, the open-gop patch may be useful)

K, back were we started, what are those cases ?

[nm]

Blu-ray: 1-2 second GOPs, AFAIK, which means you should use something like "--keyint 24 --min-keyint 1" for 24 fps video.

[juGGaKNot]

Blu-ray: 1-2 second GOPs, AFAIK, which means you should use something like "--keyint 24 --min-keyint 1" for 24 fps video.

K, blu-ray has 1-2 second gops, my uncompressed avi ( made from picture streams ) has every frame as a keyframe so the gops are at every frame right ?

so --keyint fps --min-keyint 1 for me ?

[nm]

No, you are mixing things up again. Type of the input video has nothing to do with this as LoRd_MuldeR already said. Keyint is selected to accommodate target restrictions (for example Blu-ray authoring which I mentioned above) or to keep seek intervals sane (people usually like < 10 second skips so they set keyint=10*fps).

[G_M_C]

K, blu-ray has 1-2 second gops[...]
so --keyint fps --min-keyint 1 for me ?

OK, let's make it simple for you. When encoding for Blu-ray/AVCHD then --keyint fps (rounded to the nearest integer) --min-keyint 1 is correct.

And other then that; Forget the specifications of the source. They've got nothing to do with it. Open your source using AVISynth, and you're all set.

The only thing the source provides x264 is .... a sequence of frames.

[juGGaKNot]

No, you are mixing things up again. Type of the input video has nothing to do with this as LoRd_MuldeR already said. Keyint is selected to accommodate target restrictions (for example Blu-ray authoring which I mentioned above) or to keep seek intervals sane (people usually like < 10 second skips so they set keyint=10*fps).

Ahh, thnx, i see now.

I also see that high resolutions need leyint=fps ( 720 or just 1080 ? )

Most of the resolutions people use when encoding with my exe are under 1280x800 ( 800p is becoming standard in russia and other parts ) so 720p or less is most used

Will compressing suffer ( as stated in the wiki ) if i use low keyint for blu-ray compatibility and open gop ? ( from my first test with the only build average qp is higher by 0.20 at 1184x666 )

My settings :

--level 4.0 --sar 1:1 --aud --nal-hrd --vbv-bufsize 24000 --vbv-maxrate 24000 --ref 4 --mixed-refs --bframes 4 --b-adapt 2 --weightb --direct auto --subme 9 --trellis 2 --psy-rd 1.0:0.0 --partitions all --8x8dct --me tesa --merange 32 --fullrange on --qcomp 1 --threads auto --thread-input --aq-mode 1 --aq-strength 1 --progress --no-psnr --no-ssim --no-fast-pskip --no-dct-decimate

They may seem placebo but most of the movies encoded are in the 1-5 minute interval, 10% are in the 5-10 minutes, 1% are more than 10 minutes.

OK, let's make it simple for you. When encoding for Blu-ray/AVCHD then --keyint fps (rounded to the nearest integer) --min-keyint 1 is correct.

And other then that; Forget the specifications of the source. They've got nothing to do with it. Open your source using AVISynth, and you're all set.

The only thing the source provides x264 is .... a sequence of frames.

Yes, now i understand, as i asked nm "Will compressing suffer ( as stated in the wiki ) if i use low keyint for blu-ray compatibility and open gop ? ( from my first test with the only build average qp is higher by 0.20 at 1184x666 )"

thnx.

[Audionut]

if i use low keyint for blu-ray compatibility

Unless you're authoring blu-ray's, don't even bother with "blu-ray compatibility".

Keyint has nothing to do with resolution.

Keyint involves I-frame placement. As stated before, the defaults are used for a purpose.

min-keyint = fps of video. You generally do not want more than one I-frame in a second of video.

max-keyint = 10 x fps of video. This is based on tolerance. You could gain some compressibility by having a max-keyint of 100 x fps of video. But you wouldn't then be able to seek to any of those frames.

ie: 24fps video. max-keyint = 10 x 24 = 240. You can only seek to the first and last frames of those 240 frames. You cannot seek to any frames in between.

The best way to understand is to do some test encodes for yourself.

Use the same settings for both encodes. Set min-keyint to 25. Set scene change sensitivity to 0.
For one encode use max-keyint 250. For the other use max-keyint 2500.
Now playback both encodes and try to seek in each.


edit: and imo this discussion is way off topic.

[G_M_C]

Unless you're authoring blu-ray's, don't even bother with "blu-ray compatibility".

Keyint has nothing to do with resolution.

Keyint involves I-frame placement. As stated before, the defaults are used for a purpose.

min-keyint = fps of video. You generally do not want more than one I-frame in a second of video.

max-keyint = 10 x fps of video. This is based on tolerance. You could gain some compressibility by having a max-keyint of 100 x fps of video. But you wouldn't then be able to seek to any of those frames.

ie: 24fps video. max-keyint = 10 x 24 = 240. You can only seek to the first and last frames of those 240 frames. You cannot seek to any frames in between.

That wont give a blu-ray compliant stream

Blu-ray's have strict specifications;

Resolutions are fixed at 1920x1080, 1440x1080 (anamorphic) or 1280 x 720. All resolutions can be interlaced or not.

Framerates are guided by rules too 1080 resolutions can be either 23,976 ("24p") or 25 for progressive video or 29,97 ("60i") for interlaced & pull-down or 50i. 720p can have same rates, but also has 60p.

Encoding for blu-ray has very strict rules. The minimum keyframe rate is defined in the specification as 1. Meaning it is simply always one, and not equal to the fps as you say.

The max keyint is also defined through the max GOP length.

The max GOP length is defined as 1 second long for streams with a bitrate > 15mbps (or 2 seconds long for streams < 15 mpbs rate). Since any good blu-ray encode has a bitrates > 15 mpbs, at least in peaks, it comes down to the fact that the max key-int is always equal to the framerate (rounded to the nearest integer).

GOP==1 sec ==>
place a keyframe every 1 second ==>
--max-keyint set as fps.

[juGGaKNot]

edit: and imo this discussion is way off topic.

The end result of my questions would be uncompressed > blu-ray using open gop so i do not think i am off-topic but you are the moderator and it is your decision.

That wont give a blu-ray compliant stream

Blu-ray's have strict specifications;

Resolutions are fixed at 1920x1080, 1440x1080 (anamorphic) or 1280 x 720. All resolutions can be interlaced or not.

Framerates are guided by rules too 1080 resolutions can be either 23,976 ("24p") or 25 for progressive video or 29,97 ("60i") for interlaced & pull-down or 50i. 720p can have same rates, but also has 60p.

Encoding for blu-ray has very strict rules. The minimum keyframe rate is defined in the specification as 1. Meaning it is simply always one, and not equal to the fps as you say.

The max keyint is also defined through the max GOP length.

The max GOP length is defined as 1 second long for streams with a bitrate > 15mbps (or 2 seconds long for streams < 15 mpbs rate). Since any good blu-ray encode has a bitrates > 15 mpbs, at least in peaks, it comes down to the fact that the max key-int is always equal to the framerate (rounded to the nearest integer).

GOP==1 sec ==>
place a keyframe every 1 second ==>
--max-keyint set as fps.

Great, just what i needed.

With this i think i am done so no more off-topic.

Waiting for the next patch.

[Audionut]

That wont give a blu-ray compliant stream

Thanks Captain Obvious.


should i use --keyint 500 ? seek time is not that important.

uncompressed, open gop, --open-gop --keyint 50 --min-keyint 1 is optimal for me no ?

K, blu-ray has 1-2 second gops, my uncompressed avi ( made from picture streams ) has every frame as a keyframe so the gops are at every frame right ?

so --keyint fps --min-keyint 1 for me ?

Unless you're authoring blu-ray's, don't even bother with "blu-ray compatibility".

Personally I think you guys have talked him into thinking he needs blu-ray compatibility. Which will increase bitrate/lower quality, than if you just advised to use defaults!!!!!!!!!!!!

[juGGaKNot]

Personally I think you guys have talked him into thinking he needs blu-ray compatibility. Which will increase bitrate/lower quality, than if you just advised to use defaults!!!!!!!!!!!!

:D

I want to add a blu-ray output choice ( encode to MP4 or blu-ray[y/n] ) to the normal encode exe ( using autocrop for the resolution ) and i just wanted to know the right settings.

I use the defaults for the "non blu-ray" one.

[Trahald]

I Think he's got it now. And please gentlemen, lets keep this civil.

BTW. this thread can slide (somewhat) into bluray talk as that will be the majority of the times to use short-gop&&open-gop.

[rack04]

BTW. this thread can slide (somewhat) into bluray talk as that will be the majority of the times to use short-gop&&open-gop.

How does short-gop and open-gop benefit Blu-ray backups and playback on standalone Blu-ray players? What is the benefit of this patch for Blu-ray encoding?

[nurbs]

Read the first post. :cool:

[rack04]

Read the first post. :cool:

All I see is data supporting smaller filesizes. How will this benefit me if I'm selecting a bitrate to reach a specific filesize?

[nurbs]

Better quality at the same filesize when using short gops.

[Trahald]

Ok.. here is version 5. its redone. i cut and pasted the sei and command line stuff and recoded the rest for the most part. i set tabs to spaces in notepad++ so i didnt have to change my editing style to get good resulting diff. (why didnt anyone tell me about this ;) ) .

the patch now doesnt lie to ratecontrol, introduces MMCO to x264 to deal with p references to b frames that wouldnt be in the buffer during random access, code is cleaner.

there are a couple things i want to do with it but its nearly where i want it.
here is a diff if anyone wants to play in the interim. it will patch to latest x264. I still consider this experimental so please dont compile into anything that will get automatically downloaded.

[MatMaul]

the patch now doesnt lie to ratecontrol, introduces MMCO to x264 to deal with p references to b frames that wouldnt be in the buffer during random access, code is cleaner.

I don't understand what is MMCO but DS said it is needed in order to fix the b-pyramid/DPB problem. Does your patch can help in any way to fix this issue ?

[Trahald]

I don't understand what is MMCO but DS said it is needed in order to fix the b-pyramid/DPB problem. Does your patch can help in any way to fix this issue ?

Uhm.. I dont know what that issue is. my patch only uses mmco to protect from corrupted output in a random seek. In the context i use mmco it really is a small thing (only 1 type of command is used). mmco can be more complicated. i guess id need to hear what the problem is. If its a similar issue (b-ref needing to get killed from the dpb but maybe more often) then it would be trivial to add.

[MatMaul]

some technical infos :
http://forum.doom9.org/showthread.php?t=140223
http://forum.doom9.org/showthread.php?t=142758

[Dark Shikari]

the patch now doesnt lie to ratecontrol, introduces MMCO to x264 to deal with p references to b frames that wouldnt be in the buffer during random access, code is cleaner.I'll check the rest of your patch when I get to work today, but have you tried using MMCO simply to throw away all B-references after coding them so they don't end up in the reference list?

I suspect this will improve encoding quality.

[Trahald]

I'll check the rest of your patch when I get to work today, but have you tried using MMCO simply to throw away all B-references after coding them so they don't end up in the reference list?

I suspect this will improve encoding quality.

i can. just remove the if( opengop && first p after i) part allowing it to run after every p frame (checking to see if there are any bref in the last frame and then removing them all.) if i am understanding you correctly that would be the end of p-frames referring to b-refs. only b frames would refer to them.

And just to clarify (to anyone that cant understand the code). what it currently does is mark any b-refs in the dpb (after the i-frame) as unused for reference. This is because the patch removes the bref from the x264's reference list but the decoder would never know without mmco. That way the decoder doesnt push the i-frame out for a b-ref that wont be referred to in any future frames.

[Dark Shikari]

i can. just remove the if( opengop && first p after i) part allowing it to run after every p frame (checking to see if there are any bref in the last frame and then removing them all.) if i am understanding you correctly that would be the end of p-frames referring to b-refs. only b frames would refer to them.

And just to clarify (to anyone that cant understand the code). what it currently does is mark any b-refs in the dpb (after the i-frame) as unused for reference. This is because the patch removes the bref from the x264's reference list but the decoder would never know without mmco. That way the decoder doesnt push the i-frame out for a b-ref that wont be referred to in any future frames.Can you get on #x264dev IRC on Freenode so we can do a live patch review?

[Dark Shikari]

OK, it doesn't seem to be working at all.

With --b-adapt 0 --bframes 3 --keyint 12 --open-gop, it gives the following frame pattern:

IBBBPBBBPBBBPBBPI

Additionally, with the following qpfile:

0 I -1
1 b -1
2 b -1
3 b -1
4 i -1
5 b -1
6 b -1
7 b -1
8 i -1
9 b -1
10 b -1
11 b -1
12 i -1
13 b -1
14 b -1
15 b -1
16 i -1
17 b -1
18 b -1
19 b -1
20 i -1
21 b -1
22 b -1
23 b -1
24 i -1
25 b -1
26 b -1
27 b -1
28 i -1
29 b -1
30 b -1

and --keyint 4 --bframes 3 --b-adapt 0 --frames 30, every other pair of B-frames is corrupt.

[Trahald]

I really didnt throw much at it yet. And i tried to make it clear, its not ready for a final review. (or even a preliminary one i guess ;) ) There are some things i knew had to be fixed (relatively simple stuff like second i's coming before p's) but once i figured out some of the harder stuff i get paranoid I'll get hit by a bus before i posted the work i had so far. When its closer to being really done, I'll ping you on irc in a day or 2 for a review.

[Trahald]

Here is another one. I ran many more (short) scenarios at it. Heading off to work. will visit irc tonight or maybe in the am. again. really i'd like to see it get tested real world a bit before even considering it for commit. (coding faux pas *sp*) aside.

[Dark Shikari]

Looking good:

$ for x in "x264 --open-gop" x264_old;do ./$x --bframes 3 --keyint 12 --b-adapt 0 --qp 25 SOCCER_352x288_30_orig_02.yuv -o NUL --tune psnr --psnr;done
x264 [info]: 352x288 (given by file name) @ 25.00 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 1.3
x264 [info]: slice I:26 Avg QP:22.00 size: 17182 PSNR Mean Y:41.97 U:46.06 V:47.76 Avg:43.06 Global:42.99
x264 [info]: slice P:50 Avg QP:25.00 size: 6702 PSNR Mean Y:39.47 U:44.53 V:46.42 Avg:40.69 Global:40.57
x264 [info]: slice B:224 Avg QP:27.00 size: 2624 PSNR Mean Y:38.85 U:44.04 V:45.91 Avg:40.09 Global:40.00
x264 [info]: consecutive B-frames: 0.0% 0.0% 0.0% 100.0%
x264 [info]: mb I I16..4: 6.6% 72.2% 21.2%
x264 [info]: mb P I16..4: 3.7% 19.8% 10.5% P16..4: 25.0% 11.5% 12.2% 0.0% 0 .0% skip:17.3%
x264 [info]: mb B I16..4: 0.4% 1.7% 2.1% B16..8: 31.7% 7.9% 9.2% direct: 4.1% skip:43.0% L0:37.3% L1:41.1% BI:21.5%
x264 [info]: 8x8 transform intra:62.2% inter:52.1%
x264 [info]: coded y,uvDC,uvAC intra:94.2% 65.8% 48.0% inter:21.1% 6.5% 1.5%
x264 [info]: ref P L0 82.3% 17.7%
x264 [info]: ref B L0 93.6% 6.4%
x264 [info]: PSNR Mean Y:39.223 U:44.299 V:46.156 Avg:40.444 Global:40.286 kb/s: 913.00

encoded 300 frames, 59.27 fps, 914.11 kb/s
x264 [info]: 352x288 (given by file name) @ 25.00 fps
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 1.3
x264 [info]: slice I:25 Avg QP:22.00 size: 17215 PSNR Mean Y:41.99 U:46.05 V:47.74 Avg:43.07 Global:43.00
x264 [info]: slice P:75 Avg QP:25.00 size: 6515 PSNR Mean Y:39.37 U:44.44 V:46.35 Avg:40.59 Global:40.48
x264 [info]: slice B:200 Avg QP:27.00 size: 2576 PSNR Mean Y:38.65 U:43.91 V:45.80 Avg:39.89 Global:39.81
x264 [info]: consecutive B-frames: 0.0% 0.0% 27.3% 72.7%
x264 [info]: mb I I16..4: 6.2% 72.8% 21.1%
x264 [info]: mb P I16..4: 3.3% 18.3% 9.6% P16..4: 28.3% 12.0% 13.1% 0.0% 0 .0% skip:15.4%
x264 [info]: mb B I16..4: 0.3% 1.6% 1.9% B16..8: 32.4% 7.7% 9.0% direct: 4.0% skip:43.1% L0:39.6% L1:39.2% BI:21.2%
x264 [info]: 8x8 transform intra:62.6% inter:53.6%
x264 [info]: coded y,uvDC,uvAC intra:94.1% 65.3% 47.7% inter:22.9% 7.3% 1.7%
x264 [info]: ref P L0 77.2% 14.1% 8.7%
x264 [info]: ref B L0 94.1% 5.9%
x264 [info]: PSNR Mean Y:39.108 U:44.225 V:46.099 Avg:40.333 Global:40.165 kb/s: 956.07

encoded 300 frames, 59.44 fps, 957.00 kb/s

[Trahald]

Here is the next revision. Adds code to tighten up max frame_num/lsb calc DS wrote. addressed mp4/mkv as output formats (flagging keyframes) when opengop is used. Since the formats dont really support RP (well i should say players that play the files ) you will get a couple of corrupted frames when you seek. this is not a bug.
Now an issue.. i get issues with mpchc w/dxva. powerdvd using hardware acceleration works fine, and every other player i tried the streams on, including my xbox 360. all worked ok. Im not sure if its a bug in mpchc or the patch. please test on standalones too.

[Dark Shikari]

p_mp4->p_sample->IsRAP = p_picture->i_type == X264_TYPE_IDR
+ || ( p_picture->i_type == X264_TYPE_I && p_mp4->b_open_gop ) ? 1 : 0;This is broken.

Not all I-frames are seekable, even with opengop. You need to add an extra entry to x264_picture_t named "b_keyframe" and use that instead, as I told you on IRC.

It should be:p_mp4->p_sample->IsRAP = p_picture->b_keyframe;

Also, if your patch does break the ability to place non-key I-frames, it will be instantly rejected.

[Old Timer]

Just an observation. With the following parameters in the command line: --keyint 24 --min-keyint 2 I can see in the resulting stats files that an i and an I frames can still be placed next to each other. Perhaps, the definition of min-keyint could be reviewed? Thanks.

Edit: added some stats data:

in:310 out:310 type:P q:25.00 tex:0 mv:0 misc:488 imb:0 pmb:0 smb:8160 d:s;
in:311 out:311 type:P q:24.99 tex:0 mv:0 misc:488 imb:0 pmb:0 smb:8160 d:s;
in:312 out:312 type:i q:20.00 tex:74105 mv:20571 misc:732 imb:8160 pmb:0 smb:0 d:s;
in:313 out:313 type:I q:16.00 tex:311354 mv:62126 misc:824 imb:8160 pmb:0 smb:0 d:s;
in:314 out:314 type:P q:20.98 tex:439703 mv:72472 misc:2473 imb:5247 pmb:724 smb:2189 d:s;
in:315 out:315 type:P q:16.98 tex:1131392 mv:106473 misc:663 imb:7887 pmb:273 smb:0 d:s;

[Trahald]

As default x264 goes, a small i is just another frame. Only IDRS are considered key frames. The patch, keeping with that, considers only IDRS and RPs key frames. So you might get IDR i i(rp) i i(rp) with the patch. (recovery points only being written on the 3rd and 5th frame.) Muxers should only factor in the i(RP) that way also, well muxers that even bother to check for recovery points.

[Old Timer]

This seemed to be an intersting feature, just curious - what happened to it? Has it ever been commited into the main dev branch? Thanks.

[Trahald]

I have a patch that works with 1198. but i think something new broke it again (. once everything mbtree wise settles down, weight-p is done, open gop will go into testing / DS scrutiny soon. i'll post binaries here. When all is well hopefully its commited. Like all my patches i make stuff for myself that i want, but i like to share :).

[moviefan]

Will open-gop be Blu-ray compliant?

[Trahald]

Open GOP is allowed in bluray.

[moviefan]

Very nice!

[Old Timer]

Thanks Trahald. I have been using the build you provided in this post (http://forum.doom9.org/showthread.php?p=1299552#post1299552) to test qute a few encodes on some hardware (blu-ray player, networked media-tank, DXVA capable video-card) and they all look great. Again, thanks for the nice feature.

[klinika]

This is a patch I've been waiting for too. Can't wait to finally have some medication for the keyframe pulsing that's been making bluray compatible SD backups not very attractive. A huge thanks Trahald :)

[Trahald]

Here is a test version of the open gop patch. Its compiled with nal-hrd (with small adjustments for open gop.) i just want it to get a quick public test before i represent it to #x264dev.

x264_x86_r1373_gcc_wewk.zip (http://www.mediafire.com/?ywnzjnkz2wn)
built on Dec 12 2009, gcc: 4.4.2 (x86.generic.Komisar)

patches used
x264_hrd_pd_interlace.16_r1369.diff
x264_open_gop_11.diff

[Daemon404]

Here is a test version of the open gop patch. Its compiled with nal-hrd (with small adjustments for open gop.) i just want it to get a quick public test before i represent it to #x264dev.

x264_x86_r1373_gcc_wewk.zip (http://www.mediafire.com/?ywnzjnkz2wn)
built on Dec 12 2009, gcc: 4.4.2 (x86.generic.Komisar)

patches used
x264_hrd_pd_interlace.16_r1369.diff
x264_open_gop_11.diff

In case you missed it:

[17:34] < Daemon404> wewk: when you get back, ive foudn a bug in opengop. opengop + bframes > 0 + flv = x264 doesnt write it's header (the one with all the used params)
[17:38] < Daemon404> (apparently if the nal-hrd patch is also applied this goes away?)

[shon3i]

@Trahald, thanks, testing right now :) I hope we finaly get this into git :)

[moviefan]

Two questions:
1) What is this issue with Blu-rays and keyframe pulsing? Can someone explain this a little more in detail please?
2) Concerning Daemon404's quote: Does this issue only apply to flv encodings? If I have an AVS-script as input and output raw .264, are there any known issues or should it work?

[shon3i]

something is wrong here, MUI Generator reject stream with OpenGOP's. without specific error, just "ERROR"

[moviefan]

And what does Scenarist say?

[shon3i]

And what does Scenarist say?
"ERROR" and then MUI Generation End. as usual when stream is broken/out-of-specs

cmd i used.

"x264.exe" --tune film --bitrate 5000 --pass 2 --stats x264.stats --level 4.1 --slices 4 --vbv-bufsize 30000 --vbv-maxrate 30000 --keyint 24 --min-keyint 2 --aud --nal-hrd --open-gop --output rundown1.264 rd.avs

[moviefan]

That's really ashame... I hope it's only a minor bug. By the way: The build you probably used (the one posted in http://forum.doom9.org/showthread.php?p=1352134#post1352134) is patched with x264_hrd_pd_interlace.16_r1369.diff, not x264_hrd_pd_interlace.16_r1369_fix.diff (fixed by komisar, http://doom10.org/index.php?PHPSESSID=g4jadmsigauq1t7e08j89k4od1&topic=7.msg344#msg344). Could this be a problem?

[shon3i]

I think not becuase i encode without --open-gop with same build, and passes fine.

[moviefan]

Okay, in that case that's probably not the problem. By the way: patching x264_hrd_pd_interlace.16_r1369_fix.diff and x264_open_gop_11.diff doesn't work for some reason. Whatever patching order I choose, there is always a error. If I patch open-gop first, nal-hrd fails at hunk #8 (at 2260).

Edit:
There is a conflict with


x264_open_gop_11.diff

+static inline void bs_align_10( bs_t *s )
+{
+ if( s->i_left&7 )
+ bs_write( s, s->i_left&7, 1 << ( (s->i_left&7) - 1 ) );
+}


and


x264_hrd_pd_interlace.16_r1369_fix.diff

+static inline void bs_align_10( bs_t *s )
+{
+ if( s->i_left&7 )
+ bs_write1( s, 1 );
+ bs_flush( s );
+}
+


I just removed the open-gop version, but I don't know if that is correct. At least it compiles... Someone with insight should check it.

Edit #2:
Removing the open-gop version of the code posted above is probably not such a good idea since x264 terminates because of nal-hrd problems. (although I don't know why they occurred since I kept the nal-hrd version of the code above) Now I'm trying to encode with the open-gop code part, but I suspect this won't work either...

Edit #3:
OK, none of this works. Someone has to investigate. If I encode a patched build without --open-gop, things work.

[Sharc]

Two questions:
1) What is this issue with Blu-rays and keyframe pulsing? Can someone explain this a little more in detail please?

Blu-Ray specifies GOPs of no longer than 1 second (you will find more detailed info on this in the forum). The insertion of keyframes every 1 second may produce a 'pulsing' of the picture every second which becomes more visible for low bitrate encodes (strong compression). The pulsing can be somewhat reduced by setting --ipratio 1.1 and --pbratio 1.1 instead of the x264 default values of 1.3 and 1.4 respectively.

Added:
IIRC the --pbratio setting is overruled by mb-tree, means --pbratio is only relevant when mb-tree is disabled.

[moviefan]

OK, and what can open-gop do to improve this pulsing? By the way, I've noticed such pulsing myself because of Blu-ray restrictions, but I wasn't sure if you meant the same.

[Daemon404]

Two questions:
2) Concerning Daemon404's quote: Does this issue only apply to flv encodings? If I have an AVS-script as input and output raw .264, are there any known issues or should it work?

I've only seen it with flv output + bframes > 0 + open-gop.

[Trahald]

"ERROR" and then MUI Generation End. as usual when stream is broken/out-of-specs

cmd i used.

how many frames is it.. can you see the minimum # of frames it still fails? i havent been able to duplicate but then ive only used a 1000 frames.

[shon3i]

about 5000 frames is long clip, when i press "Create Files" button in MUI Generator i automaticly get error.

I will do test again with oter clip, to be sure.

[Trahald]

In case you missed it:

[17:34] < Daemon404> wewk: when you get back, ive foudn a bug in opengop. opengop + bframes > 0 + flv = x264 doesnt write it's header (the one with all the used params)
[17:38] < Daemon404> (apparently if the nal-hrd patch is also applied this goes away?)

the flv writer only consider the last sei in an aud (since vanilla x264 only writes one sei that works fine). Hrd adds an sei and open gop adds another one. so only the recovery point is getting written. anyways.. i'll fix it. should have a binary tomorrow or tues.

[Sharc]

OK, and what can open-gop do to improve this pulsing?
No idea. Perhaps someone else may explain?

[moviefan]

Is there a chance that Scenarist is simply wrong with its error about the open-gop stream? DXVA works fine with hrd and open-gop. Of course Blu-ray specs are more restrictive, but still...

[shon3i]

Is there a chance that Scenarist is simply wrong with its error about the open-gop stream? DXVA works fine with hrd and open-gop. Of course Blu-ray specs are more restrictive, but still...
I dont think so, i aslo test with Ateme Kyrion File Encoder (based on encavc 2 engine) which have OpenGOP feature (ateme always had opengop), scenarist passes stream fine, and report in stream properties that open gop is used

[Trahald]

http://www.mediafire.com/?toqfuqwgmzy
gcc: 4.4.2 (x86.generic.Komisar) fprofiled

patches used
x264_hrd_pd_interlace.16_r1369_fix.diff
x264_open_gop_11.diff
x264_adjust_cpb.diff (http://pastebin.com/m55643cd2)

the last patch fixes the flv issue and adjusts hrd for open gop.

This one passes scenarist w/a 5000 frame vid. (although the other worked ok for me to) but try this one.

[shon3i]

Still same :( what version of Sceanarist you have? I have 5.13 which came with MUI Generator version 1.0.3314.20055

EDIT: Without --open-gop switch everything passes, ateme passes. Maybe to send you ateme stream to examine? O somehow to help you?

[Trahald]

I found it.. missing a rbsp_trailing_bits( ) call after the recovery point sei.. will have a binary in a bit

[Trahald]

x264_x86_r1373_gcc_wewk_3.zip (http://www.mediafire.com/?vzchumfvmat)
gcc: 4.4.2 (x86.generic.Komisar) fprofiled

patches used
x264_hrd_pd_interlace.16_r1369_fix.diff
x264_open_gop_11.diff
x264_adjust_cpb_2.diff (http://pastebin.com/m3e630052)

[rack04]

Using this build:

x264_x86_r1373M (http://www.mediafire.com/?mltilymzyij)

Toolchain:
mingwrt-3.16
w32api-3.13
binutils-2.20
gcc-core-4.4.2
zlib-1.2.3
yasm-0.8.0
Patch:
x264_hrd_pd_interlace.16_r1369_fix.diff (http://komisar.gin.by/x.patch/last.used/x264_hrd_pd_interlace.16_r1369.diff)
x264_open_gop_11.diff (http://forum.doom9.org/showthread.php?p=1352134#post1352134)
x264_adjust_cpb_2.diff (http://pastebin.com/m3e630052)
Build:
./configure --extra-cflags="-march=core2"
make fprofiled VIDS="/c/x264/crew_4cif.y4m /c/x264/parkrun.1280x720.yuv"

I created the following sample. I would appreciate if someone with Sceanarist could verify whether or not this stream is Blu-ray compliant.

http://www.mediafire.com/?ylwgwxmzjgx

The command line used to generate the sample:

"C:\Program Files\x264\x264.exe" --profile high --preset placebo --tune animation --crf 18 --level 4.1 --ref 4
--bframes 3 --keyint 24 --min-keyint 1 --vbv-bufsize 30000 --vbv-maxrate 24000 --slices 4 --b-pyramid strict
--sar 1:1 --aud --nal-hrd --open-gop --output "C:\Personal\Videos\hd_dolby_catalyst_lossless-output.mkv"
"C:\Personal\Videos\hd_dolby_catalyst_lossless.avs"

[Trahald]

can you encode it to .264 (raw)? mkv extract removes the aud's (or maybe the muxer... idk) but either way they arent there after demux.

[shon3i]

@Trahald it's work now :) weee, passes everything.

@rack04 do a .264 i will test for you. Anyway be cearful with --vbv-bufsize 30000 --vbv-maxrate 24000 because lower maxrate than buffer in x264 produce vbv-init higher than 1sec which can produce not smooth playback even no under/overflows in stream.

[moviefan]

@rack04 do a .264 i will test for you. Anyway be cearful with --vbv-bufsize 30000 --vbv-maxrate 24000 because lower maxrate than buffer in x264 produce vbv-init higher than 1sec which can produce not smooth playback even no under/overflows in stream.

Does Scenarist complain in that case or do you only notice at playback?

[shon3i]

Scenarist won't mux that stream, it will detect as "possible buffer underflow" and will stop muxing. I can't see real happening with tsmuxer because i have completly menu structure in scenarist.

I don't know is problem in nal hrd or vbv itself, but with other encoders CPB init delay work totaly different.

[moviefan]

So would --vbv-bufsize 25000 --vbv-maxrate 25000 (which is the maxrate for --level 4.0) be safe? I intend to have at maximum two DTS 1536kbps audio streams (if that matters).

[shon3i]

which is the maxrate for --level 4.0maxrate for L4.0 is 24000. so use --vbv-bufsize 24000 --vbv-maxrate 24000, just dont use buffer greather than maxrate. With DTS 1.5 should be fine. Your cmd upper is just fine and it's L4.1 you should only change --vbv-maxrate 24000 to 30000

[moviefan]

Strange since x264 gives a warning if --level 4.0 --vbv-maxrate > 25000 that it should be no more than 25000... That's why I wrote 25000.

[shon3i]

Strange since x264 gives a warning if --level 4.0 --vbv-maxrate > 25000 that it should be no more than 25000... That's why I wrote 25000.
Because x264 follow H264 specification for specified level, and Blu-Ray have different settings.

[rack04]

can you encode it to .264 (raw)? mkv extract removes the aud's (or maybe the muxer... idk) but either way they arent there after demux.

@rack04 do a .264 i will test for you.

Here you go. Thanks.

http://www.mediafire.com/?mmmu4mkvlml

EDIT: Patching the latest git with x264_open_gop_11.diff fails.

[moviefan]

Are there any solutions concerning open-gop + qpfile yet?

[shon3i]

Are there any solutions concerning open-gop + qpfile yet?
what is wrong with this?

[moviefan]

I noticed a crash on Wednesday when using a qpfile to insert IDR keyframes for chapter marks and it turned out on IRC that Trahald had not considered testing on qpfiles. He said he was gonna fix it soon.

[Dark Shikari]

We're slightly overhauling this case; keyframes will now be "K" in the QPfile, and will be handled differently inside of x264. "K" means IDR as usual if opengop is off, and I + recovery point if opengop is on. You'll still be able to force IDR frames, too.

[moviefan]

So I need to replace the -I by -K in the qpfile now, if I want to prepare for chapter marks and encode with open-gop?

[Dark Shikari]

So I need to replace the -I by -K in the qpfile now, if I want to prepare for chapter marks and encode with open-gop?No. IDR frames still work just fine; the point is that a user might want to be able to specify either.

[moviefan]

Hm, but you remember the issue with "cost >= 0" on Wednesday that I pointed out when I was trying to encode with r1376 + hrd + open-gop? Does open-gop need a fix or do I have to change something in the qpfile? (a little confused now)

[Dark Shikari]

Hm, but you remember the issue with "cost >= 0" on Wednesday that I pointed out when I was trying to encode with r1376 + hrd + open-gop? Does open-gop need a fix or do I have to change something in the qpfile? (a little confused now)As I already said quite a few times, that was a problem with OpenGOP: it didn't support forced IDR frames correctly.

[Selur]

Just wondering: Any news on the open-gop patch?

[Daemon404]

Just wondering: Any news on the open-gop patch?

Indeed I would also like to know this.

[kieranrk]

Some tweaks might be necessary to get it to work properly with the HRD patch. Trahald has been busy recently and so hasn't had time to look at it.

[Selur]

Thanks for the info. :)

[Trahald]

busy == lazy/health issues.. im attaching a patch to test. I patched the last rev. by hand to 1510, added a couple lines to address intra-refresh combination, and generated this patch. i really didnt test it. i made a file and it plays and the hrd info looks right (kieranrk will need to look at it).

[kieranrk]

The HRD info looks right to me as well. I'd need to test it some more but it looks fine to me. I suppose people with Blu-Ray verifiers might want to test it to be on the safe side as well.

[shon3i]

If compile ... :)

[kieranrk]

http://dl.dropbox.com/u/2701213/x264/open%20gop/x264.exe

[jpsdr]

I assume, reading this, that open-gop is something supposed to be BR compliant ?
What is the benefit of it ?

[nurbs]

It is described in the first post of this thread.

[jpsdr]

Tested with the last rack 04 build, crashed with 'qpfile' (used to define chapters on blu-ray).
The issue is still present.

[Trahald]

can you post your qpfile and command line. i couldnt duplicate the crash here.

[jpsdr]

Here the command line used, x264 crash during the 1rst pass :

@echo off

SET E_SRC=%6%1.avs
SET E_DST=%3%1.264
SET STAT_FILE=%1.stats
SET TUNING=%4
SET LOG_FILE_1=%1_log_1.txt
SET LOG_FILE_2=%1_log_2.txt

REM Set of max bitrate (ici le bitrate max)
set MAX_BR=24000

REM Set of Buffer (ici le buffer)
set BUF_BR=30000

REM 1ère passe
x264_x86.exe --profile "high" --preset "placebo" --tune %TUNING% --pass 1 --bitrate %2 --stats %STAT_FILE% --level "4.0" --qpmin 0 --qpstep 1 --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --keyint 24 --min-keyint 2 --mvrange 511 --ref 6 --bframe 3 --b-pyramid "strict" --open-gop --subme 9 --trellis 1 --me "umh" --aq-mode 2 --aud --nal-hrd "vbr" --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --qpfile %5 --threads 0 --thread-input --output NUL %E_SRC% 2> %LOG_FILE_1%

REM 2ème passe
x264_x86.exe --profile "high" --preset "placebo" --tune %TUNING% --pass 2 --bitrate %2 --stats %STAT_FILE% --level "4.0" --qpmin 0 --qpstep 1 --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --keyint 24 --min-keyint 2 --mvrange 511 --ref 6 --bframe 3 --b-pyramid "strict" --open-gop --aq-mode 2 --aud --nal-hrd "vbr" --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --qpfile %5 --threads 0 --thread-input --output %E_DST% %E_SRC% 2> %LOG_FILE_2%

Bitrate (%2) : 4526
Tuning : animation

Chapter file :

2045 I -1
17192 I -1
32639 I -1
49540 I -1
63245 I -1
79971 I -1
93840 I -1
109392 I -1
124439 I -1
139582 I -1
155038 I -1
156716 I -1
157436 I -1


Partial result log file :

avs [info]: avisynth 2.6+ detected, forcing conversion to YV12
avs [info]: 1280x720p 1:1 @ 24000/1001 fps (cfr)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile High, level 4.0
[20.7%] 32640/157449 frames, 32.70 fps, 4538.87 kb/s, eta 1:03:37 <= Crash here

[Trahald]

Silly error i missed. only came up with opengop/bpyrstrict/qpfile combo.

anywho.. 1 line and its fixed. Please test.

[jpsdr]

As soon as there will be a build.... (rack04 ??? :rolleyes: ), i'll test it.

[JEEB]

Young men, there's no need to feel down♪～

Clicky (http://x264.fushizen.eu/files/opengop/).

Otherwise unpatched, contains lavf/ffmpeg. 64bit one has ffmpeg's SSE optimizations disabled.

[jpsdr]

I don't know what ffmpeg is using for in x264, as i only use avisynth script with avi lagarith YV12 files for input...
Otherwise, thanks, i'll test it this evening (for me).

[julius666]

I don't know what ffmpeg is using for in x264, as i only use avisynth script with avi lagarith YV12 files for input...
Otherwise, thanks, i'll test it this evening (for me).

If you don't use any filter on your input video in avisynth just opening with avisource, then you don't need avisynth. You could directly feed x264 with your source.
That's what ffmpeg is for.

[Midzuki]

I don't know what ffmpeg is using for in x264, as i only use avisynth script with avi lagarith YV12 files for input...
Otherwise, thanks, i'll test it this evening (for me).

If you don't use any filter on your input video in avisynth just opening with avisource, then you don't need avisynth. You could directly feed x264 with your source.
That's what ffmpeg is for.

Well, me neither doesn't use the ffmpeg-input capabilities of the "new generation" of x264 builds, and to me, their greater filesize becomes "bloatware". :devil: This is the reason why I've asked for "7-zipped" builds — believe me, there still exist many many people who DON'T have a fast Internet connection, which means,

in the "dial-up world",
1MB == 5 minutes of download time (typically)

[shon3i]

verifier's does not show any problems with OpenGOP on short sample. I will encode whole movie to recheck.

Anyway, how far this from merging to GIT?

[Trahald]

It was approved by pengvado (about 100 revisions ago pre nal-hrd) so should be close. Ill wait till you do a longer test then submit to 'da man' for approval.

[Atak_Snajpera]

Does this patch solve "frame pulsing problem" with small --keyint (for example 24) and low bitrate?

[Trahald]

Does this patch solve "frame pulsing problem" with small --keyint (for example 24) and low bitrate?

In my (limited) testing, yes.

[jpsdr]

I've tested with .20 patched version, 1rst pass finished without problem. I've stopped the 2nd pass at around 50%, because i needed my PC to do more urgent work, but, in my concern, the problem seems solved.

[jpsdr]

Any patched version with v1570 ?

[Midzuki]

I'd rather wait until the patch has become "official". :)

[Trahald]

@shon3i

where you able to do that test

[shon3i]

I been busy these days, but i am almost finish all task i want to test, i finished with 1080p24 source with qpfile used, and it's pass ok, now i doing same with different source which require Interlaced and Pulldown.

[shon3i]

Ok here is one problem. Now i use "Long" (2 second) GOP because my max bitrate is under 15000kbps i use keyint 48 for 24p source, MUI Generator on end show "The number of frames displayed in GOP exceeds the limit"

[Trahald]

Only with open GOP? I'll look at it when I get home.
Do you have open GOP source not from the patch with 2 second gops that you can check mui generator against?

[shon3i]

Only with open GOP? I'll look at it when I get home. Yes, stream with OpenGOP and 2Sec GOP fail to index into MUIGenerator.

Yes i have stream with 1 Second GOP i will send you both samples to examine, when i get home.

[jpsdr]

I think Trahald is asking for (if i've not misunderstood) :
1) A 2 seconds open-gop source comming from somewhere else than x264+open-gop, and wich is working fine with MUIGenerator.
2) If a 2 second gop source encoded with the same version of x264 but without the open-gop is working fine

[Trahald]

I think Trahald is asking for (if i've not misunderstood) :
1) A 2 seconds open-gop source comming from somewhere else than x264+open-gop, and wich is working fine with MUIGenerator.
2) If a 2 second gop source encoded with the same version of x264 but without the open-gop is working fine
yes.. thank you..

if you can test something like those it would be great. mui generator is known not to be perfect. (not to say its not the patch)

[Trahald]

Welp.. I made a quick sample with closed gop on unpatched x264 (23.976 fps/48 frames a gop) and got the same error on mui generator. oddly, 23.976/47 frames works. Then i tried 24 fps and used gop of 48 frames and it worked. It tells me its not related to the open-gop patch , however its something id like to solve. does anyone have any short h264 encodes (preferably from a bluray but if not just anything other than made by x264) that has 48 frames between gops and is 23.976 fps? it can be opened or closed gop.

[shon3i]

I made a quick sample with closed gop on unpatched x264 (23.976 fps/48 frames a gop) and got the same error on mui generator.Hmm, i never have problem with this, i always use 2 sec GOP for mine BD5-9 backups, with recent unpatched builds from x264.nl. Please test again, even Dark's x264 Demo Blu-Ray use 2 second gop without problem.

I am right now upload sampes to mediafire.

Anyway i can made encode with Ateme encoder.

[shon3i]

Ok, here is archive http://www.mediafire.com/?yd2zqy5zgo2, that contains samples

1 sec closed GOP = PASS
1 sec open GOP = PASS
2 sec closed GOP = PASS
2 sec open GOP = FAIL!

I am currently not able to encode with Ateme, i must finish some work, i will encode tonight.

[Trahald]

Tried again. same error. (used a different file name to make sure i didnt test the wrong file)

BTW the demo disk has no 23.976 fps tracks... any other frame rates are not effected (24 fps is fine as 2*24=48)

[shon3i]

BTW the demo disk has no 23.976 fps tracks... any other frame rates are not effected (24 fps is fine as 2*24=48)
Yes but mine are prue 23.976 or 24000/1001.

Not only me, here many ppl use 2 sec closed GOP fine with scenarist, sony blu-print.

Btw, i use last Rack04 buld, maybe that is different from yours?

[Trahald]

Ugh.. i see.. its because i set 23.976 instead of 24000/1001 in my tests. mui generator couldnt deal with the odd framerate.

OK.. i think i see the issue. I use the actual frame-number to figure out when to put a i-frame. This results in the POC being exactly 48 frames apart but with a variable b (b-adapt > 0) the physical frames can end up more than 48 frames apart. I didnt have any streams that were open gop and variable-b length so I assumed POC counted. (I asked around a bit and the consensus was POC.) I can change it to physical distance.

do you have access to streams that are open gop and variable b? id like to know see what they do.

[Trahald]

This patch accounts for written distance between key frames w/opengop and a variable b . Should pass mui generator. Please test to make sure i didnt break anything.

[moviefan]

I cannot apply the patch to the r1583. "Hunk #2 FAILED at 1226", "1 out of 2 hunks FAILED -- saving rejects to file common/common.c.rej"

Is your correction especially relevant for Blu-ray encodes or was open-gop generally broken (so also for software decoders)?

[Trahald]

its really only an issue if you do 48 frame encodes and even then probably not. (it could effect 24 frame encodes but chances are the local bitrate was under the 2 second threshhold and you were safe) Having said that, the hard limits are enforced by muxers. it is for assumed real limitations in hardware , but usually the playback hardware is more resilient. This would not effect software at all.

remember any encodes made with the patch before its fully tested are experimental.

[Trahald]

I hand patched 1583 and rediffed it for you. (i was working from 1564)

[shon3i]

Ok, i am now wait for build. Btw, tomorow i will send you samples, i get all required things.

[jpsdr]

Tested on a 158000 frames video, 720px23.967fps with keyint of 48, pass Megui Ok. Used the last x64 rack04 build.

[Blue_MiSfit]

What kind of subjective quality or metric improvements does Open-GOP give in typical 1-2 second GOPs?

I capture with a 1 second GOP, and some of my clients need 2 second GOPs for deliverables. Oh, and there's that whole BluRay thing :D! I'm very excited by this patch polishing up!

Thanks guys
~MiSfit

[Trahald]

I really havent done alot of testing. As i went along i did some metric tests (read that tossed in --ssim and tested against a vanilla x264 w/closed gop) and generally open gop scored better than closed. A couple of visual tests showed reduction in keyframe pulsing. Once its done I may do some more tests but so far Im happy with the results. Anyone is free to make some metrics now of course but the final patch may change again.

[Atak_Snajpera]

Is this normal that seeking on PC is alot slower than with closed gop keint=fps*10 ?

[neuron2]

Is this normal that seeking on PC is alot slower than with closed gop keint=fps*10 ? It depends on how the seeking is implemented. For example, if you want to be able to seek to the leading B frames, then you have to go back one GOP to start decoding and with that keyint at 30fps, you have to decode 300 extra frames to get started. On the other hand, if the seek skips leading B frames then there is no reason it should be slower.

[Trahald]

In addition... Some playback software and muxing software will need to adapt to open GOP. The program streams x264 creates muxes the iframes as keyframes but other software may not acknowlege it. The stream will appear as one large GOP. This shouldnt happen with ts or m2ts (bluray) since open GOP isn't unusual there but some mp4 and mkv muxing software isn't use to it.

[moviefan]

If I have muxed open-gop x264 streams already to mkv, will the raw stream - if I demux again to raw - broken so that I cannot remux to an updated mkvmerge version that can handle open-gop correctly? The demuxed raw stream is not bit-identical to the originally encoded raw stream for Blu-ray compliant streams (that's the reason why I'm asking).

[Trahald]

v26 some small changes, mostly cosmetic.

[jpsdr]

Bad news unfortunately.
I've encoded 12 files 720px23.976fps with the following command lines :

@echo off

SET E_SRC=%6%1.avs
SET E_DST=%3%1.264
SET STAT_FILE=%1.stats
SET TUNING=%4
SET LOG_FILE_1=%1_log_1.txt
SET LOG_FILE_2=%1_log_2.txt

REM Set of max bitrate (ici le bitrate max)
set MAX_BR=15000

REM Set of Buffer (ici le buffer)
set BUF_BR=30000

REM 1ère passe
x264_x64.exe --profile "high" --preset "placebo" --tune %TUNING% --pass 1 --bitrate %2 --stats %STAT_FILE% --level "4.0" --qpmin 0 --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --keyint 48 --min-keyint 2 --mvrange 511 --ref 6 --bframe 3 --b-pyramid "strict" --open-gop --subme 9 --trellis 1 --me "umh" --aq-mode 2 --aud --nal-hrd "vbr" --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --videoformat "ntsc" --sar 1:1 --qpfile %5 --threads 0 --thread-input --output NUL %E_SRC% 2> %LOG_FILE_1%

REM 2ème passe
x264_x64.exe --profile "high" --preset "placebo" --tune %TUNING% --pass 2 --bitrate %2 --stats %STAT_FILE% --level "4.0" --qpmin 0 --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --keyint 48 --min-keyint 2 --mvrange 511 --ref 6 --bframe 3 --b-pyramid "strict" --open-gop --aq-mode 2 --aud --nal-hrd "vbr" --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --videoformat "ntsc" --sar 1:1 --qpfile %5 --threads 0 --thread-input --output %E_DST% %E_SRC% 2> %LOG_FILE_2%


Note : Bitrate was 4529.

The 7 first were done with the 1629M Rack 04 version v22 open gop.
The 5 last were done with the 1629M Rack 04 version v26 open gop.
For the 7 first :
4 are Ok with Scenarist MUI Gen.
3 have "Error : The number of fields (frames) displayed in a GOP exceeds the limit".
For the 5 last :
2 are Ok.
3 display an error.

[shon3i]

@jpsdr it's already known, Trahald make possible solution but isn't work every time.

problem is in combination of keyint 48 and openGOP.

The demuxed raw stream is not bit-identical to the originally encoded raw stream for Blu-ray compliant streams (that's the reason why I'm asking). And you will never get same stream, because calcualtions of SPS, AUD, HRD data depend on container. If you target is Blu-Ray, never pull stream through mkv or mp4, because will destroy or change calculated data.

[jpsdr]

Euh... Why are you talking about mkv/mp4 when i'm doing raw (.264) streams ? Or maybe this part was not an answer for me ?
Wasn't the point of the fix to make it working, or do you imply that it will never be possible to combine opengop and keyint 48 in any encoder (not specialy x264) whatever the solution/algorithm implemented ?
My purpose here is to notify of the problem, but now, if there is no solution...

[shon3i]

Or maybe this part was not an answer for me ?yes this part is for moviefan :) i quote him :)

Wasn't the point of the fix to make it working, or do you imply that it will never be possible to combine opengop and keyint 48 in any encoder (not specialy x264) whatever the solution/algorithm implemented ?It will be possible, but need more time, until that use it with short gop (24)

[jpsdr]

MUI Gen show pop-up GOP error always at 99% or 100% of the progress bar. I don't know if MUI Gen show pop-up always after doing all the file, or when error is found.
To check this, i'l encode a small file (around 1500 frames) with keyint of 100, to see if the progress bar information is relevant to the position of the error. But i'll not be able to test this before several hours. If in the meantime someone (maybe you shon3i ?) can confirm that MUI Gen report the error as soon as it's found, and so the progress bar information is relevant, it could help me.
If the problem occurs in fact on the end of the file, here some information for you Trahald, you could eventualy use to check :
- Maybe there is simply some problem when hitting the end of a file.
- The end of my files are fadding black, with if i remember properly, black going for around 50 frames. So, maybe, when video is all black, and maybe (or not) combined with being at the end of the file, it triggers something.
- I'll test this evening to encode the last 2000 frames of my files, see if problem can be reproducted on small files, wich will allow me to produce sample i'll be able to provide, because providing 4GB files is not something realy easy...

[Trahald]

The current version was updated by kemuri_9 A few days ago (let's call it v27) it is the one being considered for committing. I'll see if I can post the patch but I'm at work. As far as temporary fixes go, you can use something like 48-whatever your bframe setting is. If you have bframes 4 then use 44.

[jpsdr]

Ok.
First, i've been able to reproduce the problem on a small file, but... Problem occurs only when the qpfile is used.
Here you can get sample (both results with and without qpfile) :
http://rapidshare.com/files/396744909/Sample_x264.rar.html
If you can post the patch it will be VERY usefull, as i can very quickly test it.

[Trahald]

Thank you very much for testing. I was able to duplicate the issue as well. And yes the problem is the qpfile. if the requested I-frame placement was just past the end of a GOP it could cause a long gop. This fixes it in my tests.

[shon3i]

So that basicly mean we cover all situations :)

Thanks Trahald.

[jpsdr]

I just get the rack04 builds, results will be in 3-4 days, when encodes will be finished.

[moviefan]

Trahald: Did you change anything between revision 22 and revision 26 that might have fixed the artefacts that randomly appear after seeking, shown in the following picture?

http://a.imagehost.org/t/0654/screenshot.jpg (http://a.imagehost.org/view/0654/screenshot)

When I mux x264 r1629 (+ open-gop r26) streams to mkv with mkvtoolnix 4.0 there are no artefacts after seeking anymore. I use keyint 240, min-keyint 24, preset placebo and a qpfile to insert I-frames at chapter marks. Encoding has always worked fine without any errors or so. Remuxing with the latest mkvmerge hasn't fixed the issue for previously encoded movies, so I wonder if it was you who fixed this and thus whether I have to reencode all "broken" ones or not.

[jpsdr]

The encode of the 4 first of my 12 files with rack04 1629M (og v28) pass MUI Gen Ok.
Still going (with rack04 1643M this time).

[Trahald]

[QUOTE=moviefan;1407214]Trahald: Did you change anything between revision 22 and revision 26 that might have fixed the artefacts that randomly appear after seeking, shown in the following picture?

http://a.imagehost.org/t/0654/screenshot.jpg (http://a.imagehost.org/view/0654/screenshot)

edit -- actually there was a change. intra-refresh uses the code from the earlier open-gop patches but changed a little. I didnt adapt for that fully until rev 23. so thats probably it. its not a problem with the bitstream really , so give me a week. i'll whip up something that will rewrite the recovery points leaving everything else alone so you dont have to reencode.

[moviefan]

If that's not too much of an effort for you, I would really appreciate it! Would you still like to have a sample that contains a chapter mark? If so, how can I you cut the mkv file (or the raw stream) so that the sample starts with an I-frame (I think this is necessary, right?)?

[Trahald]

http://www.mediafire.com/?2b0nn2nztjj is the file and the unoptimized and ugly source http://pastebin.org/325230 (name it blah.c ) if you want to compile it. Usage 'blah raw_src.264 raw_dest.264'

this only works on raw h264 files so demux/repair rp/mux

and.. Nah.. i dont need the clip anymore.. but.. here is a way to cut it.. You demux it to raw h264. Load it into dgavcindex. just load it. select a small viewing range around a problem spot (you use the [ ] brackets to set your start / end). and hit save project and demux video.

[moviefan]

Hm, something is wrong! I still had some of the raw streams encoded with x264 r1602 (open-gop r22) and muxed them with mkvtoolnix 4.0, but it showed artefacts. Also running a demuxed stream through your tool didn't change anything. Do you have any clue what's wrong? I'm currently encoding one of the broken movies with the latest x264 revision patched with the latest open-gop revision (which produced a perfect output last time...). It'll take a few days but I'll report on the output when it's finished.

[rack04]

Hm, something is wrong! I still had some of the raw streams encoded with x264 r1602 (open-gop r22) and muxed them with mkvtoolnix 4.0, but it showed artefacts. Also running a demuxed stream through your tool didn't change anything. Do you have any clue what's wrong? I'm currently encoding one of the broken movies with the latest x264 revision patched with the latest open-gop revision (which produced a perfect output last time...). It'll take a few days but I'll report on the output when it's finished.

Look though the current patches thread and see if the artifact encodes were done with the patched version that I had a typo on the patch.

[jpsdr]

The encode of my 12 videos files of around 160000 frames each finished yesterday, and were all accepted on MUI Gen, and no error during MUX.

[Trahald]

The encode of my 12 videos files of around 160000 frames each finished yesterday, and were all accepted on MUI Gen, and no error during MUX.
Excellent. Thank you for testing.

[moviefan]

Look though the current patches thread and see if the artifact encodes were done with the patched version that I had a typo on the patch.

It might well be the case that I used a patch with a typo. I have successfully encoded a second video with open-gop and muxed it with mkvtoolnix 4.0. There are no seeking issues anymore. Thanks for everyone's support and of course Trahald's great patch (soon official feature :))!

[jpsdr]

!!! Buffer underflow !!!!!!!!!!!!
I'm doomed with this one...
I've been obliged to change the average bitrate from 4529 to 4507 beacause the result was 150MB too big, and now, i've got this...:mad:
Same encoding parameters (except little change in bitrate), same x264 version, why so misery...
I'm realy pissed of...
But, i want to say i'm not angry angainst anyone, all programmers do great job, simply realy begin to... overflow...
This time, the file is "only" 4GB...
Don't know if problem can come from opengop, nal-hrd or anything else.
I'll provide the file, hope it will help. It will be in multi-rar and upload it on rapidshare or megaupload.

The part of the video the problem occurs has a very specific pattern, i'll provide also a small part of the original source.

[shon3i]

Do you get underflow on mux with scenarist/blu-print? or x264 notice you in log?

[mp3dom]

If you have used Scenarist, you can see where the buffer occours simply watching the muxed (incomplete) TS.

[jpsdr]

@shon3i:Underflow with Scenarist.
@mp3dom:That's what i've done, i'll let the PC upload the files on rapidshare during this night, and post the links tomorow morning, for DS can take the file and analyse.
Nevertheless, it occurs on a very specific case in the pictures, but i'll let the experts find out.

[shon3i]

We already talk about this problem, if x264 is not report you in log that underflow is occur, then stream is underflow free, but however there is PTS muxing underflow which can occur (as in you case), and comerical encoders have some tools to avoid them.

[jpsdr]

I don't understand the meaning of your answer. You seem to say (but i may misunderstood) that it's somehow some kind of know problem, and that x246 is not reliable enough to be used with authoring software, and need to use comercial encoder instead.
But isn't the final purpose of x264 to be able to be used for Blu-Ray authoring, and so in final stage to haven't this kind of problem anymore ? And in the process of doing this, report when there is problems, for them to be solved ? (And provide samples when possible and needed).

For the samples... It seems that i unfortunately can't access to rapidshare from work (i clearly understand the reason...), so, i'll not be able to post the links before several hours.

[mp3dom]

Out of curiosity, what's the framerate showed in your x264 log?

[jpsdr]

Command line was :

@echo off

SET E_SRC=%6%1.avs
SET E_DST=%3%1.264
SET STAT_FILE=%1.stats
SET TUNING=%4
SET LOG_FILE_1=%1_log_1.txt
SET LOG_FILE_2=%1_log_2.txt

REM Set of max bitrate (ici le bitrate max)
set MAX_BR=15000

REM Set of Buffer (ici le buffer)
set BUF_BR=30000

REM 1ère passe
x264_x64.exe --profile "high" --preset "placebo" --tune %TUNING% --pass 1 --bitrate %2 --stats %STAT_FILE% --level "4.0" --qpmin 0 --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --keyint 48 --min-keyint 2 --mvrange 511 --ref 6 --bframe 3 --b-pyramid "strict" --open-gop --subme 9 --trellis 1 --me "umh" --aq-mode 2 --aud --nal-hrd "vbr" --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --videoformat "ntsc" --sar 1:1 --qpfile %5 --threads 0 --thread-input --output NUL %E_SRC% 2> %LOG_FILE_1%

REM 2ème passe
x264_x64.exe --profile "high" --preset "placebo" --tune %TUNING% --pass 2 --bitrate %2 --stats %STAT_FILE% --level "4.0" --qpmin 0 --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --keyint 48 --min-keyint 2 --mvrange 511 --ref 6 --bframe 3 --b-pyramid "strict" --open-gop --aq-mode 2 --aud --nal-hrd "vbr" --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --videoformat "ntsc" --sar 1:1 --qpfile %5 --threads 0 --thread-input --output %E_DST% %E_SRC% 2> %LOG_FILE_2%


bitrate : 4507, tuning : animation
video : 720p lagarith YV12

Log of 1rst pass :

avs [info]: 1280x720p 1:1 @ 24000/1001 fps (cfr)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile High, level 4.0
x264 [info]: frame I:4135 Avg QP:12.12 size:110872
x264 [info]: frame P:49770 Avg QP:15.13 size: 44741
x264 [info]: frame B:103559 Avg QP:17.34 size: 9943
x264 [info]: consecutive B-frames: 6.3% 10.1% 15.7% 68.0%
x264 [info]: mb I I16..4: 32.9% 49.4% 17.7%
x264 [info]: mb P I16..4: 7.1% 12.2% 2.9% P16..4: 26.6% 16.2% 5.5% 0.5% 0.6% skip:28.4%
x264 [info]: mb B I16..4: 0.5% 0.9% 0.3% B16..8: 26.2% 6.8% 1.8% direct: 5.8% skip:57.8% L0:34.8% L1:47.5% BI:17.7%
x264 [info]: final ratefactor: 12.94
x264 [info]: 8x8 transform intra:53.2% inter:65.8%
x264 [info]: direct mvs spatial:100.0% temporal:0.0%
x264 [info]: coded y,uvDC,uvAC intra: 62.0% 68.0% 53.0% inter: 23.6% 25.5% 7.4%
x264 [info]: i16 v,h,dc,p: 55% 13% 13% 19%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 16% 13% 17% 8% 8% 9% 8% 10% 11%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 17% 13% 12% 8% 12% 11% 9% 9% 9%
x264 [info]: i8c dc,h,v,p: 47% 21% 21% 12%
x264 [info]: Weighted P-Frames: Y:5.1%
x264 [info]: ref P L0: 65.0% 11.9% 11.4% 5.9% 4.4% 1.2% 0.1%
x264 [info]: ref B L0: 93.6% 3.9% 1.8% 0.8%
x264 [info]: ref B L1: 97.0% 3.0%
x264 [info]: kb/s:4525.15

encoded 157464 frames, 34.64 fps, 4525.15 kb/s


Log of 2nd pass :

avs [info]: 1280x720p 1:1 @ 24000/1001 fps (cfr)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [info]: profile High, level 4.0
x264 [info]: frame I:4135 Avg QP:13.21 size: 96220
x264 [info]: frame P:49770 Avg QP:14.83 size: 46058
x264 [info]: frame B:103559 Avg QP:18.30 size: 9697
x264 [info]: consecutive B-frames: 6.3% 10.1% 15.7% 68.0%
x264 [info]: mb I I16..4: 29.7% 48.3% 22.0%
x264 [info]: mb P I16..4: 5.6% 13.6% 3.3% P16..4: 26.5% 18.5% 4.0% 0.4% 0.3% skip:27.8%
x264 [info]: mb B I16..4: 0.5% 0.8% 0.3% B16..8: 25.8% 7.3% 1.7% direct: 6.6% skip:57.1% L0:35.3% L1:46.7% BI:18.0%
x264 [info]: 8x8 transform intra:56.9% inter:55.0%
x264 [info]: direct mvs spatial:99.5% temporal:0.5%
x264 [info]: coded y,uvDC,uvAC intra: 58.7% 64.4% 49.2% inter: 20.1% 22.0% 8.1%
x264 [info]: i16 v,h,dc,p: 60% 12% 12% 17%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 16% 14% 18% 8% 8% 9% 7% 10% 10%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 17% 12% 17% 8% 11% 10% 8% 8% 8%
x264 [info]: i8c dc,h,v,p: 47% 21% 21% 11%
x264 [info]: Weighted P-Frames: Y:5.4%
x264 [info]: ref P L0: 68.6% 11.2% 11.0% 5.4% 3.0% 0.7% 0.0%
x264 [info]: ref B L0: 93.6% 4.3% 1.6% 0.5%
x264 [info]: ref B L1: 97.2% 2.8%
x264 [info]: kb/s:4500.14

encoded 157464 frames, 10.23 fps, 4500.14 kb/s

[mp3dom]

The curiosity was only on the framerate. Sometimes after IVTC (if someone performed it) the framerate is not perfectly 24000/1001 but 2997/125 which could create some buffer under/overflow. Yours is correctly 24000/1001 so no problem here :)

[jpsdr]

The file is the result of IVTC via avisynth script (doubleweave + pulldown) from a 29.97fps DVD source.
But problem occurs on a very specific scene, were video becomes pure noise, with only black and white points, but not brutaly, "slowly", but brutaly comes back to normal. Problem seems to occurs during the frames were there is this brutal transition of pure black and white noise (so high bandwith i think), to a standard anime scene, were bandwith is probably realy lower.
The thing is that the same video encoded with bitrate at 4529 hasn't trig the error ! It must be realy on the edge.
I hope DS and/or Trahald who worked on the nal_hrd part will be able to get the files i'll post, and find out the source of the problem, and find a solution.

[mp3dom]

Maybe you could lower the bufsize a bit but:
- It's not granted to work
- You need to do an encode which could take up a lot of time (depending by your settings/video length)

It's in these circumstances that segment encoding is the manna from heaven :)

[shon3i]

I don't understand the meaning of your answer. You seem to say (but i may misunderstood) that it's somehow some kind of know problem, and that x264 is not reliable enough to be used with authoring software, and need to use comercial encoder instead.x264 is good enough for authoring apps, but sometimes can produce streams that can cause muxing underflows, while i newer seen that with any comercial encoder before, they have some buffer fullness control, which used for prevention, however comercial encoders recommend to keep VBV buffer size and Maxrate tied as possible (for example --vbv-maxrate 15000 --vbv-bufsize 15000), to keep 1 sec buffer period, which is recommend in most cases.

So this is known issue with x264, and is already discussed in NAL-HRD forum.

But problem occurs on a very specific scene, were video becomes pure noise, with only black and white points, but not brutaly, "slowly", but brutaly comes back to normal. Problem seems to occurs during the frames were there is this brutal transition of pure black and white noise (so high bandwith i think), to a standard anime scene, were bandwith is probably realy lower.I notice same, and mp3dom aslo faced with same behavior, with same scenario (black background).

Since i faced with this problem earlier, before official blu-ray support is commited, i can't remember what source i had. if you can provide source that can always make this behavior i think will finaly x264 devs can do something about this.

btw. try with mp3dom suggestions, i aslo suggest to keep buffer tied to maxrate, iirc mp3dom soloved his problem with --qpcomp 0.5, which change ratecontrol decisions on that frames.

[jpsdr]

If i understand, you suggest for buffer the following : if (maxrate<=30000) buffer=maxrate; else buffer=30000;
and it's what commercial encoders recommend, i am correct ? If yes, maybe you can add this recommendation on the 1rst post of the encoding for Blu-Ray thread you've created.
I can try this for buffer.
As DS doesn't recommend to change qpcomp, i prefer not touch it. I will this evening provide the rapidshare links for the datas, and redo the encode of the video, and test if it works. But, with the non determinist part of x264, who know if simply redoing the exact same thing it will not pass ? As difference was only average bitrate going from 4529 to 4507. But, as it's a little long, i'll try the "sure way" : buffer=maxrate, this will allow me to try redo my authoring tomorrow evening.

[mp3dom]

however comercial encoders recommend to keep VBV buffer size and Maxrate tied as possible (for example --vbv-maxrate 15000 --vbv-bufsize 15000), to keep 1 sec buffer period, which is recommend in most cases.
Uhmm, I've never seen that kind of raccomandation. The only thing that I've noticed with commercial encoders is that if you set a too lower bitrate, the encoder warns you to raise at least to <value> to avoid buffer problems. For example CineVision for L4.1 with 30Mbps buffer, allow you to have a minimum average bitrate of 17 Mbps on 40Mbps (max). Also suggest to have an average bitrate at least 30% less than maxrate.

[shon3i]

If i understand, you suggest for buffer the following : if (maxrate<=30000) buffer=maxrate; else buffer=30000;No just use buffer=maxrate in all cases (15000,15000 in your case), aslo higher maxrate than 30000 can make same scenario or even worse, it's not recommend to use max possible max rate which is 40000.

however this will not guarantee soultion for your problem. But solove in my case, while qpcomp solove in mp3dom case. So solution exist, but a matter of time and nerves ;)

[shon3i]

Also suggest to have an average bitrate at least 30% less than maxrate. I saw that, i think i saw this recommendation in sony knowlege base, they are recommend keep avg bitrate closer to maxrate, and recommend 1 sec buffer, which basicly means keep buffer close to maxrate.

[jpsdr]

No just use buffer=maxrate in all cases (15000,15000 in your case),
So solution exist, but a matter of time and nerves
This case is not my only case, this is why i've said that...
In fact, i've the following cases :
FHD : 40000,30000 level 4.1
HD-1 : 40000,30000 level 4.1
HD-2 : 24000,30000 level 4.0 (change to 24000,24000 so)
HD-3 : 15000,30000 level 4.0 (change to 15000,15000 so)
(2 sec GOP)
SD : 15000,30000 level 4.0 (change to 15000,15000 so)
(2 sec GOP).

To preserve time and nerve, is to be fixed in x264. Back home very soon, will post the links.

[jpsdr]

Ok.
DS and Trahal : I've PM'ed you Christmas presents:D
Notify me when you've finished to get them.:thanks:
You've the encoding parameters in post #208 http://forum.doom9.org/showthread.php?p=1409240#post1409240
Forget : x264 version used is the 1643M x64 version of rack04.

[shon3i]

This case is not my only caseYou tried all of this combinations and always face with problem?

i hope x264 devs will conclude something from your source, and put this issue to bed :)

[jpsdr]

You tried all of this combinations and always face with problem?
Noooo...
What i was saying it's that i'm having different encoding configurations, so according to your advise, i may change buffer to maxrate in the configurations where maxrate<30000.

i hope x264 devs will conclude something from your source, and put this issue to bed :)
hummm........... DS asked me what he was supposed to fix as there is no buffer underflow in the log.....:confused:
There was problems before, with dark parts, but they were clearly identified, and solved since.
But maybe the true answer of my case is doing what you said, according commercial encoders suggestion => buffer=maxrate if maxrate<=30000.

[jpsdr]

@shon3i : I've PM'ed you the links for my file. If i remember correctly (and not mix with mp3dom...), you have tools to check Blu-Ray compliancy, so, if it's possible, if you could get the file and check it.
I've also had the small original sample, and if eventualy you want to play with it and see if you can reproduce something...

[mp3dom]

I've the verifier too, but shon3i has more updated version (so, more reliable) :)
Regarding the commercial encoders, it could be a suggestion to have bufsize=maxrate but anyway they don't force you to do so (personally I don't like it)

[jpsdr]

First, re-encoding the video with buffer=maxbr=15000 mux pass without buffer underflow error.
I've decided, just curious, to install and test bitrateviewer 2.1.1.
......... What was my surprise when i saw the results !!!!
The viewer reported average and peak, i must say in first, that average were always perfectly adequate with the average reported in the log file. So, i consider the information provided by the bitrate viewer reliable.

Here are the peaks values (in kpbs) for encoded files with maxbr=15000 and maxbuff=30000 :
17643, 18226, 18507, 18748, 17913, 16061,14696, 19224, 17180, 22257.
Wohh... !!!! Except for one, that's a way higher than the max bitrate set... Is it normal ??????? Or is there a big problem ???
With a GOP setting of 2s, i don't think these files stay Blu-Ray compliant !
Here are the peaks values (in kpbs) for encoded files with maxbr=15000 and maxbuff=15000 :
15338(*), 14962.
(*) : This was the file wich previously created buffer underflow.
The peak for the (*) file encoded with maxbr=15000 and maxbuff=30000 is 21490.
This is this file i've given the links to DS and kieranrk.
I've restart the encoding of the 10 file encoded with maxbr=15000 and maxbuff=30000 now with maxbr=15000 and maxbuff=15000.
Results not before monday...

[mp3dom]

High peaks doesn't create problem as long as the buffer is filled with data that either don't overflow or underflow. With maxrate=bufsize the 'excursions' of the peaks is more limited over time (hence I don't like it very much).

[jpsdr]

Well, when i limit the bitrate at 15000, i'm a little surprise to see peaks higher than 20000... But if it's normal...

[mp3dom]

It depends by what's the occupancy of the VBV. A high peak could be fully allowed for a short period if the VBV can manage it. I've seen some BD that sometimes have spikes over 60 Mbps (the max allowed is 40 Mbps)

[Trahald]

@jpsdr
I didnt see if you had already done this, but can you make the same encode exact same settings but without open-gop and use a totally unpatched x264 then see if it passes.

Attached is a newer version of the patch. it has changed a little.. the option is " --open-gop coded " if you are doing bluray and " --open-gop display " if you are doing anything else. It does not address the buffer issue (we are still determining if there is one)

[Trahald]

Well, when i limit the bitrate at 15000, i'm a little surprise to see peaks higher than 20000... But if it's normal...

Yes.. it can peak to 25k and still be valid. its not the same as say clipping a audio stream. peaks are ok as long as the buffer can recover.

[shon3i]

It does not address the buffer issue (we are still determining if there is one) Open gop have nothing with underflows here, it's alredy known with unpached bulids and before officialy Blu-Ray commit, even with your HRD patch.

There is probably some undocumented rule, that apply when is muxing to TS. Since all comercial encoders handle this.

[Dark Shikari]

Open gop have nothing with underflows here, it's alredy known with unpached bulids and before officialy Blu-Ray commit, even with your HRD patch.

There is probably some undocumented rule, that apply when is muxing to TS. Since all comercial encoders handle this.Or your muxer is broken. :rolleyes:

Stop spreading FUD about "underflows" that don't exist.

07:18 <kierank> lol his file muxes fine

[shon3i]

Stop spreading FUD about "underflows" that don't exist.An you stop FUD about broken muxers because muxers are just fine and newer have problem with other encoders, and always only with x264. All muxers fail at same place, some little restrictive like tsmuxer will mux stream, but clearly show after that BDAV-STD Model model is completly broken

[Dark Shikari]

All muxers fail at same place, some little restrictive like tsmuxer will mux stream, but clearly show after that BDAV-STD Model model is completly brokenThen where is the proof? Show me where x264 violates the NAL-HRD model, and I will fix it. Whining endlessly without showing proof serves no purpose whatsoever.

Nobody has been able to give me a stream produced by a recent x264 that fails validation in any stream analyzer that I own.

[kieranrk]

Open gop have nothing with underflows here, it's alredy known with unpached bulids and before officialy Blu-Ray commit, even with your HRD patch.

There is probably some undocumented rule, that apply when is muxing to TS. Since all comercial encoders handle this.

I have tested the stream jpsdr sent me and it worked fine with muigenerator.

[jpsdr]

I have tested the stream jpsdr sent me and it worked fine with muigenerator.
Fortunately, otherwise, i'll not even be able to import it in my Blu-Ray project, and so start a mux.

[jpsdr]

@jpsdr
I didnt see if you had already done this, but can you make the same encode exact same settings but without open-gop and use a totally unpatched x264 then see if it passes.

I will, but it will take a little while.
As i said, as problem seems realy on the edge, i don't even know, because of the non-determinist part of x264, if redoing the same thing will not produce a file wich will pass. Indeed, i will also try it, doing a second time the "same" file, to see if it still produces the problem, or magicaly disapered.

[Dark Shikari]

I just spent the past 4 hours rewriting HRD. It's now almost sunrise. So before I completely conk out, try this build (http://x264.nl/developers/Dark_Shikari/x264.exe). It's extraordinarily beta; I'd be shocked if I didn't break at least something horribly. It's almost surely not correct... but we might as well try it. And it's going to be necessary if we intend to fix it.

Includes opengop, threadpool, and various other things that I've locally committed. The relevant patch (http://pastebin.org/343062).

(Thanks to Manao for help understanding a bit of the batshit insanity behind the HRD math.)

[jpsdr]

Thanks, i'll try, results at best in 8 hours, but most probably tomorrow. Good night...

[shon3i]

(Thanks to Manao for help understanding a bit of the batshit insanity behind the HRD math.) Many thanks. I hope it's right thing and solution for mux underflow problem. So i been partialy right?

[jpsdr]

Includes opengop, threadpool, and various other things that I've locally committed.

Are you sure ? Because when i've tried it, it said "unrecognise option" on "--open-gop".

.... "unrecognisze option" on "--nal-hrd".
Doing things at down are not always good...:D
Better take a second look after a good sleep...

[shon3i]

@jpsdr,
both hrd and open gop have sub option

--nal-hrd vbr
--open-gop coded

"coded" is mandatory for blu-ray.

[sneaker_ger]

I can't get them to work either, so jpsdr is probably right. (x264 reports as "core:85 r1416+2 3a4ddc5")

[Dark Shikari]

I can't get them to work either, so jpsdr is probably right. (x264 reports as "core:85 r1416+2 3a4ddc5")$ ./x264 --version
x264 0.99.1649+9 38e7992

Erm... what in the world x264 did you download? I've reuploaded it again just to make sure, but FTP told me I was replacing an identical file, so...

Edit: Replaced it with a new one anyways to fix the stupid bug I introduced. Try it now.

[Dark Shikari]

03:42 < kemuri-_9> Dark_Shikari: go yell at jarod, the link you gave for the
x264 in the open gop thread is giving diff builds of x264
depending on how i download it, e.g. in chrome it's the old
ass build people are bitching about, in wget it's the right
one


Mediafire link (http://www.mediafire.com/?njynu3mnlqy) because jarod's servers are a broken pile of crap.

[jpsdr]

@jpsdr,
--open-gop coded
"coded" is mandatory for blu-ray.
Not yet apparently, or at least, with the version posted by DS.
Beside, i feed x264 with an avs file, but i've an error message saying that raw yuv needs resolution :confused:
First time i see that... Is it normal ?
............ Humm... Ok...It try to open the file 'coded'... Well, ,removing it, and.......... :eek:


avs [info]: avisynth 2.6+ detected, forcing conversion to YV12
avs [info]: 1280x720p 1:1 @ 24000/1001 fps (cfr)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2
x264 [warning]: HRD with (maxrate*timescale*bufsize) > 2^63 not supported
x264 [error]: x264_encoder_open failed


Command line :

@echo off

SET E_SRC=%6%1.avs
SET E_DST=%3%1.264
SET STAT_FILE=%1.stats
SET TUNING=%4
SET LOG_FILE_1=%1_log_1.txt
SET LOG_FILE_2=%1_log_2.txt

REM Set of max bitrate (ici le bitrate max)
set MAX_BR=15000

REM Set of Buffer (ici le buffer)
set BUF_BR=30000

REM 1ère passe
x264_x86v2.exe --profile "high" --preset "placebo" --tune %TUNING% --pass 1 --bitrate %2 --stats %STAT_FILE% --level "4.0" --qpmin 0 --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --keyint 48 --min-keyint 2 --mvrange 511 --ref 6 --bframe 3 --b-pyramid "strict" --open-gop --subme 9 --trellis 1 --me "umh" --aq-mode 2 --aud --nal-hrd "vbr" --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --videoformat "ntsc" --sar 1:1 --qpfile %5 --threads 0 --thread-input --output NUL %E_SRC% 2> %LOG_FILE_1%

REM 2ème passe
x264_x86v2.exe --profile "high" --preset "placebo" --tune %TUNING% --pass 2 --bitrate %2 --stats %STAT_FILE% --level "4.0" --qpmin 0 --vbv-maxrate %MAX_BR% --vbv-bufsize %BUF_BR% --keyint 48 --min-keyint 2 --mvrange 511 --ref 6 --bframe 3 --b-pyramid "strict" --open-gop --aq-mode 2 --aud --nal-hrd "vbr" --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --videoformat "ntsc" --sar 1:1 --qpfile %5 --threads 0 --thread-input --output %E_DST% %E_SRC% 2> %LOG_FILE_2%

bitrate = 4507, tuning = animation.
Wanted to let it run during the night, but it'll be postpone...
See you maybe tomorrow.

[sneaker_ger]

$ ./x264 --version
x264 0.99.1649+9 38e7992

Erm... what in the world x264 did you download? I've reuploaded it again just to make sure, but FTP told me I was replacing an identical file, so...

Edit: Replaced it with a new one anyways to fix the stupid bug I introduced. Try it now.

Well, I guess either both jpsdr and I messed up big time or you should spend the next night rewriting your ftp server. :rolleyes:

[Dark Shikari]

Well, I guess either both jpsdr and I messed up big time or you should spend the next night rewriting your ftp server. :rolleyes:jarod runs the FTP server, not me. As mentioned above, it is in fact horribly broken.

We've (almost surely) fixed the HRD problems locally; just a couple more bugfixes were necessary. Now we have to add in support for larger timebase*maxrate*bufsize.

[Dark Shikari]

OK, HRD rewrite/modifications finished, try this build (http://www.mediafire.com/?m0yntzkbxdz). It should pass mux.

[shon3i]

OK, HRD rewrite/modifications finished, try this build (http://www.mediafire.com/?m0yntzkbxdz). It should pass mux.
Thanks. So what is the difference between old and new HRD modeling?

[Dark Shikari]

Thanks. So what is the difference between old and new HRD modeling?The new one has infinite precision.

[Midzuki]

The new one has infinite precision.

:eek: :cool:
:thanks: