Getting aligned GOPs in different profiles in x264

[benwaggoner]

When encoding for adaptive streaming, it is critical that all bitrates have IDRs aligned, to enable seamless stream switching.

The typical way to do this is to set each stream to an identical fixed, non-adaptive GOP size. Having adaptive GOPs synchronized between bitstreams has obvious advantages, but isn't supported by all formats and players, unfortunately. And isn't supported in off-the-shelf encoders either (except for the Smooth Streaming VC-1 SDK from Microsoft).

The simple way to handle this in x264 is simply setting --keyint and --min-keyint to the same value. And I've confirmed this works fine if all the bitstreams are the same Profile@Level. However, IDRs are not aligned when I make High 3.1 and Main 3.0 encodes from the same source.

For example, (using Simple x264 launcher, so Mode, Preset, and Tuning are all set in the GUI), I use these settings:

Main Profile 3.0:
2-Pass, preset veryslow, tuning film
--level 3.0 --keyint 48 --min-keyint 48 --bframes 3 --b-pyramid strict --ref 4 --slices 1 --vbv-bufsize 10000 --vbv-maxrate 2000 --rc-lookahead 48 --aq-mode 2 --aq-strength 1.2 --no-fast-pskip --nal-hrd vbr --non-deterministic --colorprim bt709 --transfer bt709 --colormatrix bt709

High Profile 3.1:
2-Pass, preset veryslow, tuning film
--level 3.1 --keyint 48 --min-keyint 48 --bframes 3 --b-pyramid strict --ref 4 --slices 1 --vbv-bufsize 17500 --vbv-maxrate 2500 --rc-lookahead 48 --aq-mode 2 --aq-strength 1.1 --nr 250 --no-fast-pskip --nal-hrd vbr --non-deterministic --colorprim bt709 --transfer bt709 --colormatrix bt709

The resulting files won't package in IIS Transform Manager, warning that there isn't IDR alignment.

Jumping into them with the Tektronix MTSE4A stream analyzer, I see the IDRs wind up in different locations.

Main Profile Level 3.0 starts with these IDR frames.

1
49
97
145
192
239
286
333
380
427
474


And High Profile Level 3.1 has these:

1
48
95
142
189
236
283
330
377
424
471
518


I'm attaching the log files from a test as well.

Anyone have any idea what's happening here? Any suggestions?

The best workaround I've thought of is to read the I frames out of the log file and then force those as I in a QPfile. But this seems like a bug to me.

[poisondeathray]

What happens when you use --slow-firstpass with high encodes ? (high isn't used in the 1st pass otherwise, maybe that's the difference?)

Any difference with 1pass abr ?

[benwaggoner]

What happens when you use --slow-firstpass with high encodes ?
I haven't tried, but will later.

Why do you think that might make a difference? The Main/High switch between 1st and 2nd passes?

Hmmm. I should try 1 pass as well...

[poisondeathray]

There are differences as well, you are using --nr 250 in the high encodes, but not in the main, also different aq strenghts, vbv values. It might cause x264 to choose different frametype placements and gop sizes

It might be as simple as different settings causing x264 to choose frametype placements

[benwaggoner]

I stripped the encode down to 1 pass and JUST-- level, --keyint, and --min-keyint. And it actually made alignment worse!

It seems like x264 isn't honoring my --min-keyint for some reason, making it slightly (but disastrously) different in these different modes. High Profile seems worse.

Hopefully forcing them in the qpfile will work, but hopefully one of the devs can look into this.

[Blue_MiSfit]

So you're saying that x264 doesn't seem to produce files with fixed GOP structure across different bitrates?

I'm not sure what's happening there, it definitely seems odd.

Have you thought about using adaptive GOP, but consistent across all bitrates? My understanding is that x264's frametype decision should be consistent for the same content across multiple bitrates. The differences come into play when the resolution changes. So, to make (adaptive) GOP structure consistent across multiple bitrates, re-use the stats file from the first pass on each of the different bitrates. I've found starting at the top resolution / bitrate and working down seems to work the best.

Also.. at one point, mbtree interfered with mixed resolutions and matched GOPs. I'm not sure if this is still the case.

[Blue_MiSfit]

Oh... what about adaptive b-frames? That wouldn't change IDR placement, right? It would only affect P/B choice?

Also... tried setting --no-scenecut?

[benwaggoner]

So you're saying that x264 doesn't seem to produce files with fixed GOP structure across different bitrates?
Actually, it seems to do it fine with different bitrates. It's between Main 3.0 and High 3.1 I see the difference, but multiple bitrates within each of those all align perfectly.

I should try High 3.0 v. 3.1 and High 3.0 v. Main 3.0 to see if it's Profile or Level or what.

My understanding is that x264's frametype decision should be consistent for the same content across multiple bitrates. The differences come into play when the resolution changes.
Ah! I am using a different frame size with Level 3.0 and Level 3.1 (as you'd expect).

Adaptive streaming needs to be able to have the same GOP structure in different frame sizes as well as bitrates.

So, to make (adaptive) GOP structure consistent across multiple bitrates, re-use the stats file from the first pass on each of the different bitrates. I've found starting at the top resolution / bitrate and working down seems to work the best.
That makes sense.

[Blue_MiSfit]

I've been whining in #x264. You need to disable scenecut if you want constant GOP structure.

Also (10:45:08 AM) Skyler_: don't touch min-keyint, that's not what it does.

Finally, if you re-use the stats file from the top resolution / bitrate to generate all other streams, your GOP structure should match. I tested this awhile back and it did work.

[sneaker_ger]

min-keyint is capped at keyint/2+1
Try disabling scenechange detection, as suggested by Blue-MiSfit. (--no-scenecut)

Also.. at one point, mbtree interfered with mixed resolutions and matched GOPs. I'm not sure if this is still the case.

It just didn't work, IIRC. Didn't have to do anything with IDR-placement.
http://git.videolan.org/gitweb.cgi?p=x264.git;a=commit;h=913485d26b19dddb6340f7115843d63cde8bb836

Using a qpfile or re-using the complete stats file (to save time) should be better than fixed GOPs, though. As you can see from the git log comment, the penalty for re-using the stats can be very low.

[benwaggoner]

I've been whining in #x264. You need to disable scenecut if you want constant GOP structure.
Ah. Will I still get non-IDR i-frames as appropriate for compression?

Also, if this isn't what --min-keyint does, what does it do :)?

Finally, if you re-use the stats file from the top resolution / bitrate to generate all other streams, your GOP structure should match. I tested this awhile back and it did work.
Good. Great to hear that mbtree will work with it now! Do we know if that make it into build 2008?

[SassBot]

Also, if this isn't what --min-keyint does, what does it do :)?

It does exactly what the name of the option says, but you missed the part in the explanation of it about how it's capped at keyint/2 + 1 as sneaker points out. So in reality while you specified 48 as min-keyint it was set to 25 since your value exceeded the maximum allowed value. The relevant code being:


if( h->param.i_keyint_min == X264_KEYINT_MIN_AUTO )
h->param.i_keyint_min = X264_MIN( h->param.i_keyint_max / 10, fps );
h->param.i_keyint_min = x264_clip3( h->param.i_keyint_min, 1, h->param.i_keyint_max/2+1 );

[Blue_MiSfit]

@Ben:

http://mirror01.x264.nl/x264/changelog.txt

It looks like the mb-tree fix was pushed on June 8th, in rev 2205. I'm assuming you typo'd 2008 and meant 2208, in which case yes you should have that fix. If not, time to update :devil:

Derek

[sneaker_ger]

Is no one but SassBot reading my posts? :confused:

[benwaggoner]

@Ben:

http://mirror01.x264.nl/x264/changelog.txt

It looks like the mb-tree fix was pushed on June 8th, in rev 2205. I'm assuming you typo'd 2008 and meant 2208, in which case yes you should have that fix. If not, time to update
Yes, typo :).

I'm working on my big .bat file to test all this right now.

[benwaggoner]

Is no one but SassBot reading my posts? :confused:
I am, and they were really informative, thanks!

Related to your info, if I wanted totally fixed cadence, I'd need to do that with a qpfile, right? Basically one where every 48th frame (for 24p @ 2sec) is specified I, and no other information. I imagine I should also use --open-gop to let the mid-chunk I-frames be not IDR. Some muxers get confused if there are extra IDRs.

Getting devices to not require fixed GOP sizes would be preferable, of course.

[sneaker_ger]

A qpfile alone setting the IDR-frames is not sufficient, as the scenecut detection can throw in additional IDR-frames. You have to turn the scenecut detection off - a qpfile is not needed. But I really suggest you leave it on and re-use the stats file instead as suggested by Blue_MiSfit.
I don't know what container you are using, but the last time I looked, open-gop was not recommended in an mp4 container. Seeking to non-IDR I-frames for example does not work.

[Blue_MiSfit]

deleted

[benwaggoner]

A qpfile alone setting the IDR-frames is not sufficient, as the scenecut detection can throw in additional IDR-frames. You have to turn the scenecut detection off - a qpfile is not needed. But I really suggest you leave it on and re-use the stats file instead as suggested by Blue_MiSfit.
I don't know what container you are using, but the last time I looked, open-gop was not recommended in an mp4 container. Seeking to non-IDR I-frames for example does not work.
But the log file approach wouldn't give a fixed IDR, pattern though, right?

Allowing variable GOP is preferable, but there may be platforms that require a strictly fixed IDR pattern.

[sneaker_ger]

But the log file approach wouldn't give a fixed IDR, pattern though, right?

Correct. If you absolutely need a fixed IDR-pattern, you need to use --keyint and --no-scenecut AFAIK.
You could of course combine the two to save some time while still retaining a fixed pattern.

[benwaggoner]

One complexity I rediscovered is that CRF isn't allowed in multi-pass encoding. My fantasy would be able to have each bitrate be capped by both CRF and by maximum bitrate. For example, a 5000 Kbps stream with CRF 19 and a 3000 Kbps stream with CRF 21. Each stream would stay at the target CRF unless more than the allowed maxrate was required. That combination would ensure that a given chunk would always be smaller in a lower bitrate stream than in a higher bitrate stream. Varying just CRF, the peaks could be identical at the peak bitrate. Varying just bitrate, quality could be identical on the easier parts of the video.

Combining CRF and maxrate would solve this nicely. However, using CRF in a two-pass encode isn't allowed , and throws my new best friend:
x264 [error]: constant rate-factor is incompatible with 2pass.
x264 [error]: x264_encoder_open failed

Any suggestions? The best I've come up with is to calculate a target --bitrate based on an initial CRF first pass to add to the --vbv-maxrate. Seems a bit clunky, though.

[Dark Shikari]

VBV + CRF in 1-pass mode works just fine. Why do you need 2-pass?

[benwaggoner]

VBV + CRF in 1-pass mode works just fine. Why do you need 2-pass?
I'm using -p 2 for the alternate bitrates of the initial encode to force aligned IDRs.

[Blue_MiSfit]

And using CRF instead of just accepting CBR at your target maxrates is to save bandwidth, right?

It's an interesting question.

Derek

[benwaggoner]

And using CRF instead of just accepting CBR at your target maxrates is to save bandwidth, right?
Exactly. No need for 5 Mbps credits, and so on.

Maybe I really just want a --min-crf parameter.

[detmek]

Maybe a new feature - export qp file with all or just IDR frames so it can be used for another encode.

[videoRx]

Ben,

Did you try Dark Shikari's suggestion with using CRF and VBV with 1 pass? Have you confirmed that you really need the first pass log for keyframe alignment across bitrates? I'm encoding with ffmpeg/x264 for use with Wowza, and to date, I haven't had any issues with just using avg bitrate, max rate, and VBV-buffer size to have consistent chunking. I should likely do some deeper analysis of the output from x264 as you have (Tektronix app) to confirm that all is well.

Also, I'm curious why you're using nal_hrd = vbr instead of cbr?

-Robert

[benwaggoner]

Did you try Dark Shikari's suggestion with using CRF and VBV with 1 pass? Have you confirmed that you really need the first pass log for keyframe alignment across bitrates? I'm encoding with ffmpeg/x264 for use with Wowza, and to date, I haven't had any issues with just using avg bitrate, max rate, and VBV-buffer size to have consistent chunking. I should likely do some deeper analysis of the output from x264 as you have (Tektronix app) to confirm that all is well.
Wowza might be more flexible in GOP alignment than other tools. Among other things I'm using is IIS Transform Manager, which can handle an occasional misalignment, but all the streams need to have the same IDR.

As long as I use a qpfile CRF+VBV should work fine, but I need that qpfile.

Also, I'm curious why you're using nal_hrd = vbr instead of cbr?
Because I'm using CRF, and so these really are VBR files. And nal_hrd can help some muxers more easily repackage as fragmented MPEG-4. It's not strictly required, but makes it go faster and is generally recommended.

[videoRx]

Thanks for the reply, Ben. I don't have access to expensive analysis tools like the Tektronix one, but when I review a 1 pass CRF + VBV file across different bitrates and resolutions in a Flash Player-based metadata handler, the seek times on each version is aligned with the same time interval (and seek times in Flash Player are based on keyframe positions). Interestingly, though, if I'm specifying a 6 second GoP in FFmpeg/x264 for 29.97fps footage (180 frame GoP), the keyframes are 6.006 seconds apart, but they're at least consistently 6.006 seconds apart.

[benwaggoner]

Thanks for the reply, Ben. I don't have access to expensive analysis tools like the Tektronix one, but when I review a 1 pass CRF + VBV file across different bitrates and resolutions in a Flash Player-based metadata handler, the seek times on each version is aligned with the same time interval (and seek times in Flash Player are based on keyframe positions). Interestingly, though, if I'm specifying a 6 second GoP in FFmpeg/x264 for 29.97fps footage (180 frame GoP), the keyframes are 6.006 seconds apart, but they're at least consistently 6.006 seconds apart.
6.006 seconds is 180 frames @ 29.97 fps (30/1.001, more precisely). To get 6.000 it would need to be a real 30 fps.

[videoRx]

Duh, of course! Do any encoders allow floats for keyframe interval, to accommodate fractional frame rates?

[sneaker_ger]

It's not possible to have a keyframe interval of exactly 6 seconds if there are no frames that are exactly 6 seconds apart in the first place.

[videoRx]

@sneaker_ger, thanks for the reply. My concern is that streaming media servers are set to chunk a keyframe-aligned stream at a specific interval, and to date, I have never seen docs take floats for the value, only integers. Based on what I see in Wowza Media Server logs, it looks like it's built to take fractional framerates into account, as it will break on 6.006 instead of 6.0 seconds. But I shouldn't be surprised by that, right? The industry loves to round fractional frame rates in a number of software tools.

[benwaggoner]

@sneaker_ger, thanks for the reply. My concern is that streaming media servers are set to chunk a keyframe-aligned stream at a specific interval, and to date, I have never seen docs take floats for the value, only integers. Based on what I see in Wowza Media Server logs, it looks like it's built to take fractional framerates into account, as it will break on 6.006 instead of 6.0 seconds. But I shouldn't be surprised by that, right? The industry loves to round fractional frame rates in a number of software tools.
Well, all NTSC video *is* fractional frame rate, and so that is supported by all modern encoders and servers I know of. Every Smooth Streaming manifest I see actually has 2.002 sec between the "2 sec" chunks.

The last tool I remember that got fits from fractional frame rates was Adobe Premiere versions pre 4.0 (early-mid 90's). They treated "29.97" as exactly 29.97, which caused an audio drift with long-form content.