H264 splitter?

[asarian]

I wonder, do H264 video/file splitters exist? Generally video splitter split across I-Frames, which sounds okay. But what about B-Frames then? What if they look back to before the last I-Frame? (Or is that not allowed?). Maybe something that splits on IDR frames?

I'm also looking into going the AviSynth route, with EDLmaker, to cut on scene changes (although that looks even riskier).

[videoh]

Yes, of course such splitters exist. If you have open GOPs, you'll need something that re-encodes the GOP. VideoRedo is often used for AVC editing. Source filters, such as DGSource(), will also let you serve arbitrary clips via an Avisynth script, avoiding having to actually split source streams.

[asarian]

Yes, of course such splitters exist. If you have open GOPs, you'll need something that re-encodes the GOP. VideoRedo is often used for AVC editing. Source filters, such as DGSource(), will also let you serve arbitrary clips via an Avisynth script, avoiding having to actually split source streams.

Thx. I'll look into VideoRedo. Already found it mentioned several times, and it comes highly recommended. :)

Yeah, definitely don't want things split on open GoP's, as that would be horrible.

[asarian]

Yes, of course such splitters exist. If you have open GOPs, you'll need something that re-encodes the GOP. VideoRedo is often used for AVC editing. Source filters, such as DGSource(), will also let you serve arbitrary clips via an Avisynth script, avoiding having to actually split source streams.

Turns out VideoRedo is an absolute disaster! First of all, it can't handle lossless H264 (really); and then it bumped out on my lowly mpeg-1 source, saying it had no audio (which it didn't, of course, but who cares?!).

All I want is a h264 splitter than can 'GoP-safe' split an input file into multiple parts, LOL. So, "of course such splitters exist." does not feel so natural to me any more as I thought it would.

[vivan]

Turns out VideoRedo is an absolute disaster! First of all, it can't handle lossless H264 (really);Lossless H.264 is different from "regular" H.264 (High Profile I mean), so it's not that surprising.

Splitting not on IDR is a bad idea, so tools should avoid that (and instead of splitting on selected frame they split on the nearest IDR).
Try mkvtoolnix, mp4box/yamb or ffmpeg...

[sneaker_ger]

Mkvtoolnix splitting does not work correctly with open GOPs.

[Dust Signs]

I wrote a H.264 bit stream splitter for a project at work once, but I assume that it won't be of much help for you since it
1. Does not allow you to split at open GOPs, only closed GOPs with IDRs
2. Splits the bit stream into all of its GOPs at once and puts them in an output directory (so splitting is not limited to a certain frame or point in time)
Although one could trivially extend this and merge the files back together (which costs an unnecessary amount of time and disk space), I assume that this is not what you are looking for.

If you are looking for a splitting tool, you could try the following workaround which worked for me at times: Put your H.264 bit stream into an MP4 container (using mp4box), then cut it (using mp4box) and re-extract it (using mp4box). Of coure, this will only cut at the nearest IDR frame, but if this is what you are looking for, this may be a workaround to get what you want.

Dust Signs

[asarian]

1. Does not allow you to split at open GOPs, only closed GOPs with IDRs

No, no, that's a GOOD thing! :) Splitting inside an open GoP invariably means you're gonna get glitches. In fact, I was looking for precisely a tool like yours, LOL, that just doesn't blindly split (like every 1,000th frame or something), but does so without ruining the project.

2. Splits the bit stream into all of its GOPs at once and puts them in an output directory (so splitting is not limited to a certain frame or point in time)
Although one could trivially extend this and merge the files back together (which costs an unnecessary amount of time and disk space), I assume that this is not what you are looking for.

Again, LOL, this is precisely what I'm looking for! It will be trivial to splice a multitude of GoP sets into AviSynth. And, since they're spit on GoP already, it will be incredibly easy to encode with overlap!

Tl;dr: please, post it! :) And thank you!

[sneaker_ger]

If you want to process in AviSynth that implies complete re-encoding. Why would you need to split files on GOP borders if you can split/trim on arbitrary frames in AviSynth anyways?

[asarian]

If you want to process in AviSynth that implies complete re-encoding. Why would you need to split files on GOP borders if you can split/trim on arbitrary frames in AviSynth anyways?

The idea is to split up my encoding into several standalone parts, so all parts need to be split on closed GoP's. I can then use AviSynth to encode, say, 1,000 GoP's at a time, instead of having one wickedly long, 200+ hours encoding project, only to have it crash at 90% or something.

These parts will be encoded losslessly, and then will undergo a second x264 pass, encoded with a small overlap, in a process that only does the x264 encoding (and not hefty other AviSynth stuff).

[sneaker_ger]

Why don't you resume your lossless (or lossy) encode from the point it crashed (minus a few GOPs at the end that might be broken because of the crash)?

[asarian]

Why don't you resume your lossless (or lossy) encode from the point it crashed (minus a few GOPs at the end that might be broken because of the crash)?

I suppose I could. But I rather just have a tool that will make it easier, and cleaner, just splitting up a project into several parts automagically. :)

Besides, few progs there be that even take a lossless h264 stream for me to determine the right split-point to begin with.

[Dust Signs]

Tl;dr: please, post it! :) And thank you!

I'll have to clarify about the license I release it under with my employer first, so this might take a bit.

Dust Signs

[videoh]

When splitting AVC files you have more to worry about than just whether the GOP is open. You also must ensure that each split you make is given copies of the applicable SPS/PPS. They may not exist before every IDR.

[Dust Signs]

When splitting AVC files you have more to worry about than just whether the GOP is open. You also must ensure that each split you make is given copies of the applicable SPS/PPS. They may not exist before every IDR.

It very much depends on the types of streams that you are dealing with. If you want to be able to split every possible H.264-compliant stream there is, a lot of complex cases have to be considered.
If you limit yourself to some types of bit streams, things become a lot easier. The tool I wrote, for example, used to be applied to demuxed H.264 bit streams from Blu-rays or bit streams of very similar structure. In this case, every closed GOP has an SPS, a PPS and an IDR picture, while open GOPs have additional IDR pictures, which makes it quite easy to detect them and to split the bit stream accordingly (open GOPs are combined with the next closed GOP).
For other types of bit streams, it would of course be trivial to copy the SPS and PPS to those GOPs which don't have them, but it would be a lot harder to detect open GOPs in the first place.

Dust Signs

[videoh]

open GOPs have additional IDR pictures That makes no sense, because an IDR is not open.

it would of course be trivial to copy the SPS and PPS to those GOPs which don't have them I suppose you'll think it's trivial until you try to do it. ;)

[STaRGaZeR]

Unless I haven't understood the OP, he only wants to split at IDR frames, nothing more and nothing less. Mkvtoolnix is just perfect for the job, no need to mess with anything else. Even more so if the resulting parts are to be encoded to something else afterwards.

[sneaker_ger]

No, mkvtoolnix does split on non-IDR-I-frames for open GOP files resulting in broken files. It's only safe for non-open GOP.

[Dust Signs]

That makes no sense, because an IDR is not open.

I am sorry, but this is not true. There are several Blu-rays out there which use open GOPs with IDRs (instead of I frames). It seems really strange (and is a pain to process), but it is not as uncommon as one might think.

Dust Signs

[STaRGaZeR]

No, mkvtoolnix does split on non-IDR-I-frames for open GOP files resulting in broken files. It's only safe for non-open GOP.

Why do you suppose his stream is encoded with open GOPs? The OP doesn't specify anything about it, and the probability of his stream having them is small. He should try mkvtoolnix, and if that doesn't work, then use other tools. It's absurd to start working with the most difficult tools if the easy ones work.

[asarian]

Thx for all the suggestions, guys! But it looks like maybe this whole splitting process is a lot more complex than I thought. :) And if it can't be done 100% safely, then splitting up a 200 hours process would still be kinda useless.

[asarian]

Why do you suppose his stream is encoded with open GOPs? The OP doesn't specify anything about it, and the probability of his stream having them is small. He should try mkvtoolnix, and if that doesn't work, then use other tools. It's absurd to start working with the most difficult tools if the easy ones work.

My input source would be the lossless h264 stream (from an original MPEG-1 movie).

[STaRGaZeR]

Did you encode it yourself with x264? I guess you did. Then, if you didn't specify --open-gop (default is closed GOP) you have a perfect candidate for mkvtoolnix. Try it :)

[asarian]

Did you encode it yourself with x264? I guess you did. Then, if you didn't specify --open-gop (default is closed GOP) you have a perfect candidate for mkvtoolnix. Try it :)

Yes, I encode the lossless h264 step just as an intermediate state, to be used (in parts, preferably) for a wicked long-taking upscaling process. :)

[videoh]

I am sorry, but this is not true. There are several Blu-rays out there which use open GOPs with IDRs (instead of I frames). It seems really strange (and is a pain to process), but it is not as uncommon as one might think.
The very existence of the IDR as the first frame of the GOP guarantees that the GOP is closed, i.e., that it makes no reference to previous pictures.

Please provide a link to such a file and explain why you think the GOPs are open. Thank you.

[Dust Signs]

One concrete example is the first chapter of the German Blu-ray of the movie "Gandhi". GOP #99/#100 (the numbering depends on how whether you count open GOPs as GOPs) is such an open GOP which contains an IDR picture, but is not decodable from the IDR picture onwards when split (details see below). I cannot provide a sample, though, due to two reasons: 1. the file is quite large (about 1 GB), 2. the bit stream needs to be decrypted in order to be processed (so even if I cut out a few GOPs around the one in question, I am not allowed to share the content).
This is what GOP #99 looks like (excluding AUD and SEI; all pictures consist of four slices each):
* SPS
* PPS
* IDR
* 5*P
* IDR
* 12*P
Note that there is no SPS/PPS pair before the second IDR picture, so the stream cannot be split there. Of course, as I said, it is trivial to just copy the SPS and PPS to make splitting work again, but it is easier to combine the two GOPs to one.

Dust Signs

[foxyshadis]

That's not an open GOP, where one or more of the first GOP's frames reference the next IDR or a frame following it. That's just a Blu-ray spec violation.

[Dust Signs]

Is this really a spec. violation? There are dozens of Blu-rays like this (not only German ones).
Regarding open GOPs: I thought the definition of an open GOP was that it was not decodable by itself starting from the key frame (which is true for the example that I gave, since the SPS and PPS are missing). I am aware that there are open GOPs with I frames where the order is like you describe. Those are not decodable by themselves starting from the key frame, either.

Dust Signs

[sneaker_ger]

I think usually you decide about "open"/"closed" by looking at the references at or over I frames, not the headers. In formats like mkv or mp4 you don't need to store the headers in the bitstream itself at all. The problem is that these terms are "historical", the H.264 specs don't know about "open GOP" or "closed GOP". (Even the term "GOP" itself is not used there.)

[foxyshadis]

Is this really a spec. violation? There are dozens of Blu-rays like this (not only German ones).
Regarding open GOPs: I thought the definition of an open GOP was that it was not decodable by itself starting from the key frame (which is true for the example that I gave, since the SPS and PPS are missing). I am aware that there are open GOPs with I frames where the order is like you describe. Those are not decodable by themselves starting from the key frame, either.

Dust Signs

Blu-ray spec (and HD-DVD and DVD specs) say that every random-access frame has to be immediately decodable, no matter where you seek from, which means that either the SPS/PPS always has to included, or every single one in the entire stream has to be identical (since at a minimum, the first SPS/PPS of whatever chapter it starts at will be decoded and kept in memory). Admittedly, the latter situation is probably pretty common, so it's only a violation if any are different. If not, it's just a waste including it for most GOPs.