Splitting H.264 elementary stream "cleanly" without re-encode

[GrandAdmiralThrawn]

Hello!

I tried to do this before using ffmpeg, but somehow failed, it would always create 0-Byte files as output (I do not remember the exact options I used unfortunately).

So, what I want to do is this: I have an H.264 elementary stream (.264 file) that I want to split up into many fragments without re-encoding anything. Preferrably I would like to do it on Linux commandline, but Windows would also be ok.

The reason is that i want to play around with distributed computing clusters using PBS systems, and so far I have just split up the stream using dd, which is extremely dirty as it just cuts somewhere in the middle of a frame etc...

So, is there an easy way to split up say a 1GB .264 file into 50 pieces with each piece being around the same size (equal in either MBytes or frame counts)?

Thanks!

[LoRd_MuldeR]

DGIndexNV can do that, I think.

Also MKV Toolnix can split, though you may need to create a temporary MKV file you extract later...

[GrandAdmiralThrawn]

I have DGIndex and DGAVCIndex, but not DGIndexNV (wasn't that a commercial CUDA version?).

But I'm using MKV Toolnix all the time, creating a temporary MKV, splitting into multiple MKVs and then demuxing them again would be ok for me, so I'm going to try that, thanks for the hint!

I'll let you know whether I was successful. :)

[Guest]

You're going to run into problems with open GOPs. It can be worked around but it is not a simple file split.

[GrandAdmiralThrawn]

Ok, I have tried this now, and it seems that MKV Toolnix does exactly what I wanted. There is one interesting thing though!

I told it to split the file every 13700 Kilobytes (to reach an amount of 50 pieces from my ~669MB file). However, the MKV pieces are not of equal size, and MKV compression (zlib/bz2/..) is turned off.

Could it be, that MKV Toolnix respects GOPs and splits the stream right before a new I-Frame, even if it means to slightly deviate from the specified target file size?

Is that even possible? If it is, that would be absolutely perfect, right? Or why else would the target files be of slightly different sizes? The difference can be quite significant, min-2-max around 2MBytes. The test stream comes from the OS movie "Elephants Dream" in 1080p. I'm using a slightly older version of MKV Toolnix, which is the GUI version 5.7.0.

[Guest]

Elephant's Dream has closed GOPs if I remember correctly. Can you check it?

[LoRd_MuldeR]

Could it be, that MKV Toolnix respects GOPs and splits the stream right before a new I-Frame, even if it means to slightly deviate from the specified target file size?

Anything else would break the stream. It unavoidably has to split at GOP boundaries, I think.

And with "OpenGOP" it won't be possible to make a "clean" cut, because the last B-Frame(s) of previous GOP will depend on the IDR-Frame of the next one.

(Well, unless you include the "problematic" IDR frame twice, once at the end of segment n and once at the beginning of segment n+1 !?)

[sneaker_ger]

Yes, basically. As implied by neuron2, mkvmerge does not differentiate between IDR-Frames and recovery points, so it does not work correctly on things like OpenGOP.

[Guest]

(Well, unless you include the "problematic" IDR frame twice, once at the end of segment n and once at the beginning of segment n+1 !?) You have to include the previous GOP so that the orphaned frames can be decoded. Then you need to know the number of frames to skip from that extra GOP. So the fragments will have overlap. This is what I referred to earlier as a workaround. I know it works because I implemented it for a well-known streaming provider.

[GrandAdmiralThrawn]

I am afraid that I do not know what kind of GOPs Elephants Dream uses. I wasn't even aware of any kind of GOPs where there would be inter-GOP dependencies...

This is beyond my level of knowledge. Could you advise how I could determine the type of GOP that Elephants Dream uses? As far as I know the stream that I have is still not a transcoded version of the movie, should be original.

Thanks!

[Guest]

I would just open it in DGIndexNV, navigate on the timeline to the middle of the movie, then hit single step play from that point. If the first few frames are macroblocked, then the GOP is open. Of course, you could also just save project from the start and inspect the DGI file, but you would need to know about GOPs and frame types to be able to interpret it.

[LoRd_MuldeR]

I am afraid that I do not know what kind of GOPs Elephants Dream uses. I wasn't even aware of any kind of GOPs where there would be inter-GOP dependencies...

There is an explanation here:
http://forum.doom9.org/showpost.php?p=1300124&postcount=32

[Guest]

I checked Elephant's Dream and it has closed GOPs.

[GrandAdmiralThrawn]

Thanks for the link (and for checking the file before me)! So I assume that if MKV Toolnix does split at GOP boundaries, it should be fine at least for Elephants Dream, right?

So I would only need to inspect the split stream parts and see where in the order of frames the split was done to find out whether it's been done right by MKV Toolnix?

But I'm guessing that the tool knows what it's doing..