AVC/h.264 to MKV. Can it be done?

[The_Keymaker]

Hello forum, I need some help. Has anyone been able to successfully mux AVC/h.264 encoded HD-DVDs into an MKV container?

Since the introduction of HD-DVD and with the backup tools available on this site, I have been successfully backing up my VC-1 encoded HD-DVDs. I usually rip the movie to hard drive using AnyDVD, rebuild and demux the EVOs using EVOdemux, if necessary extract AC-3 audio using eac3to and my own EAC3toGUI utility, Mux into MKV using GDSmux and set time codes and chapters using MKVMerge (MKVtoolnix).

However, I have not yet been able to successfully mux an AVC/h.264 encoded HD-DVD to MKV! GDSmux and MKVmerge will not mux AVC/h.264 files correctly. Sure I could re-encode the files, but who wants to wait 18 - 24 hours for an encode? and even then, the results are not lossless. After all, the main reason I buy HD-DVDs is for the higher resolution and detail.

Alternatively I would be willing to use another container format as long as it allows me to mux the video with my chosen audio track and does not require re-encoding the AVC/h.264 to a lossy format.

Thanks in advance.

The_Keymaker

[nurbs]

I have no experience with hddvd, but can you demux a raw avc stream from the evo? If so you can mux that to mp4 using mp4box (YAMB is a good gui) and then mux the resulting mp4 with MKVmerge.

[honai]

Same problem here. I successfully remuxed the following AVC streams to MKV:

- captures from Premiere HD
- X-Men 3 Blu-ray

While it failed on:

- several Blu-ray titles, including Casino Royale and Pirates of the Caribbean.

I have prepared a number of sample files, each is 200MB or less. I think this is covered by fair use rights. Source is POTC: The Curse of the Black Pearl (Blu-ray). Sample files are cut from the beginning of the respective streams.


potc-cotbp-sample.m2ts - ripped from Blu-ray, untouched, includes video and all audio streams (DD, LPCM) - http://www.sendspace.com/file/fgqo9o
potc-cotbp-sample.ps - PES video stream extracted from the .m2ts with TsRemux 0.0.17 - http://www.sendspace.com/file/z56dvm
potc-cotbp-sample.es - elementary video stream extracted from the .m2ts with TsRemux 0.0.17 - http://www.sendspace.com/file/hgi6he
potc-cotbp-sample.ts - transport stream with video only, remuxed from .m2ts with TsRemux 0.0.17 - http://www.sendspace.com/file/w9y1xv
potc-cotbp-sample.mkv - elementary video stream demuxed with Nero Splitter and fed into Haali Matroska Muxer (2007-06-03) - http://www.sendspace.com/file/cqc9p4


My findings:

- potc-cotbp-sample.ts plays in MPC with Haali splitter from 2007-06-03 and CoreAVC 1.5.0, but playback is choppy
- potc-cotbp-sample.mkv does not play at all with above configuration, fps is stuck at 0.00, but progress slider advances, black screen

The route of going through GraphEdit with Haali Media Splitter -> Haali Matroska Muxer didn't work at all on the .m2ts, the splitter stopped after a few seconds into the video stream, hence the try with Nero splitter.

I did some more tests:

potc-cotbp-sample-xport.es - elementary video stream extracted from the .m2ts with xport 0.96 - http://www.sendspace.com/file/yxb9wx
potc-cotbp-sample-xport.mkv - above elementary video stream muxed into MKV with mkvmerge 2.0.2 (2007-06-23) - http://www.sendspace.com/file/0zpdcl


That .mkv does play somehow, but video output is garbled and contains lots of green blocks.

The same effect can be observed if you mux the .ps/.es streams from the first list into .mkv with mkvmerge, or into .mp4 with mp4box/YAMB - garbled video.

Maybe there are still bugs in xport/TsRemux concerning AVC streams? The authors might want to take a look at the sample files. ;)

xport prints this info for the original .m2ts:


program = 1, video channel = 1, audio channel = 0
Program Number = 0 (0x0000), Program Map PID = 31 (0x001f)
Program Number = 1 (0x0001), Program Map PID = 256 (0x0100)
program descriptor = 0x05, 0x04, 0x48, 0x44, 0x4d, 0x56
program descriptor = 0x88, 0x04, 0x0f, 0xff, 0xff, 0xfc
Video PID = 4113 <0x1011>, type = 0x1b
ES descriptor for stream type 0x1b = 0x05, 0x08, 0x48, 0x44, 0x4d, 0x56, 0xff, x1b, 0x61, 0x3f
ES descriptor for stream type 0x1b = 0x28, 0x04, 0x64, 0x00, 0x29, 0x3f
ES descriptor for stream type 0x1b = 0x86, 0x07, 0xe1, 0x00, 0x00, 0x00, 0x7e, x7f, 0xff
ES descriptor for stream type 0x80 = 0x05, 0x08, 0x48, 0x44, 0x4d, 0x56, 0xff, x80, 0x61, 0xff
ES descriptor for stream type 0x81 = 0x05, 0x04, 0x41, 0x43, 0x2d, 0x33
ES descriptor for stream type 0x81 = 0x81, 0x03, 0x06, 0x48, 0x0f
ES descriptor for stream type 0x81 = 0x05, 0x04, 0x41, 0x43, 0x2d, 0x33
ES descriptor for stream type 0x81 = 0x81, 0x03, 0x06, 0x48, 0x0f
ES descriptor for stream type 0x81 = 0x05, 0x04, 0x41, 0x43, 0x2d, 0x33
ES descriptor for stream type 0x81 = 0x81, 0x03, 0x06, 0x48, 0x0f
ES descriptor for stream type 0x81 = 0x05, 0x04, 0x41, 0x43, 0x2d, 0x33
ES descriptor for stream type 0x81 = 0x81, 0x03, 0x08, 0x28, 0x05
ES descriptor for stream type 0x81 = 0x05, 0x04, 0x41, 0x43, 0x2d, 0x33
ES descriptor for stream type 0x81 = 0x81, 0x03, 0x08, 0x28, 0x05
ES descriptor for stream type 0x81 = 0x05, 0x04, 0x41, 0x43, 0x2d, 0x33
ES descriptor for stream type 0x81 = 0x81, 0x03, 0x08, 0x28, 0x05
0 frames before first I-frame
High Profile
Level = 4.1
First Video PTS = 0x0002bf20
ts rate = unspecified
packets for pid 0 <0x0000> = 446, first = 1, last = 1039713
packets for pid 31 <0x001f> = 45, first = 4, last = 1025147
packets for pid 256 <0x0100> = 893, first = 2, last = 1041609
packets for pid 4097 <0x1001> = 446, first = 5, last = 1040173
packets for pid 4113 <0x1011> = 771164, first = 6, last = 1041667
packets for pid 4352 <0x1100> = 190553, first = 886, last = 1041569
packets for pid 4353 <0x1101> = 18438, first = 837, last = 1041461
packets for pid 4354 <0x1102> = 18438, first = 838, last = 1041462
packets for pid 4355 <0x1103> = 18438, first = 839, last = 1041463
packets for pid 4356 <0x1104> = 6225, first = 843, last = 1041579
packets for pid 4357 <0x1105> = 6220, first = 953, last = 1040973
packets for pid 4358 <0x1106> = 6225, first = 847, last = 1041580
packets for pid 4608 <0x1200> = 51, first = 672855, last = 817001
packets for pid 4609 <0x1201> = 160, first = 168136, last = 257026
packets for pid 4610 <0x1202> = 153, first = 168545, last = 257041
packets for pid 4611 <0x1203> = 1080, first = 191454, last = 1010947
packets for pid 4612 <0x1204> = 1175, first = 268221, last = 984482
packets for pid 4613 <0x1205> = 1204, first = 61475, last = 935638
packets for pid 4614 <0x1206> = 160, first = 168137, last = 257028
packets for pid 4615 <0x1207> = 153, first = 168546, last = 257042
coded frames = 0, video fields = 0

[dmz01]

- potc-cotbp-sample.ts plays in MPC with Haali splitter from 2007-06-03 and CoreAVC 1.5.0, but playback is choppy


You need to have at least one audio stream for the playback not to be choppy. It is the sound card that provides the master clock during playback of a/v on a PC. (see the clock icon on the audio renderer in graphedt)


The route of going through GraphEdit with Haali Media Splitter -> Haali Matroska Muxer didn't work at all on the .m2ts, the splitter stopped after a few seconds into the video stream, hence the try with Nero splitter.


Haali needs to update his splitter to support the blu-ray/avchd standard of storing sound in TS. See my notes on TsRemux.

I somehow doubt that the issues you encounter with the elementary streams are due to xport/tsremux as demuxing from TS is very straight forward.

[dmz01]

The big issue for me with the MKV container in general, is that the elementary streams are modified prior to storage inside the medium. Headers and maybe other data is removed from the stream and stored in a separate coded specific blob. It's very hard to reconstruct the original streams from the MKV modified ones. This is why for example MKV->TS in TsRemux only supports MPEG2 and AC3 for now. The issue with AVC/h.264 is even more pronounced because no public documentation exist on how an AVC stream is modified. The only public information from Haali:
http://haali.cs.msu.ru/mkv/codecs.pdf
conviniently leaves that part out. :(

With respect to your issues with TS->MKV are you sure its not due to the modifications that the MKV muxer does to the AVC stream?

[honai]

You need to have at least one audio stream for the playback not to be choppy. It is the sound card that provides the master clock during playback of a/v on a PC.

Are you sure about that? I've converted a number of VC-1 streams to .mkv/.ts with no audio streams in the container, and playback was smooth at the specified framerate, even at very high bitrate.

Haali needs to update his splitter to support the blu-ray/avchd standard of storing sound in TS.

I should have mentioned that the same happens when I feed a .ts w/o any sound to Haali Media Splitter - it just stops after a few seconds, so it's likely not related to sound issues.

I somehow doubt that the issues you encounter with the elementary streams are due to xport/tsremux as demuxing from TS is very straight forward.

My educated guess is that some AVC elementary streams demuxed with xport/TsRemux miss or have erroneous headers since mkvmerge itself properly muxes a number of AVC streams from other sources.

.m2ts -> .ts = video is displayed correctly
.m2ts -> .es -> mkv = video display is messed up

EDIT:

Headers and maybe other data is removed from the stream and stored in a separate coded specific blob. It's very hard to reconstruct the original streams from the MKV modified ones.

Yes, but as I noted above mkvmerge handles many AVC elementary streams just fine, same as Haali Matroska Muxer. For instance:

- captures from Premiere HD
- captures from Sky HD
- X-Men 3 on Blu-ray

The demuxed streams play fine w/ or w/o audio tracks.

[The_Keymaker]

@honai,

I've had the same experience when feeding .ts files with no audio into GDSmux: the muxing just stops somewhere between 2% and 14% completion.

@dmz01,

My .ts files are obtained by demuxing using TSremux. The source files are .EVO files rebuilt from EVOdemux. By the way for some reason, .ts files produced by your program (TSremux) only contains video, not audio. Even when I select the audio track for inclusion.

The ONLY h.264 movie I've had success at muxing into MKV was the HD-DVD Feast. I used the EVOdemux to combine the EVOs, demuxed the audio track and muxed them successfully using GDSmux and MKVmerge.


I would really like to find a soulution to this problem, as I have several h.264 HD-DVDs sitting on my hard drive.

The_Keymaker

[honai]

With respect to your issues with TS->MKV are you sure its not due to the modifications that the MKV muxer does to the AVC stream?

Could very well be.

I've done a little more research.

The potc-cotbp-sample.ts from above won't properly remux into MKV with Haali Matroska Muxer, the remuxing process in GraphEdit simply stops after a few seconds.

Here's the headers as displayed by MPC/CoreAVC:


VIDEOINFOHEADER2:
dwInterlaceFlags: 0x00000000
dwCopyProtectFlags: 0x00000000
dwPictAspectRatioX: 16
dwPictAspectRatioY: 9
dwControlFlags: 0x00000000
dwReserved2: 0x00000000

MPEG2VIDEOINFO:
dwStartTimeCode: 0
cbSequenceHeader: 117
dwProfile: 0x00000064
dwLevel: 0x00000029
dwFlags: 0x00000004

BITMAPINFOHEADER:
biSize: 40
biWidth: 1920
biHeight: 1080
biPlanes: 1
biBitCount: 24
biCompression: avc1
biSizeImage: 0
biXPelsPerMeter: 1
biYPelsPerMeter: 1
biClrUsed: 0
biClrImportant: 0


However, if I use the progress slider in GraphEdit and advance to the position where the opening title ends, and the camera slowly pans through the mist to the girl on the ship ... the remuxing works!

Here's the headers as written by Haali Matroska Muxer in the resulting video stream:


VIDEOINFOHEADER2:
dwInterlaceFlags: 0x00000000
dwCopyProtectFlags: 0x00000000
dwPictAspectRatioX: 1920
dwPictAspectRatioY: 1080
dwControlFlags: 0x00000000
dwReserved2: 0x00000000

MPEG2VIDEOINFO:
dwStartTimeCode: 0
cbSequenceHeader: 109
dwProfile: 0x00000064
dwLevel: 0x00000029
dwFlags: 0x00000004

BITMAPINFOHEADER:
biSize: 40
biWidth: 1920
biHeight: 1080
biPlanes: 1
biBitCount: 24
biCompression: avc1
biSizeImage: 0
biXPelsPerMeter: 1
biYPelsPerMeter: 1
biClrUsed: 0
biClrImportant: 0


Turns out the cbSequenceHeader is a different one. In the original .m2ts and remuxed to .ts it's 117, in the working .mkv it's 109. Remuxing the elementary stream with xport->mkvmerge from .m2ts->.mkv also yields 117 in the .mkv, and that file won't play.

Could the problem lie in the possibility that the opening credits and the main feature have a different structure (cbSequenceHeader=117 for opening credits, cbSequenceHeader=109 for main feature), and that in the mastering process the studio concatenated two different AVC streams (e.g. to accomodate for localized titles) which throw Haali Matroska Muxer off?

EDIT:

Just found this:

http://www.bunkus.org/videotools/mkvtoolnix/doc/ChangeLog


2007-01-11 Moritz Bunkus <moritz@bunkus.org>

* mkvmerge: bug fix: MPEG-1/2 video: The sequence and GOP headers
are not removed from the bitstream anymore. This should fix the
blockiness if the sequence headers change mid-stream. Fix for
Bugzilla bug #167.


Perhaps mkvmerge does remove the sequence header from AVC streams after all.

EDIT:

Did another test with Elecard XMuxer Pro. The AVC ES muxed just fine into TS, so the problems seems to lie in the way mkvmerge/Haali handle the AVC ES.

[Haali]

The big issue for me with the MKV container in general, is that the elementary streams are modified prior to storage inside the medium. Headers and maybe other data is removed from the stream and stored in a separate coded specific blob. It's very hard to reconstruct the original streams from the MKV modified ones. This is why for example MKV->TS in TsRemux only supports MPEG2 and AC3 for now. The issue with AVC/h.264 is even more pronounced because no public documentation exist on how an AVC stream is modified. The only public information from Haali:
http://haali.cs.msu.ru/mkv/codecs.pdf
conviniently leaves that part out. :(

Hehe, this is not documented because it's not actually modified in any way. Storage of h.264 in mkv is copied verbatim from mp4, CodecPrivate == codec init data from stsd, etc.

[vlada]

I think I have very similar problem with a file from Panasonic AVCHD camcorder (sample (http://www.zshare.net/download/307776947097cb/)). Remuxing using TSRemux / MKVToolnix or MP4Box to MP4 or MKV corrupts the file. But remuxing to MKV in Graphedit using Haali's filters works fine. Could anyone confirm this?

Btw. the video is interlaced/PAFF so all libavcodec based decoders will crash trying to decode it.

[SealTooGreat]

The big issue for me with the MKV container in general, is that the elementary streams are modified prior to storage inside the medium. Headers and maybe other data is removed from the stream and stored in a separate coded specific blob. It's very hard to reconstruct the original streams from the MKV modified ones.

Take a look at MKVextractGUI.

[foxyshadis]

I have no experience with hddvd, but can you demux a raw avc stream from the evo? If so you can mux that to mp4 using mp4box (YAMB is a good gui) and then mux the resulting mp4 with MKVmerge.

mkvmerge has supported import of raw avc streams for a few months now.

[Mosu]

Yes, mkvmerge removes the sequence headers and stores the first ones that it finds in the CodecPrivate. Yes, I know that this a problem with changing sequence headers in the middle of the stream, and yes, I plan to change that someday. That being said, I don't spend any time coding mkvtoolnix at the moment, so your guess is as good as mine when mkvmerge will handle such streams correctly.

[Mosu]

Could the problem lie in the possibility that the opening credits and the main feature have a different structure (cbSequenceHeader=117 for opening credits, cbSequenceHeader=109 for main feature), and that in the mastering process the studio concatenated two different AVC streams (e.g. to accomodate for localized titles) which throw Haali Matroska Muxer off?

That is indeed the case. Each PPS and SPS info header as an ID, and mkvmerge only keeps the first for each ID in the CodecPrivate. But there seem to be a lot of files for which the contents of the PPS and SPS change for the same ID. I'm pretty sure that this isn't what the inventors of h.264 had in mind, but it is reality, and mkvmerge should deal with that properly. It doesn't at the moment.

[vlada]

Could someone have a look at my sample, please? I don't think this is the same case (change of sequence headers), because the file is a recording from a camcorder.

Also I'm experiencing the same problems with MP4 (created by MP4Box). What about your samples?

[honai]

@mosu

Thanks for clarifying that. So for the time being the only work-around seems to be that we split the source, feed all parts independently to mkvmerge, and link the resulting .mkv files.

Do you know of a tool that could help in identifying when PPS/SPS change in the stream?

[Mosu]

Sorry, but my knowledge of tools dealing with h.264 files is very, very limited, so I cannot remmend anything.

[CruNcher]

Hehe if we didn't know that the Segmentation Encoding the studios do is the problem , some could realy think this is somekind of protection ala macrovision at least it works out for the studios as we see here we have mighty probs muxing such streams into other containers like mp4/mkv, but anyway does it really matter much because most of us reencode it into a complete new file so only that the parser is able to catch this is important imho, but yeah i had such problems too and funny thing is useing such streams benchmarking in timecodec makes no problems only real playback does and seeking resulting in those nice green blocks and @ the beginning even crashed mplayer (segfault with original .m2ts h.264 streams when trying to seek, dunno about now :P ) ;)

[Okraml]

... but anyway does it really matter much because most of us reencode it into a complete new file ...

Why?
I wouldn't reencode a AVC or VC1 file cause the video can only get worse or much bigger (AVC/VC1 have nearly the same Quality/Size ratio as X264). I would only reencode MPEG2 to get a much smaller size (if you transcode to X264). But MPEG2 (and VC1) make(s) no problems with remuxing. The only problems are AVC videostreams.

:-) Okraml

[Mosu]

Ok guys. Here's a mkvtoolnix build that will not remove the SPS and PPS NALUs. I've tried it with the few files that I have, and it indeed works with a file that prior mkvtoolnix versions did not work with. Please give it a try:

http://www.bunkus.org/videotools/mkvtoolnix/win32/pre/mkvtoolnix-unicode-2.0.2-build20070815-1.rar

[honai]

Wow, thanks Mosu! Works flawlessly so far, I just successfully muxed the AVC ES stream from Casino Royale (where the previous builds didn't work), no errors or problems, plays perfectly!

EDIT:

... in MPC, Haali Media Splitter, CoreAVC 1.5.0. Don't know how other configurations behave in respect to those NALUs being present mid-stream.

EDIT:

Confirmed to work with Casino Royale (Blu-ray) and Black Snake Moan (HD-DVD). So I guess the problems with AVC from HD discs are solved. Congrats mosu!

[Okraml]

Ok guys. Here's a mkvtoolnix build that will not remove the SPS and PPS NALUs. I've tried it with the few files that I have, and it indeed works with a file that prior mkvtoolnix versions did not work with. Please give it a try:

http://www.bunkus.org/videotools/mkvtoolnix/win32/pre/mkvtoolnix-unicode-2.0.2-build20070815-1.rar

http://www.r-treff.de/vbb/images/smilies/standard/bow.gif
Exactly what was needed.

:-) Okraml

[DreckSoft]

Confirmed to work with The Rock. I couldn't mux this one with any other method. THANKS!

[homerpez]

I'm not having so much luck here...

D/L the new build of mkvmerge in this thread, experimenting with just a straight backup of 300 on Blu Ray (since I have kids... fingers... you know.) I usually just back up to 720p since my TV is only 720p anyway, but I decided I would try my first "full quality" backup.

I first tried to backup by re-encoding it to 1080p and the size of a DVD-9... details looked great, but backgrounds and clouds were VERY blocky. So this was a no-go.

I then decided, why not just demux the VC1 stream and audio, and save it this way?

Using tsremux, I saved the Dolby 5.1 as ac3, and the VC1 stream as H264.... then AVC.... then 264... then (you get the idea). The point here is, none of them would load at all in MKVMERGE. I tried renaming the video stream everything under the sun, including the supported formats, and they refuse to load.

I get the "Return Code 2: video has unknown type" error, every single time I try and add the stream (except when I name it VC1, then it just goes in as an attachment).

I have tested the demuxed VC1 stream in PowerDVD, and it plays perfectly. MKVMerge just doesn't seem to know what to do with it.

Before anyone answers:

- Yes, I do understand I can simply remux to a .ts or .m2ts stream directly using TSremux. I just would LIKE to use MKV because it's always given me consistent results (I have 2 machines - one plays m2ts, one doesnt. Both have identical drivers and programs installed).

- Yes, I do know I can re-encode the video stream, however, I already did (see above) and the results aren't to my liking at 1080p. I could go to 720p, but...

Is there anything special I need to be doing when I demux the video here?

[Mosu]

mkvmerge does not support VC1 at the moment. mkvmerge does not care about file names or file name extensions because it always reads the start of the file and determines its type from the content.

[Okraml]

@homerpez
Just use gdsmux with the m2ts file to create a .mkv file.
This resulting .mkv can then be opened in mkvmerge (for muxing with other audiotracks or to splitt).

:-) Okraml

[DreckSoft]

Could it be that MKVToolNix doesn't handle interlaced PAFF in H264 as well? See here:
http://forum.doom9.org/showthread.php?p=1037055#post1037055

[juffi]

@honai,

How do you remux Premiere HD?

I have tried to remux Anixe HD and some other 1080i streams to .mkv using TSRemux and TSConverter for demuxing and mkvmerge for remuxing. Resulting .mkv plays video in half speed and audio normally (of course absolutely no sync).

Any advice appreciated.