Another AVCHD/Blu-Ray demuxer

[clark15b]

I'm proud to present my own M2TS files demuxer taken from AVCHD camera and Blu-ray disks. Utility have a number of advantages and features when you working with AVCHD materials.

There is a major issue: joining multiple scenes located in different M2TS (MTS)/TS files into one file with audio/vidio sync (optional).

After brief topic examination it came to me that such thing can be done by eac3to, tsMuxer or simple 'copy /b' command in Windows and 'cat' in *nix-es, just addition one file in the end of another.

But in practice this is not good idea. The question is each M2TS file audio and video may be varied by length. This is urgent for the NTSC with fractional FPS, for example 29.97. Audio length (AC3 fps) and video frames (H264 JVT NAL) could not be variable, consequently there is no way to combine they at the end of the TS stream. Easiest way is to adjust video to audio, in this case variable fps is needed. In result we have audio stream 1 msec longer then video. It is not hard to calculate that for the 100 scenes we will get tremendous audio delay for 100 msec at the end.

When playing M2TS files one after one player begins from start and there are no visible synchronization issues. But when we join files, extracting elementary streams and muxing to MKV we are dropping information for synchronization completely. Muxer (mkvmerge) have no idea about initial TS stream properties and therefore sets its own time marks, supposing video and audio synchronized (I would remind you h.264 stream doesn't contain any metadata about FPS and so on).

Here the program features:

* runs either by Windows, Linux and FreeBSD;
* completely opensource , GPLv2;
* understands M2TS and TS file formats (type detetection by magic sequence);
* batch processing;
* joining multiple scenes;
* demux to ES or PES streams;
* extensions for the output files would be set depending of elementary stream;
* takes files list and/or directories list as command line argument (it is possible to specify a path to files for example private/avchd/bdmv/stream, program will take all appropriate files by itself);
* extracts shooting date and time of the each clip from mpls files in camera (private/avchd/bdmv/playlist/*.mpl);
* extracts elementary streams all programs in TS container or selected by number;
* makes files with time codes for appropriate elementary streams synchronizationl (mkvmerge timecode format v2);
* determining h.264 and AC3 elementary stream FPS;
* determining audio delay after video in each file and brings appropriate corrections into time codes (urgent for Blu-ray, you not needed to calculate and specify delay by yourself when muxing into MKV);
* trying to align elementary streams endings inside each scene;
* produces file with scenes for saving information about initial material dividing into multiple M2TS files. Scene names gives after file names or shooting time/date from the playlist (mkvmerge xml format);
* produces SRT file with subtitles for saving information about shooting time/date from the playlist;
* shows invoking mkvmerge command for muxing produced material into MKV;
* support MPEG2 Video, MPEG2 Audio, H.264, VC-1, AC3, L-PCM elementary stream types;
* parses and shows information about TS stream without elementary streams extraction (shows streams types. length, offset, and fps);
* dump TS structure (M2TS timecodes, PTS/DTS ...).

Unfortunately there is only console version available yet. It was tested against AVCHD cameras by Sony, Canon, Panasonic, some Blu-ray disks and broadcast TS.

Competitors disadvantages:
* xport - Unable to join files and can't take TS from the standard input. Output file names not reflecting content type;
* eac3to - No source code available, only Windows;
* tsMuxer - Also no source code, not so useful console version, joining files in GUI version is not comfortable.

And no one of them able to form files for proper streams synchronization for mkvmerge, no ability to generate file with scenes list, and no shooting time/date extraction.

I suppose my program may be an alternative to xport.

GUI Windows setup (win32): http://tsdemuxer.googlecode.com/files/tsdemuxer_setup.exe
Windows binaries (win32): http://tsdemuxer.googlecode.com/files/tsdemux.zip
Linux binaries (gcc4, static): http://tsdemuxer.googlecode.com/files/tsdemux.tar.gz
Sources: http://code.google.com/p/tsdemuxer/source/browse
Project home page: http://code.google.com/p/tsdemuxer


Simple invocation form: 'tsdemux.exe -m -j -d x:\private\avchd\bdmv\stream' or 'tsdemux.exe -j 00001.mts 00002.mts'. Result will be in five files in current directory: video stream , audio stream, tmc-file for audio, tmc-file for video, xml-file with scenes. Then you should pass all of these to mkvmerge.

Command line options:
tsdemux.exe [-d src] [-l mpls] [-o dst] [-c channel] [-u] [-j] [-m] [-z] [-p] [-e mode] [-v] *.ts|*.m2ts ...

-d discard all TS/M2TS/MTS files from specified directory (use it multiple times for different directories, files will be sorted ascending). For the AVCHD/Blu-ray it should be 'private/avchd/bdmv/stream'.
-l path to AVCHD/Blu-ray playlist mpl/mpls file (usually it is 'private/avchd/bdmv/playlist/*.mpls'). If this option set then for the names of scenes shooting time/date will be used also.
-o redirect output to another directory
-c channel number in TS stream (if not specified then all streams from all programs will be extracted, but scene file for mkvmerge will not generated).
-u demux unknown streams
-j join elementary streams
-m shows mkvmerge invoking options for muxing results in MKV.
-z demux to PES streams (instead of elementary streams).
-p deny elementary streams extraction, only analyze with information output.
-e dump TS structure to STDOUT (mode=1: dump M2TS timecodes, mode=2: dump PTS/DTS, mode=3: human readable PTS/DTS dump)
-v turn on verbose output

Valid files extensions: m2ts, mts, ts.
Output files could be with extensions: sup, m2v, 264, vc1, ac3, m2a, pcm. Files with time codes for each elementary stream for mkvmerge (tmc) and mkvmerge scene file (chapter.xml) will be generated as well.

Send bugreports to comments, please :)

[neuron2]

Very interesting, thank you for your work and contribution. Is there a GUI in your plans?

[clark15b]

Sure, I will take care about GUI if community will show interest in project.

[neuron2]

Well, it would certainly make a big advance over xport, whereas right now, the delta is relatively small. I wish I had time to help with it.

[clark15b]

Currently it is of importance for me to test program functions with different signal sources (different AVCHD camcorders, Blu-ray disks with different codecs). Will be very gratefull for help with this. I will work on GUI in the mean time.

[deank]

A lot of people willing to help you in testing will have troubles without a GUI. You can get test results faster if you compile one for them.

I don't have AVCHD camcoder, but I have a lot of samples and will do some tests and will let you know.

Dean

[noee]

Thank you for the tool, I will begin testing with my AVCHD (Canon HF100) files promptly. I, too, would welcome a GUI and would be willing to help test that as well.

[noee]

I have tried a few AVCHD files and I am having an issue with the resultant MKV. Whether I mux it myself with MkvmergGUI or your command line, the resultant MKV file takes a very long time to open and it is solid disk activity while waiting for it to open.

I'm using Haali Media Splitter and the 2.7.0 MKVToolnix and I've tried opening it with MPC-HC and jRiver MC13, both exhibit the same extended time to open the MKV. Once open, playback is as expected.

HTH

[clark15b]

Thanks for your interest, I will make GUI soon.

noee
I do not think that a problem in muxer and MKV, I have no problem with reproduction of original M2TS files and resulting MKV on the computer. I use Haali Media Splitter, CoreAVC and MediaPlayer Classic.

[turbojet]

Firet impressions is it's really fast, only seconds slower then tsmuxer. eac3to and xport are slow in comparison for some reason.

If it had mkv/mp4 input, avchd output, sup/srt input/output, and multiple audio support I'd be real interested in it mainly because of the cli. Do you have any plans for any of these?

I tried it on a retail VC-1 BD and found some problems.
- detected the 23.976 VC-1 stream as 29.97 fps, tsmuxer and eac3to detect 23.976.
- when I used -l with BDMV\PLAYLIST directory on same BD and got 'invalid file format'
- -d BDMV\PLAYLIST\00000.mpls (main movie) crashed tsdemux
- chapters.xml only contained 3 chapters starting at 3 hours, the movie is only 90 minutes long
- After demuxing the main movie m2ts on same BD it gave these errors:
** bad last PTS in stream 0xfd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xfd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xfd ('vc1'), length 5212595.84ms (
+1244.71ms)
** align stream 0xfd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xfd ('ac3'), length 5212256.00ms (
+904.87ms)
** align stream 0xfd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xfd ('ac3'), length 5212256.00ms (
+904.87ms)
** align stream 0xfd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xfd ('ac3'), length 5212256.00ms (
+904.87ms)
** align stream 0xfd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xfd ('bin'), length 5347985.00ms (
+136633.87ms)
** align stream 0xfd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xfd ('ac3'), length 5212256.00ms (
+904.87ms)
** align stream 0xfd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xbd ('bin'), length 14257757.24ms
(+9104356.30ms), delay 57950.19ms
** align stream 0xbd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xbd ('bin'), length 10041529.73ms
(+4926292.13ms), delay 96113.53ms
** align stream 0xbd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xbd ('bin'), length 10256699.44ms
(+5112182.63ms), delay 66834.32ms
** align stream 0xbd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xbd ('bin'), length 4963921.22ms (
+2.01ms), delay 247431.92ms
** align stream 0xbd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xbd ('bin'), length 4963919.87ms,
delay 247431.27ms
** align stream 0xbd length to 14315708.00ms
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00001.m2ts: channel 1, stream 0xfd ('vc1'), length 11970.28ms (+1
0288.73ms)
K:\BDMV\STREAM\00001.m2ts: channel 1, stream 0xfd ('ac3'), length 12000.00ms (+1
0318.46ms)
K:\BDMV\STREAM\00001.m2ts: channel 1, stream 0xbd ('bin'), length 745.34ms, dela
y 936.20ms
** bad last PTS in stream 0xfd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00002.m2ts: channel 1, stream 0xfd ('vc1'), length 11887.67ms
K:\BDMV\STREAM\00002.m2ts: channel 1, stream 0xfd ('ac3'), length 11904.00ms (+1
6.33ms)
K:\BDMV\STREAM\00003.m2ts: channel 1, stream 0xfd ('vc1'), length 5872.53ms
** align stream 0xfd length to 5873.00ms
** bad last PTS in stream 0xfd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00004.m2ts: channel 1, stream 0xfd ('vc1'), length 23274.80ms
** align stream 0xfd length to 23275.00ms
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00005.m2ts: channel 1, stream 0xbd ('bin'), length 1363.13ms
** align stream 0xbd length to 1364.00ms
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00006.m2ts: channel 1, stream 0xbd ('bin'), length 1386.13ms
** align stream 0xbd length to 1387.00ms
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xfd ('vc1'), length 10010.00ms (+8
386.79ms)
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xfd ('ac3'), length 10016.00ms (+8
392.79ms)
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xfd ('ac3'), length 10016.00ms (+8
392.79ms)
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xfd ('ac3'), length 10016.00ms (+8
392.79ms)
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xfd ('ac3'), length 10016.00ms (+8
392.79ms)
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xfd ('ac3'), length 10016.00ms (+8
392.79ms)
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xbd ('bin'), length 687.01ms, dela
y 936.20ms
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xbd ('bin'), length 687.01ms, dela
y 936.20ms
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xbd ('bin'), length 687.01ms, dela
y 936.20ms
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xbd ('bin'), length 687.01ms, dela
y 936.20ms
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xbd ('bin'), length 687.01ms, dela
y 936.20ms
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00008.m2ts: channel 1, stream 0xbd ('bin'), length 207.40ms
** align stream 0xbd length to 208.00ms
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00009.m2ts: channel 1, stream 0xbd ('bin'), length 207.40ms
** align stream 0xbd length to 208.00ms

[clark15b]

Unfortunately I didn't plan to do reverse conversion. It is completely different and more tricky task.

[turbojet]

Oh ok shucks I guess I'll have to keep trying to get through to tsmuxer developer to add some simple cli. Thanks anyways.

After muxing the demuxed files to m2ts/mkv files it chose the last audio track to demux and it needs about +200 ms delay.

[clark15b]

turbojet
Thank you! Do you use Windows or Unix?
Could you please send me BDMV/PLAYLIST (at least several files)?
Please specify command line which you were using.
Could you please send a part of main movie M2TS file?

[turbojet]

This was windows, I could test linux if you think it would make a difference.

Here's the files you requested (http://www.sendspace.com/file/u0lzbq).

I used tsdemux -d K:\BDMV\STREAM\00000.m2ts to demux

[deank]

Hi, clark15b!

I'm having a problem with the -l switch. Every time I try to use it, tsdemux crashes.

[clark15b]

turbojet
Thank you, try 'tsdemux K:\BDMV\STREAM\00000.m2ts' for demux.

deank
Whether you have the description of mpls files?

[turbojet]

turbojet
Thank you, try 'tsdemux K:\BDMV\STREAM\00000.m2ts' for demux.


It detected 23.976 fps and there was no chapters in the xml other than that the same output.

[clark15b]

turbojet
I have correctly understood, now it has correctly detect fps?
One scene will be in xml if one initial M2TS file is specified.

[deank]

clark, I'd suggest you this:


50 if(memcmp(ptr,"MPLS0100",8))
51 return -1;


you can either make it ..."MPLS0",5... or check for MPLS0200, too.

and


58 unsigned char* pp=ptr+68;
60 if(pp+num*82>ptr+len)
...
72 pp+=82;

This can lead you to troubles. ptr[68] and [69] keep the length of the current segment, so it is correct to increment pp in line 72 not by 82 but with the value in 68/69.

Then for the next cycle use pp as the base for other chapter calculations.

[turbojet]

turbojet
I have correctly understood, now it has correctly detect fps?
One scene will be in xml if one initial M2TS file is specified.

Yes but I think on the first try it was reading 00007.m2ts which is the last file and it is indeed 29.97 fps

[clark15b]

deank
Thanks, this is true way. I am in the progress.

turbojet
Similar on the truth, tool show last (current) fps. If you mix different sources you will see fps from last movie.

[clark15b]

deank
Some mpls files have strange length prefix before '00000' clip name (red highlight) - 256, but real section length - 266. Other sections is correct.

Besides, the sections counter (blue highlight) contains 1 when we have 3 sections. Near to it there is one more number (yellow highlight) - 2. In the sum it turns out 3.
Usually the first contains actual quantity of sections, and the second 0.

I in confusion.

This is hex dump of turbojet`s example:

00000000 4D 50 4C 53 │ 30 32 30 30 │ 00 00 00 3A │ 00 00 01 96 │ 00 00 00 00 │ 00 00 00 00 MPLS0200...:............
00000018 00 00 00 00 │ 00 00 00 00 │ 00 00 00 00 │ 00 00 00 00 │ 00 00 00 0E │ 00 01 00 00 ........................
00000030 00 00 01 0F │ 40 00 00 00 │ 40 00 00 00 │ 01 58 00 00 │ 00 01 00 02 │ 01 00 30 30 ....@...@....X........00
00000048 30 30 30 4D │ 32 54 53 00 │ 01 00 00 07 │ FF F8 0E 02 │ F7 8F 00 00 │ 00 00 00 00 000M2TS.....ÿø..÷.......
00000060 00 00 00 00 │ 00 00 00 DE │ 00 00 01 05 │ 05 02 00 00 │ 00 00 00 00 │ 00 00 09 01 .......Þ................
00000078 10 11 00 00 │ 00 00 00 00 │ 05 EA 61 00 │ 00 00 09 01 │ 11 00 00 00 │ 00 00 00 00 .........êa.............
00000090 05 81 61 65 │ 6E 67 09 01 │ 11 03 00 00 │ 00 00 00 00 │ 05 83 61 65 │ 6E 67 09 01 ..aeng............aeng..
000000A8 11 04 00 00 │ 00 00 00 00 │ 05 81 31 65 │ 6E 67 09 01 │ 11 01 00 00 │ 00 00 00 00 ..........1eng..........
000000C0 05 81 61 66 │ 72 61 09 01 │ 11 02 00 00 │ 00 00 00 00 │ 05 81 61 73 │ 70 61 09 01 ..afra............aspa..
000000D8 12 00 00 00 │ 00 00 00 00 │ 05 90 65 6E │ 67 00 09 01 │ 12 01 00 00 │ 00 00 00 00 ..........eng...........
000000F0 05 90 66 72 │ 61 00 09 01 │ 12 02 00 00 │ 00 00 00 00 │ 05 90 73 70 │ 61 00 09 01 ..fra.............spa...
00000108 12 03 00 00 │ 00 00 00 00 │ 05 90 66 72 │ 61 00 09 01 │ 12 04 00 00 │ 00 00 00 00 ..........fra...........
00000120 05 90 73 70 │ 61 00 09 02 │ 00 00 14 00 │ 00 00 00 00 │ 05 91 65 6E │ 67 00 09 02 ..spa.............eng...
00000138 01 00 14 00 │ 00 00 00 00 │ 05 91 66 72 │ 61 00 00 00 │ 00 24 00 03 │ 00 00 00 01 ..........fra....$......
00000150 00 1C 30 30 │ 30 30 35 4D │ 32 54 53 00 │ 00 00 02 00 │ 00 07 FF F8 │ 00 08 C5 18 ..00005M2TS.......ÿø..Å.
00000168 00 00 00 00 │ 00 00 00 00 │ 00 24 00 03 │ 00 00 00 01 │ 00 1C 30 30 │ 30 30 36 4D .........$........00006M
00000180 32 54 53 00 │ 00 00 02 00 │ 00 07 FF F8 │ 00 08 C5 35 │ 00 00 00 00 │ 00 00 00 00 2TS.......ÿø..Å5........
00000198 01 1A 00 14 │ 00 01 00 00 │ 00 07 FF F8 │ FF FF 00 00 │ 00 00 00 01 │ 00 00 00 EB ..........ÿøÿÿ.........ë
000001B0 29 C4 FF FF │ 00 00 00 00 │ 00 01 00 00 │ 01 B6 62 65 │ FF FF 00 00 │ 00 00 00 01 )Äÿÿ.........¶beÿÿ......
000001C8 00 00 02 14 │ 17 34 FF FF │ 00 00 00 00 │ 00 01 00 00 │ 03 03 A0 3B │ FF FF 00 00 .....4ÿÿ..........*;ÿÿ..
000001E0 00 00 00 01 │ 00 00 03 C4 │ 6C 2D FF FF │ 00 00 00 00 │ 00 01 00 00 │ 04 A5 2E 20 .......Äl-ÿÿ.........¥.
000001F8 FF FF 00 00 │ 00 00 00 01 │ 00 00 05 34 │ 9A 7E FF FF │ 00 00 00 00 │ 00 01 00 00 ÿÿ.........4.~ÿÿ........
00000210 05 F0 3E C3 │ FF FF 00 00 │ 00 00 00 01 │ 00 00 06 86 │ 15 65 FF FF │ 00 00 00 00 .ð>Ãÿÿ...........eÿÿ....
00000228 00 01 00 00 │ 07 7E 35 E2 │ FF FF 00 00 │ 00 00 00 01 │ 00 00 08 43 │ EE DB FF FF .....~5âÿÿ.........CîÛÿÿ
00000240 00 00 00 00 │ 00 01 00 00 │ 08 BB 12 13 │ FF FF 00 00 │ 00 00 00 01 │ 00 00 09 3C .........»..ÿÿ.........<
00000258 2C AC FF FF │ 00 00 00 00 │ 00 01 00 00 │ 0A 1A A4 19 │ FF FF 00 00 │ 00 00 00 01 ,¬ÿÿ..........¤.ÿÿ......
00000270 00 00 0B 04 │ AD 77 FF FF │ 00 00 00 00 │ 00 01 00 00 │ 0C 19 36 D8 │ FF FF 00 00 ....*wÿÿ..........6Øÿÿ..
00000288 00 00 00 01 │ 00 00 0C B2 │ D5 3E FF FF │ 00 00 00 00 │ 00 01 00 00 │ 0D 2C 7D A3 .......²Õ>ÿÿ.........,}£
000002A0 FF FF 00 00 │ 00 00 00 01 │ 00 00 0E 02 │ CB 92 FF FF │ 00 00 00 00 │ ÿÿ..........Ë.ÿÿ....

[clark15b]

deank
I commited last version to googlecode (sources only). Try now if you can build it independently.

[deank]

There is nothing wrong with the numbers and everything is what it should be - it is just something you've not seen in AVCHD.

All lengths in red are correct, you're just miscalculating because you're ignoring other data. Also - values in blue and yellow mean different things.

Contact me via email dean[-at-]deanbg.com or skype (deanrk).

[clark15b]

UPDATE is available.

Version 1.02

- fix in mpls parser, thanks to deank
- fix PMT parser
- use Blu-Ray/AVCHD mpls as playlist if specified (example: './tsdemux -l private/avchd/bdmv/playlist/00000.mpl -d private/avchd/bdmv/stream/ -o out -m')
- display pid and track number
- verbose output switch
- fix win32 crash if invalid directory specified

[Kurtnoise]

Does it work when the files are read directly from the disc ?

I tried this (on windows system) :
tsdemux -p "G:\BDMV\STREAM\00000.m2ts"
but tsdemux doesn't parse anything (the disc is decrypted w/ AnyDVD HD). If I put my file on the hard drive, all is fine. What's wrong ?

[clark15b]

Very strange, any difference should not be.

[espero]

@clark15b:
Thank you very much for your great prog. THE ONLY ONE which takes care about audio / video sync (while joining mts from camcorders).
I used your prog with success but... I do not need mkv as output. Please let me know if there is a possibility to mux joined video and audio files directly to mts? I can of course use tsmuxer (instead of mkv merge) or other program but they dont care about .tmc files generated by your great programm and than audio and video are still out of sync.

[clark15b]

espero
Hmm.. you right, tsmuxer can`t use .tmc files. I can try write tool special for join .mts files. It will just override timecodes.

[mic64]

Hi

I was looking for a program, which could be run natively under Linux, so thank you for your effort.
But somehow I cant get this to work with anything.
Under Linux the -l switch just does nothing.
Pointing tsdemux directly to a m2ts file, gives me "Value too large for defined data type"

Can this version already be used under Linux?

[clark15b]

mic64
Yes, this program is initially developed for linux.
Take sources from svn and build it:
svn checkout http://tsdemuxer.googlecode.com/svn/trunk/
make

[deank]

clark,

I added your tool to multiAVCHD, so people with camcoders can input STREAM folders directly.

A lot of tests were done the last few days:

1) If I use *.264 + *.ac3 it all works fine
2) If I create MKV file from these 2 and then convert to m2ts - output is distorted (frame artifacts, framerate changes, etc).

(if you play the MKV file - it is okay, but all breaks after it is converted to m2ts by tsmuxer)

So I decided to use *.ac3+*.264+chapters.xml and load them as a title in multiAVCHD. It now works fine for a lot of users as it seems.

Thanks again!

Dean

[clark15b]

deank
I am very glad that have helped you!

[deank]

Well, you helped a lot of happy users out there.

I don't have a camcoder so I can't use the feature myself, but a lot of people appreciate your contribution!

Is there a chance for you to look at a sample file (2 .ts files in fact).

This is the scenario:

1) 1 original .ts file
2) File is processed by some ts-cutter/editor to remove commercials
3) Result is 2 .ts files
4) If played alone - both files have perfect video/audio sync.
5) If joined with your application (or tsmuxer) second part gets out of sync.

I'll attach the file in a minute (40MB) if you have time/will to check it.

Dean

(it was send to me by fellow russian/moscow guy) :)

Here is the link: pr.exe (http://rapidshare.com/files/243366074/pr.exe) - 7zip self extractable - contains 2 .ts files.

[clark15b]

ok, i will look next week (i leave for the weekend, tomorrow at us a holiday).

[clark15b]

This is the scenario:

1) 1 original .ts file
2) File is processed by some ts-cutter/editor to remove commercials
3) Result is 2 .ts files
4) If played alone - both files have perfect video/audio sync.
5) If joined with your application (or tsmuxer) second part gets out of sync.


I have understood in what business and I can think up something.
They wish to use what container... MKV or TS?

[deank]

Well they use .ts or .m2ts.

The best would be to join multiple .ts or multiple .m2ts without this audio asyncronization.

:) Dean

p.s. Thanks for looking into it.

[clark15b]

Give me 2-3 days, I will write tool for correct join multiple ts/m2ts to one ts/m2ts file.
If they want to replace the container there will be problems.

[clark15b]

mic64
Try './tsdemux - < filename.m2ts'.

[clark15b]

deank

This is output of my new demuxer:

../samples/ts2/PRINCE-01.ts: TS stream detected (packet length=188)
pid=4113 (0x1011), ch=1, id=1, type=0x1b (264), stream=0xe0, fps=25.00, len=88800ms (2217/2281 frames) : =79ms=>[@@@@@]
pid=4352 (0x1100), ch=1, id=2, type=0x81 (ac3), stream=0xfd, fps=31.25, len=88640ms (2770 frames) : [@@@@@]<=239ms=
../samples/ts2/PRINCE-02.ts: TS stream detected (packet length=188)
pid=4113 (0x1011), ch=1, id=1, type=0x1b (264), stream=0xe0, fps=25.00, len=122400ms (3058/3058 frames) : =111ms=>[@@@@@]
pid=4352 (0x1100), ch=1, id=2, type=0x81 (ac3), stream=0xfd, fps=31.25, len=122272ms (3821 frames) : [@@@@@]<=239ms=

Video track in first clip have 2217 TS frames, but JVT NAL (h.264 ES) contain 2281 frames :(

Besides I see displacement of the beginning and the end of streams (blue highlight).

You can`t simple merge ES of this clips.

I try prepare correct MKV, then I will think
over TS.

[clark15b]

deank

The first part has variable fps, it is a principal cause of out of sync.

My tool can build timecodes for fixed fps now.

Here that has turned out: http://rapidshare.com/files/247838496/PRINCE-01.track_4113.rar.html
First part has out of sync, second part without audio asyncronization.

Now i try join this to one ts with variable fps....mkvmerge does not approach (it is possible but difficult) :(

[clark15b]

UPDATE is available.

Version 1.50

- new demux engine (parse difficult streams)
- new timecodes generator for mkvmerge (+ h.264, AC3 ES parser)
- allow large files in Linux
- dump TS structure
- batch processing (without join clips)
- PES output (instead of elementary streams)
- a lot of changes

Now I work over join of several TS/M2TS streams in one.

[clark15b]

GUI here: http://tsdemuxer.googlecode.com/files/tsdemuxer_setup.exe

Download Microsoft Visual C++ 2008 Redistributable Package if problems: http://www.microsoft.com/downloads/details.aspx?FamilyID=9b2da534-3e03-4391-8a4d-074b9f2bc1bf&displaylang=en

[clark15b]

My new project - PS3Muxer
http://forum.doom9.org/showthread.php?t=148627

[clark15b]

UPDATE is available.

Version 1.52

Now produces SRT file with subtitles for saving information about shooting time/date from the playlist and TS timecodes.

Example:
./tsdemux -m -j -c 1 -d samples/hf100/private/avchd/bdmv/stream/ -l samples/hf100/private/avchd/bdmv/playlist/00000.mpl

00015.track_4113.264
00015.track_4113.tmc
00015.track_4352.ac3
00015.track_4352.tmc
chapters.xml
timecodes.srt

[Ghitulescu]

Well, if I may have a suggestion

Just landed today over this thread, read all the comments, but I'm not really sure if it touches the following subject.

I have .M2TS from HDTV, that contain subtitles both as TXT and DVB SUB.

Is it possible to demux these subtitles?
Is it possible to join multiple .M2TS or .TS files (seamless, no asynch)? I used copy /b from DOS ;) with good results
Is it possible to check/reorder the timecodes for joined files?

All the questions are related to subtitles. Apparently I'm the only one on this planet that cares about subtitles in HDTV. I'm still looking for a clear method to recover/work them.

Thanks

PS: videoformat is H.264 for HDTV and MPEG-2 for SDTV. Audio are generally a mixture of AC-3 and MP2, usually 1 AC-3 and at least 1 MP2.

[clark15b]

Is it possible to demux these subtitles?
Yes, '-u' switch

Is it possible to join multiple .M2TS or .TS files (seamless, no asynch)? I used copy /b from DOS with good results
No, the program can not create M2TS/TS file as output... elementary streams + timecodes only.
'copy /b' is a true way, but stream not absolutely correct.

[Ghitulescu]

Yes, '-u' switch

Ok, I'll give it a try.

[Chug a Bug]

Hi, I've been looking for a tool for joining .m2ts files, this would fit the bill except... it won't demux DTS-MA. AC3 is fine, but not this.

[Adub]

I believe TSMuxer can join m2ts files. Why don't you give that a try?

[clark15b]

Chug a Bug
Yes, i don`t know how to demux HD-audio.

Sorry :(

[mariner]

How to join m2ts files properly?

I'm proud to present my own M2TS files demuxer taken from AVCHD camera and Blu-ray disks. Utility have a number of advantages and features when you working with AVCHD materials.

There is a major issue: joining multiple scenes located in different M2TS (MTS)/TS files into one file with audio/vidio sync (optional).

After brief topic examination it came to me that such thing can be done by eac3to, tsMuxer or simple 'copy /b' command in Windows and 'cat' in *nix-es, just addition one file in the end of another.

But in practice this is not good idea. The question is each M2TS file audio and video may be varied by length. This is urgent for the NTSC with fractional FPS, for example 29.97. Audio length (AC3 fps) and video frames (H264 JVT NAL) could not be variable, consequently there is no way to combine they at the end of the TS stream. Easiest way is to adjust video to audio, in this case variable fps is needed. In result we have audio stream 1 msec longer then video. It is not hard to calculate that for the 100 scenes we will get tremendous audio delay for 100 msec at the end.

When playing M2TS files one after one player begins from start and there are no visible synchronization issues. But when we join files, extracting elementary streams and muxing to MKV we are dropping information for synchronization completely. Muxer (mkvmerge) have no idea about initial TS stream properties and therefore sets its own time marks, supposing video and audio synchronized (I would remind you h.264 stream doesn't contain any metadata about FPS and so on).

Here the program features:

* runs either by Windows, Linux and FreeBSD;
* completely opensource , GPLv2;
* understands M2TS and TS file formats (type detetection by magic sequence);
* batch processing;
* joining multiple scenes;
* demux to ES or PES streams;
* extensions for the output files would be set depending of elementary stream;
* takes files list and/or directories list as command line argument (it is possible to specify a path to files for example private/avchd/bdmv/stream, program will take all appropriate files by itself);
* extracts shooting date and time of the each clip from mpls files in camera (private/avchd/bdmv/playlist/*.mpl);
* extracts elementary streams all programs in TS container or selected by number;
* makes files with time codes for appropriate elementary streams synchronizationl (mkvmerge timecode format v2);
* determining h.264 and AC3 elementary stream FPS;
* determining audio delay after video in each file and brings appropriate corrections into time codes (urgent for Blu-ray, you not needed to calculate and specify delay by yourself when muxing into MKV);
* trying to align elementary streams endings inside each scene;
* produces file with scenes for saving information about initial material dividing into multiple M2TS files. Scene names gives after file names or shooting time/date from the playlist (mkvmerge xml format);
* produces SRT file with subtitles for saving information about shooting time/date from the playlist;
* shows invoking mkvmerge command for muxing produced material into MKV;
* support MPEG2 Video, MPEG2 Audio, H.264, VC-1, AC3, L-PCM elementary stream types;
* parses and shows information about TS stream without elementary streams extraction (shows streams types. length, offset, and fps);
* dump TS structure (M2TS timecodes, PTS/DTS ...).

Unfortunately there is only console version available yet. It was tested against AVCHD cameras by Sony, Canon, Panasonic, some Blu-ray disks and broadcast TS.

Competitors disadvantages:
* xport - Unable to join files and can't take TS from the standard input. Output file names not reflecting content type;
* eac3to - No source code available, only Windows;
* tsMuxer - Also no source code, not so useful console version, joining files in GUI version is not comfortable.

And no one of them able to form files for proper streams synchronization for mkvmerge, no ability to generate file with scenes list, and no shooting time/date extraction.

I suppose my program may be an alternative to xport.

GUI Windows setup (win32): http://tsdemuxer.googlecode.com/files/tsdemuxer_setup.exe
Windows binaries (win32): http://tsdemuxer.googlecode.com/files/tsdemux.zip
Linux binaries (gcc4, static): http://tsdemuxer.googlecode.com/files/tsdemux.tar.gz
Sources: http://code.google.com/p/tsdemuxer/source/browse
Project home page: http://code.google.com/p/tsdemuxer


Simple invocation form: 'tsdemux.exe -m -j -d x:\private\avchd\bdmv\stream' or 'tsdemux.exe -j 00001.mts 00002.mts'. Result will be in five files in current directory: video stream , audio stream, tmc-file for audio, tmc-file for video, xml-file with scenes. Then you should pass all of these to mkvmerge.

Command line options:
tsdemux.exe [-d src] [-l mpls] [-o dst] [-c channel] [-u] [-j] [-m] [-z] [-p] [-e mode] [-v] *.ts|*.m2ts ...

-d discard all TS/M2TS/MTS files from specified directory (use it multiple times for different directories, files will be sorted ascending). For the AVCHD/Blu-ray it should be 'private/avchd/bdmv/stream'.
-l path to AVCHD/Blu-ray playlist mpl/mpls file (usually it is 'private/avchd/bdmv/playlist/*.mpls'). If this option set then for the names of scenes shooting time/date will be used also.
-o redirect output to another directory
-c channel number in TS stream (if not specified then all streams from all programs will be extracted, but scene file for mkvmerge will not generated).
-u demux unknown streams
-j join elementary streams
-m shows mkvmerge invoking options for muxing results in MKV.
-z demux to PES streams (instead of elementary streams).
-p deny elementary streams extraction, only analyze with information output.
-e dump TS structure to STDOUT (mode=1: dump M2TS timecodes, mode=2: dump PTS/DTS, mode=3: human readable PTS/DTS dump)
-v turn on verbose output

Valid files extensions: m2ts, mts, ts.
Output files could be with extensions: sup, m2v, 264, vc1, ac3, m2a, pcm. Files with time codes for each elementary stream for mkvmerge (tmc) and mkvmerge scene file (chapter.xml) will be generated as well.

Send bugreports to comments, please :)

Greetings Anton. May thanks for writing this detailed analysis of how to properly join m2ts files.

1. I have come across your wonderful program by way of Dean Kasabow's multiAVCHD. It offers two options for joining m2ts: tsdemux and eac3to. Both produce identical video stream, but the AC3 stream produced by eac3to is slightly longer in duration. Does this mean that the eac3to method (eac3to.exe 1+2+3+... -demux) gives incorrect results?

2. multiAVCHD then uses tsmuxer to mux together the 264 and ac3 streams. Do you think this will produce the correct result, as I remember reading somewhere that tsmuxer does not understand tmc file and may mess up the audio time stamp? If not, what would you suggest to correctly mux together a m2ts stream?

3. The tsdemux version used in multiAVCHD is dated May, and supports the -i instead of the -j option. While the 264 and ac3 output are identical, the tmc file for ac3 differs. How does this affect the accuracy of final joined m2ts file?

4. If I use copy/b to join the m2ts files and demux the audio stream, it would differ from both the tsdemux and eac3to output. Can we conclude that copy/b is not the proper way to join m2ts files?

5. The original source I use are actually mp4 with AAC audio. They were first converted by multiAVCHD into m2ts with AC3 audio since blu-ray does not accept AAC. Would it be a better option to first join these mp4 files, and what would you suggest to perform this task?

Many thanks and best regards.

[cafevincent]

I'm demuxing some bluray files and it seems tsdemux doesn't demux the dts-hd streams (not sure about normal dts). Is that normal?

edit: also I am trying to figure out how to keep out any chapters on the finished mkv, does that come into play during the demuxing?