Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > (HD) DVD, Blu-ray & (S)VCD > (HD) DVD & Blu-ray authoring

Reply
 
Thread Tools Search this Thread Display Modes
Old 4th May 2009, 09:42   #1  |  Link
clark15b
Registered User
 
clark15b's Avatar
 
Join Date: Apr 2009
Location: Moscow, Russia
Posts: 61
Another AVCHD/Blu-Ray demuxer

I'm proud to present my own M2TS files demuxer taken from AVCHD camera and Blu-ray disks. Utility have a number of advantages and features when you working with AVCHD materials.

There is a major issue: joining multiple scenes located in different M2TS (MTS)/TS files into one file with audio/vidio sync (optional).

After brief topic examination it came to me that such thing can be done by eac3to, tsMuxer or simple 'copy /b' command in Windows and 'cat' in *nix-es, just addition one file in the end of another.

But in practice this is not good idea. The question is each M2TS file audio and video may be varied by length. This is urgent for the NTSC with fractional FPS, for example 29.97. Audio length (AC3 fps) and video frames (H264 JVT NAL) could not be variable, consequently there is no way to combine they at the end of the TS stream. Easiest way is to adjust video to audio, in this case variable fps is needed. In result we have audio stream 1 msec longer then video. It is not hard to calculate that for the 100 scenes we will get tremendous audio delay for 100 msec at the end.

When playing M2TS files one after one player begins from start and there are no visible synchronization issues. But when we join files, extracting elementary streams and muxing to MKV we are dropping information for synchronization completely. Muxer (mkvmerge) have no idea about initial TS stream properties and therefore sets its own time marks, supposing video and audio synchronized (I would remind you h.264 stream doesn't contain any metadata about FPS and so on).

Here the program features:

* runs either by Windows, Linux and FreeBSD;
* completely opensource , GPLv2;
* understands M2TS and TS file formats (type detetection by magic sequence);
* batch processing;
* joining multiple scenes;
* demux to ES or PES streams;
* extensions for the output files would be set depending of elementary stream;
* takes files list and/or directories list as command line argument (it is possible to specify a path to files for example private/avchd/bdmv/stream, program will take all appropriate files by itself);
* extracts shooting date and time of the each clip from mpls files in camera (private/avchd/bdmv/playlist/*.mpl);
* extracts elementary streams all programs in TS container or selected by number;
* makes files with time codes for appropriate elementary streams synchronizationl (mkvmerge timecode format v2);
* determining h.264 and AC3 elementary stream FPS;
* determining audio delay after video in each file and brings appropriate corrections into time codes (urgent for Blu-ray, you not needed to calculate and specify delay by yourself when muxing into MKV);
* trying to align elementary streams endings inside each scene;
* produces file with scenes for saving information about initial material dividing into multiple M2TS files. Scene names gives after file names or shooting time/date from the playlist (mkvmerge xml format);
* produces SRT file with subtitles for saving information about shooting time/date from the playlist;
* shows invoking mkvmerge command for muxing produced material into MKV;
* support MPEG2 Video, MPEG2 Audio, H.264, VC-1, AC3, L-PCM elementary stream types;
* parses and shows information about TS stream without elementary streams extraction (shows streams types. length, offset, and fps);
* dump TS structure (M2TS timecodes, PTS/DTS ...).

Unfortunately there is only console version available yet. It was tested against AVCHD cameras by Sony, Canon, Panasonic, some Blu-ray disks and broadcast TS.

Competitors disadvantages:
* xport - Unable to join files and can't take TS from the standard input. Output file names not reflecting content type;
* eac3to - No source code available, only Windows;
* tsMuxer - Also no source code, not so useful console version, joining files in GUI version is not comfortable.

And no one of them able to form files for proper streams synchronization for mkvmerge, no ability to generate file with scenes list, and no shooting time/date extraction.

I suppose my program may be an alternative to xport.

GUI Windows setup (win32): http://tsdemuxer.googlecode.com/file...uxer_setup.exe
Windows binaries (win32): http://tsdemuxer.googlecode.com/files/tsdemux.zip
Linux binaries (gcc4, static): http://tsdemuxer.googlecode.com/files/tsdemux.tar.gz
Sources: http://code.google.com/p/tsdemuxer/source/browse
Project home page: http://code.google.com/p/tsdemuxer


Simple invocation form: 'tsdemux.exe -m -j -d x:\private\avchd\bdmv\stream' or 'tsdemux.exe -j 00001.mts 00002.mts'. Result will be in five files in current directory: video stream , audio stream, tmc-file for audio, tmc-file for video, xml-file with scenes. Then you should pass all of these to mkvmerge.

Command line options:
tsdemux.exe [-d src] [-l mpls] [-o dst] [-c channel] [-u] [-j] [-m] [-z] [-p] [-e mode] [-v] *.ts|*.m2ts ...

-d discard all TS/M2TS/MTS files from specified directory (use it multiple times for different directories, files will be sorted ascending). For the AVCHD/Blu-ray it should be 'private/avchd/bdmv/stream'.
-l path to AVCHD/Blu-ray playlist mpl/mpls file (usually it is 'private/avchd/bdmv/playlist/*.mpls'). If this option set then for the names of scenes shooting time/date will be used also.
-o redirect output to another directory
-c channel number in TS stream (if not specified then all streams from all programs will be extracted, but scene file for mkvmerge will not generated).
-u demux unknown streams
-j join elementary streams
-m shows mkvmerge invoking options for muxing results in MKV.
-z demux to PES streams (instead of elementary streams).
-p deny elementary streams extraction, only analyze with information output.
-e dump TS structure to STDOUT (mode=1: dump M2TS timecodes, mode=2: dump PTS/DTS, mode=3: human readable PTS/DTS dump)
-v turn on verbose output

Valid files extensions: m2ts, mts, ts.
Output files could be with extensions: sup, m2v, 264, vc1, ac3, m2a, pcm. Files with time codes for each elementary stream for mkvmerge (tmc) and mkvmerge scene file (chapter.xml) will be generated as well.

Send bugreports to comments, please

Last edited by clark15b; 28th August 2009 at 13:12. Reason: version 1.52
clark15b is offline   Reply With Quote
Old 4th May 2009, 14:21   #2  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,901
Very interesting, thank you for your work and contribution. Is there a GUI in your plans?
Guest is offline   Reply With Quote
Old 4th May 2009, 14:28   #3  |  Link
clark15b
Registered User
 
clark15b's Avatar
 
Join Date: Apr 2009
Location: Moscow, Russia
Posts: 61
Sure, I will take care about GUI if community will show interest in project.
clark15b is offline   Reply With Quote
Old 4th May 2009, 14:34   #4  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,901
Well, it would certainly make a big advance over xport, whereas right now, the delta is relatively small. I wish I had time to help with it.
Guest is offline   Reply With Quote
Old 4th May 2009, 16:27   #5  |  Link
clark15b
Registered User
 
clark15b's Avatar
 
Join Date: Apr 2009
Location: Moscow, Russia
Posts: 61
Currently it is of importance for me to test program functions with different signal sources (different AVCHD camcorders, Blu-ray disks with different codecs). Will be very gratefull for help with this. I will work on GUI in the mean time.
clark15b is offline   Reply With Quote
Old 4th May 2009, 16:36   #6  |  Link
deank
Programmer (or just 教务长)
 
deank's Avatar
 
Join Date: Oct 2008
Location: Valencia, Spain
Posts: 4,251
A lot of people willing to help you in testing will have troubles without a GUI. You can get test results faster if you compile one for them.

I don't have AVCHD camcoder, but I have a lot of samples and will do some tests and will let you know.

Dean
__________________
multiAVCHD - donate | popBD | uncropMKV | mkv2avi | easySUP
deank is offline   Reply With Quote
Old 4th May 2009, 16:37   #7  |  Link
noee
Registered User
 
Join Date: Jan 2007
Posts: 530
Thank you for the tool, I will begin testing with my AVCHD (Canon HF100) files promptly. I, too, would welcome a GUI and would be willing to help test that as well.
noee is offline   Reply With Quote
Old 4th May 2009, 18:36   #8  |  Link
noee
Registered User
 
Join Date: Jan 2007
Posts: 530
I have tried a few AVCHD files and I am having an issue with the resultant MKV. Whether I mux it myself with MkvmergGUI or your command line, the resultant MKV file takes a very long time to open and it is solid disk activity while waiting for it to open.

I'm using Haali Media Splitter and the 2.7.0 MKVToolnix and I've tried opening it with MPC-HC and jRiver MC13, both exhibit the same extended time to open the MKV. Once open, playback is as expected.

HTH
noee is offline   Reply With Quote
Old 5th May 2009, 08:41   #9  |  Link
clark15b
Registered User
 
clark15b's Avatar
 
Join Date: Apr 2009
Location: Moscow, Russia
Posts: 61
Thanks for your interest, I will make GUI soon.

noee
I do not think that a problem in muxer and MKV, I have no problem with reproduction of original M2TS files and resulting MKV on the computer. I use Haali Media Splitter, CoreAVC and MediaPlayer Classic.
clark15b is offline   Reply With Quote
Old 5th May 2009, 08:57   #10  |  Link
turbojet
Registered User
 
Join Date: May 2008
Posts: 1,840
Firet impressions is it's really fast, only seconds slower then tsmuxer. eac3to and xport are slow in comparison for some reason.

If it had mkv/mp4 input, avchd output, sup/srt input/output, and multiple audio support I'd be real interested in it mainly because of the cli. Do you have any plans for any of these?

I tried it on a retail VC-1 BD and found some problems.
- detected the 23.976 VC-1 stream as 29.97 fps, tsmuxer and eac3to detect 23.976.
- when I used -l with BDMV\PLAYLIST directory on same BD and got 'invalid file format'
- -d BDMV\PLAYLIST\00000.mpls (main movie) crashed tsdemux
- chapters.xml only contained 3 chapters starting at 3 hours, the movie is only 90 minutes long
- After demuxing the main movie m2ts on same BD it gave these errors:
Code:
** bad last PTS in stream 0xfd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xfd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xfd ('vc1'), length 5212595.84ms (
+1244.71ms)
** align stream 0xfd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xfd ('ac3'), length 5212256.00ms (
+904.87ms)
** align stream 0xfd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xfd ('ac3'), length 5212256.00ms (
+904.87ms)
** align stream 0xfd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xfd ('ac3'), length 5212256.00ms (
+904.87ms)
** align stream 0xfd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xfd ('bin'), length 5347985.00ms (
+136633.87ms)
** align stream 0xfd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xfd ('ac3'), length 5212256.00ms (
+904.87ms)
** align stream 0xfd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xbd ('bin'), length 14257757.24ms
(+9104356.30ms), delay 57950.19ms
** align stream 0xbd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xbd ('bin'), length 10041529.73ms
(+4926292.13ms), delay 96113.53ms
** align stream 0xbd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xbd ('bin'), length 10256699.44ms
(+5112182.63ms), delay 66834.32ms
** align stream 0xbd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xbd ('bin'), length 4963921.22ms (
+2.01ms), delay 247431.92ms
** align stream 0xbd length to 14315708.00ms
K:\BDMV\STREAM\00000.m2ts: channel 1, stream 0xbd ('bin'), length 4963919.87ms,
delay 247431.27ms
** align stream 0xbd length to 14315708.00ms
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00001.m2ts: channel 1, stream 0xfd ('vc1'), length 11970.28ms (+1
0288.73ms)
K:\BDMV\STREAM\00001.m2ts: channel 1, stream 0xfd ('ac3'), length 12000.00ms (+1
0318.46ms)
K:\BDMV\STREAM\00001.m2ts: channel 1, stream 0xbd ('bin'), length 745.34ms, dela
y 936.20ms
** bad last PTS in stream 0xfd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00002.m2ts: channel 1, stream 0xfd ('vc1'), length 11887.67ms
K:\BDMV\STREAM\00002.m2ts: channel 1, stream 0xfd ('ac3'), length 11904.00ms (+1
6.33ms)
K:\BDMV\STREAM\00003.m2ts: channel 1, stream 0xfd ('vc1'), length 5872.53ms
** align stream 0xfd length to 5873.00ms
** bad last PTS in stream 0xfd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00004.m2ts: channel 1, stream 0xfd ('vc1'), length 23274.80ms
** align stream 0xfd length to 23275.00ms
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00005.m2ts: channel 1, stream 0xbd ('bin'), length 1363.13ms
** align stream 0xbd length to 1364.00ms
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00006.m2ts: channel 1, stream 0xbd ('bin'), length 1386.13ms
** align stream 0xbd length to 1387.00ms
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xfd ('vc1'), length 10010.00ms (+8
386.79ms)
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xfd ('ac3'), length 10016.00ms (+8
392.79ms)
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xfd ('ac3'), length 10016.00ms (+8
392.79ms)
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xfd ('ac3'), length 10016.00ms (+8
392.79ms)
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xfd ('ac3'), length 10016.00ms (+8
392.79ms)
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xfd ('ac3'), length 10016.00ms (+8
392.79ms)
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xbd ('bin'), length 687.01ms, dela
y 936.20ms
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xbd ('bin'), length 687.01ms, dela
y 936.20ms
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xbd ('bin'), length 687.01ms, dela
y 936.20ms
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xbd ('bin'), length 687.01ms, dela
y 936.20ms
K:\BDMV\STREAM\00007.m2ts: channel 1, stream 0xbd ('bin'), length 687.01ms, dela
y 936.20ms
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00008.m2ts: channel 1, stream 0xbd ('bin'), length 207.40ms
** align stream 0xbd length to 208.00ms
** bad last PTS in stream 0xbd, use first PTS and current frame rate for fix it
K:\BDMV\STREAM\00009.m2ts: channel 1, stream 0xbd ('bin'), length 207.40ms
** align stream 0xbd length to 208.00ms

Last edited by turbojet; 5th May 2009 at 09:20.
turbojet is offline   Reply With Quote
Old 5th May 2009, 09:15   #11  |  Link
clark15b
Registered User
 
clark15b's Avatar
 
Join Date: Apr 2009
Location: Moscow, Russia
Posts: 61
Unfortunately I didn't plan to do reverse conversion. It is completely different and more tricky task.
clark15b is offline   Reply With Quote
Old 5th May 2009, 09:23   #12  |  Link
turbojet
Registered User
 
Join Date: May 2008
Posts: 1,840
Oh ok shucks I guess I'll have to keep trying to get through to tsmuxer developer to add some simple cli. Thanks anyways.

After muxing the demuxed files to m2ts/mkv files it chose the last audio track to demux and it needs about +200 ms delay.

Last edited by turbojet; 5th May 2009 at 09:36.
turbojet is offline   Reply With Quote
Old 5th May 2009, 09:47   #13  |  Link
clark15b
Registered User
 
clark15b's Avatar
 
Join Date: Apr 2009
Location: Moscow, Russia
Posts: 61
turbojet
Thank you! Do you use Windows or Unix?
Could you please send me BDMV/PLAYLIST (at least several files)?
Please specify command line which you were using.
Could you please send a part of main movie M2TS file?
clark15b is offline   Reply With Quote
Old 5th May 2009, 09:56   #14  |  Link
turbojet
Registered User
 
Join Date: May 2008
Posts: 1,840
This was windows, I could test linux if you think it would make a difference.

Here's the files you requested.

I used tsdemux -d K:\BDMV\STREAM\00000.m2ts to demux
turbojet is offline   Reply With Quote
Old 5th May 2009, 10:11   #15  |  Link
deank
Programmer (or just 教务长)
 
deank's Avatar
 
Join Date: Oct 2008
Location: Valencia, Spain
Posts: 4,251
Hi, clark15b!

I'm having a problem with the -l switch. Every time I try to use it, tsdemux crashes.
__________________
multiAVCHD - donate | popBD | uncropMKV | mkv2avi | easySUP
deank is offline   Reply With Quote
Old 5th May 2009, 11:08   #16  |  Link
clark15b
Registered User
 
clark15b's Avatar
 
Join Date: Apr 2009
Location: Moscow, Russia
Posts: 61
turbojet
Thank you, try 'tsdemux K:\BDMV\STREAM\00000.m2ts' for demux.

deank
Whether you have the description of mpls files?
clark15b is offline   Reply With Quote
Old 5th May 2009, 11:24   #17  |  Link
turbojet
Registered User
 
Join Date: May 2008
Posts: 1,840
Quote:
Originally Posted by clark15b View Post
turbojet
Thank you, try 'tsdemux K:\BDMV\STREAM\00000.m2ts' for demux.
It detected 23.976 fps and there was no chapters in the xml other than that the same output.
turbojet is offline   Reply With Quote
Old 5th May 2009, 11:57   #18  |  Link
clark15b
Registered User
 
clark15b's Avatar
 
Join Date: Apr 2009
Location: Moscow, Russia
Posts: 61
turbojet
I have correctly understood, now it has correctly detect fps?
One scene will be in xml if one initial M2TS file is specified.
clark15b is offline   Reply With Quote
Old 5th May 2009, 12:41   #19  |  Link
deank
Programmer (or just 教务长)
 
deank's Avatar
 
Join Date: Oct 2008
Location: Valencia, Spain
Posts: 4,251
clark, I'd suggest you this:

Code:
50    if(memcmp(ptr,"MPLS0100",8))
51        return -1;
you can either make it ..."MPLS0",5... or check for MPLS0200, too.

and

Code:
58    unsigned char* pp=ptr+68;
60    if(pp+num*82>ptr+len)
...
72        pp+=82;
This can lead you to troubles. ptr[68] and [69] keep the length of the current segment, so it is correct to increment pp in line 72 not by 82 but with the value in 68/69.

Then for the next cycle use pp as the base for other chapter calculations.
__________________
multiAVCHD - donate | popBD | uncropMKV | mkv2avi | easySUP

Last edited by deank; 5th May 2009 at 12:47.
deank is offline   Reply With Quote
Old 5th May 2009, 13:20   #20  |  Link
turbojet
Registered User
 
Join Date: May 2008
Posts: 1,840
Quote:
Originally Posted by clark15b View Post
turbojet
I have correctly understood, now it has correctly detect fps?
One scene will be in xml if one initial M2TS file is specified.
Yes but I think on the first try it was reading 00007.m2ts which is the last file and it is indeed 29.97 fps
turbojet is offline   Reply With Quote
Reply

Tags
avchd, blu-ray, mux

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 08:28.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.