HowTo Edit H264/AVC TS Streams

[Pookie]

You can try rebuilding the index with AVImaster


http://www.thozie.de/dnn/Home/tabid/36/EntryID/31/Default.aspx


Avimaster download link is on the right side of that page

Procedure: After AVC2AVI_mod creates your output.avi file, you would then run:

AVImaster.exe output.avi index.avi


Index.avi would then (theoretically) be your fixed up file. You can test if it works by loading up the resulting file in VirtualDub.

Would someone mind trying this? I'm not sure if it works or not. My access to h264 TS is limited on this side of the pond.

[crypto]

The lastest version of avi2avc_mod (http://forum.doom9.org/showthread.php?t=118267) handles BBC HD streams and builds key frame flags and an index. The resulting segmented avi loads perfectly into vdub.

[blubberbirne]

The resulting segmented avi loads perfectly into vdub.

What did you installed to open the avi in virtual dub? VD gives me an error that no H264 vfw codc is installed ??!

[crypto]

What did you installed to open the avi in virtual dub? VD gives me an error that no H264 vfw codc is installed ??!

ffdshow latest build. During setup select vdub support. (Links are in the avc2avi mod (http://forum.doom9.org/showthread.php?p=900451#post900451) thread)

[blubberbirne]

thx, will test it ;)

[Oxygen]

Well, seems i found a nearly working way.

Transmuxing the ts file into an mkv file via a dshow filter based on haali filters works fine.

Only one draw back of this solution, a desynch between audio & video seems to occur. Nothing unfixable though :)

[VincentNguyen]

Oxygen,

Could you describe your procedure to transcode directly to MKV using your dshow filter ? Did you fix the sync issue ?

[Selur]

btw. only way that worked for me to get multiple .ts files (of one movie) into a synch playing mkv:

1. cut the files with TSPE and create for each cut a new .ts file
2. remux each of these files with graphedit and Haalis muxer into new .mkv files
3. join these .mkv files with avi-mux gui

Cu Selur

Ps.: Could you describe your procedure to transcode directly to MKV using your dshow filter ?
probably like I do it:
1. start graphedit
2. throw the ts in it
3. remove all the filters except the one that's opening the file
4. insert Haalis muxer
5. connect the two filters
6. hit the play button

[Oxygen]

Ps.:
probably like I do it:
1. start graphedit
2. throw the ts in it
3. remove all the filters except the one that's opening the file
4. insert Haalis muxer
5. connect the two filters
6. hit the play button

Exact :)

[JK1974]

Sorry for this basic question, but I can´t remember the full name of TSPE or missed this tool in the previous threads - or is it just a typo? I just remember DVBViewer´s TS Player and TSConvert as tools for cutting H.264-TS files.
Second, I thought there is currently no way to demux H.264 video and audio streams in sync and that you have to sync them by cutting the audio streams manually. Are there really no sync problems with the graphedit procedure you just described or does it just work for your files? Or does it work because you cut the stream in the first step?

[Oxygen]

Transport Stream Packet Editor (TS Packet Editor)

As for the synch issues, i said from the start i have some.

I didn't check yet with a clean ts file as an input, but TSPE could do the trick, as i think the desynch is due to a badly cut ts file from the start.

[JK1974]

@Oxygen:
Thanks for your info. I also found more discussion in a German language web board - maybe because in Europe H.264-HDTV is in the first place and not MPEG-2-HDTV.

[Oxygen]

Please, don't be afraid to share your findings with us here, my german is way too underused to be good enough for such researchs :/

[JK1974]

@Oxygen:
They don´t have any new findings :), but as the Germans (nearly) don´t have MPEG2 transmissions, they are focussed on editing H.264 recordings only which is IMHO some advantage compared to the discussions here. But Selur is also there, and because here visits doom9 more often than me and posts more I am sure that he will keep you/us always up-to-date on the latest experiments there :)

[Selur]

atm. we got three main ways:

I. (the way I do it ;))
1. cut the files with TSPE and create for each cut a new .ts file
2. remux each of these files with graphedit and Haalis muxer into new .mkv files
3. join these .mkv files with avi-mux gui

and

II. (bergi 's way)
- looking for TSPE for cut points (better scan clip and use 'test' to be sure'
- using TSCut to cut the stream (since it's about 2 times faster than TSPE)
- use GraphEdit to mux to mkv (if you only got one audio stream skip the rest; otherwise only mux the video)
- use xport with -n parameter to demux multiple audiostreams
- use PTScalc get the delay of the audio streams (seems like this only functions when -n in xport was used)
- feed avimux gui with the mkv file and the ac3 files and use the delay values from ptscalc

Haven't tried II, since I got some problems with some of the tools bergi uses, but I asked him for commandline parameters & Co. (he also wrote a little gui for tscut which he might share with the community)

the above comes from the german doom9/gleitz board, there's also a discussion on the german movie2digital board, with another approach by madshi & Schmendrick1 (which I don't use since I don't use the required ds filters:

III.
h264tsto v1.0, freeware by madshi.net

Usage: h264tsto srcFilename.ts destFilename.mkv|destFilename.h264

This tool can either demux the H.264 stream to a raw *.h264 file.
Or it can mux the H.264 stream into a Matroska (*.mkv) container.

Demuxing into a raw *.h264 file needs the following filters to be installed:
(1) Elecard or Cyberlink Demuxer (empgdmx.ax/CLDemuxer.ax)
(2) Dump (dump.ax, shipping with GraphEdit)

Muxing into a Matroska container needs the following filters to be installed:
(1) Haali Media Splitter
(2) Haali Matroska Muxer

http://madshi.net/h264tsto.zip
I am sure that he will keep you/us always up-to-date on the latest experiments there
I try to if I remember to do so. ;)

Can't tell you which way is the 'best'/'most elegant' one since I only use I. since it's easy and works fine for me with the german Pro7 and englisch SkyHD streams I tried. (got no DVB-S2 compatible hardware myself so I only use captures provided by some friends)

Cu Selur

Ps.: if anyone uses xport: bergi needs some additional infos about pts <-> audio delay (in ms) so that he can modify xport (if he finds the time ;))

[drmpeg]

Ps.: if anyone uses xport: bergi needs some additional infos about pts <-> audio delay (in ms) so that he can modify xport (if he finds the time ;))
Hi, I'm the author of xport. The current version at:

http://www.w6rz.net/xport.zip

has been updated to handle H.264 video streams. It will properly start the video elementary stream on the first I-frame/field encountered, and will cut the audio elementary stream to match that I-frame/field. That is, the audio delay will be zero (when NOT using the -n option).

Ron

[bergi]

has been updated to handle H.264 video streams. It will properly start the video elementary stream on the first I-frame/field encountered, and will cut the audio elementary stream to match that I-frame/field. That is, the audio delay will be zero (when NOT using the -n option).That didn't work for me. Perhaps I missed something, but with the -n option and the right delay in the mkv file (calculated with PTScalc) everything was ok. xport already outputs the difference, but not in milliseconds. Could you add this feature to xport? Also an option to demux multiple audiostreams at the same time (without hardcoded names) would be cool, but the difference in ms stuff is more important for me.

[drmpeg]

That didn't work for me. Perhaps I missed something, but with the -n option and the right delay in the mkv file (calculated with PTScalc) everything was ok. xport already outputs the difference, but not in milliseconds. Could you add this feature to xport? Also an option to demux multiple audiostreams at the same time (without hardcoded names) would be cool, but the difference in ms stuff is more important for me.
The difference that xport reports is in 90 kHz ticks. So the delay in milliseconds would be difference/90.

For example, here's a typical dump from xport:

First Video PTS = 0x00025c1f
First Audio PTS = 0x00026bae, 1103

So the audio delay would be 1103/90 or 12.25 milliseconds.

I've thought about adding the other features you mention. I guess that I consider xport to be more like sample code than an actual application (given that it's command line only). However, the tool does have some following, so maybe I should spend some time on it. ;)

Ron

[Pookie]

"However, the tool does have some following, so maybe I should spend some time on it"

Indeed...Easily the fastest demuxer out there. Mandatory for the toolkit. Whatever you do, please keep it scriptable.

[xlazom00]

my experience

F:\>xport.exe star.wars.episode.four.a.new.hope.1080p.h264.oar.dd51.ts 3 1 1
program = 3, video channel = 1, audio channel = 1
Program Number = 3 (0x0003), Program Map PID = 48 (0x0030)
program descriptor = 0x0c, 0x04, 0x80, 0xb4, 0x81, 0x68
program descriptor = 0x10, 0x06, 0xc1, 0x5a, 0x01, 0xc0, 0x08, 0x00
Video PID = 49 <0x0031>, type = 0x1b
Audio PID = 52 <0x0034>, type = 0x81
ES descriptor for stream type 0x81 = 0x6a, 0x04, 0x41, 0x43, 0x2d, 0x33
5 frames before first I-frame
Main Profile
Level = 4.0
First Video PTS = 0x519ccc73
First Audio PTS = 0xf6db7d03, -1522621104
Audio Bitrate = 384000, Audio Sampling Rate = 48000
Audio Mode = 2/0, bsid = 6, bsmod = 0

packets for pid 0 <0x0000> = 2057, first = 2350, last = 10103841
packets for pid 48 <0x0030> = 2057, first = 2351, last = 10103842
packets for pid 49 <0x0031> = 9844451, first = 1, last = 10106140
packets for pid 52 <0x0034> = 231542, first = 3, last = 10106103
packets for pid 8190 <0x1ffe> = 26033, first = 360, last = 10106097
mpeg frames = 0, video fields = 0


this is from mplayer while i'm playling ac3 track. same with h264. But it's only first coupe frames
Starting playback...
a52: CRC check failed!
a52: error at resampling
a52: CRC check failed!
a52: error at resampling
a52: CRC check failed!
a52: error at resampling
a52: CRC check failed! (14:44.5) ??,?% 25%
a52: error at resampling
a52: CRC check failed!
a52: error at resampling
a52: CRC check failed!
a52: error at resampling
a52: CRC check failed! (14:44.5) 1.3% 49%
a52: error at resampling
a52: CRC check failed!
a52: error at resampling

[bergi]

I think I have done something wrong last time using xport. xport sync works fine!

Here is my small howto ts->mkv:
- Find cutpoints with TSPE
- Cut with bergtscut
- Use GraphEdit to remux the videostream from the ts file to mkv file
- Demux audiostream(s) with xport (without -n!)
- Mux video mkv file and audiostream(s) with mkvmerge/AVI-Mux GUI (delay=0)

bergtscut (https://www.axolotlfarm.org/~bergi/projects/bergsat/bergtscut20070111.zip) is a small command line programm for cutting ts files. It's a very stupid program, it just cuts files 188byte aligned, but with a big buffer (~100mb).

[Pookie]

Right on, bergi. Your new app might be good for postprocessing Comskip as well.

--Note: bergtscut.exe is using the info from the "Byte" fields in TSPE.

bergtscut.exe 2317476 121971952 in.ts out.ts

The above example is saving the data between 2.3MB and 121.9MB (approximately)

Guide and download link for TSPE :

http://www.flaskmpeg.info/board/thread.php?postid=68161&sid=4bb930ec4ef749a2557276284d1451b5#post68161

[wiak]

checked a tool called ts packet editor?
http://www.techpowerup.com/downloads/435
:D

[apk]

I followed the instructions but only got this:


D:\>1\mencoder.exe "D:\1\testcap.ts" -of rawvideo -ovc copy -oac copy -noskip -o
D:\1\test.264
MEncoder dev-SVN-r19260-4.0.3 (C) 2000-2006 MPlayer Team
CPU: AMD Athlon(tm) 64 Processor 3500+ (Family: 15, Model: 47, Stepping: 0)
CPUflags: Type: 15 MMX: 1 MMX2: 1 3DNow: 1 3DNow2: 1 SSE: 1 SSE2: 1
Compiled for x86 CPU with extensions: MMX MMX2 3DNow 3DNowEx SSE SSE2

93 audio & 210 video codecs
success: format: 0 data: 0x0 - 0x3437a50
TS file format detected.
VIDEO MPEG2(pid=1502) AUDIO A52(pid=1505) NO SUBS (yet)! PROGRAM N. 0
TS_PARSE: COULDN'T SYNC
MPEG: FATAL: EOF while searching for sequence header.
Video: Cannot read properties.

[Hans Ohlo]

i got the following output from xport:

D:\__encode\test>.\..\__tools\xport\xport -n test.1080p.h264.dd51.part1.ts 3 1 1
program = 3, video channel = 1, audio channel = 1
Program Number = 3 (0x0003), Program Map PID = 48 (0x0030)
program descriptor = 0x0c, 0x04, 0x80, 0xb4, 0x81, 0x68
program descriptor = 0x10, 0x06, 0xc1, 0x5a, 0x01, 0xc0, 0x08, 0x00
Video PID = 49 <0x0031>, type = 0x1b
Audio PID = 52 <0x0034>, type = 0x81
ES descriptor for stream type 0x81 = 0x6a, 0x04, 0x41, 0x43, 0x2d, 0x33
First Audio PTS = 0xfe935f8d, -23899059
Audio Bitrate = 384000, Audio Sampling Rate = 48000
Audio Mode = 2/0, bsid = 6, bsmod = 0
2 frames before first I-frame
Main Profile
Level = 4.0
First Video PTS = 0xfe94c24a

packets for pid 0 <0x0000> = 1931, first = 1, last = 9760750
packets for pid 48 <0x0030> = 1931, first = 2, last = 9760751
packets for pid 49 <0x0031> = 9518357, first = 3, last = 9761084
packets for pid 52 <0x0034> = 214722, first = 19, last = 9761071
packets for pid 8190 <0x1ffe> = 24143, first = 107, last = 9761002
mpeg frames = 0, video fields = 0

what should i make of this? when i mux the output streams to a mkv container without any delays added the audio is slightly out of sync (i would say max +500ms). but how can i get the exact delay from this output?

[drmpeg]

i got the following output from xport:

D:\__encode\test>.\..\__tools\xport\xport -n test.1080p.h264.dd51.part1.ts 3 1 1
program = 3, video channel = 1, audio channel = 1
Program Number = 3 (0x0003), Program Map PID = 48 (0x0030)
program descriptor = 0x0c, 0x04, 0x80, 0xb4, 0x81, 0x68
program descriptor = 0x10, 0x06, 0xc1, 0x5a, 0x01, 0xc0, 0x08, 0x00
Video PID = 49 <0x0031>, type = 0x1b
Audio PID = 52 <0x0034>, type = 0x81
ES descriptor for stream type 0x81 = 0x6a, 0x04, 0x41, 0x43, 0x2d, 0x33
First Audio PTS = 0xfe935f8d, -23899059
Audio Bitrate = 384000, Audio Sampling Rate = 48000
Audio Mode = 2/0, bsid = 6, bsmod = 0
2 frames before first I-frame
Main Profile
Level = 4.0
First Video PTS = 0xfe94c24a

packets for pid 0 <0x0000> = 1931, first = 1, last = 9760750
packets for pid 48 <0x0030> = 1931, first = 2, last = 9760751
packets for pid 49 <0x0031> = 9518357, first = 3, last = 9761084
packets for pid 52 <0x0034> = 214722, first = 19, last = 9761071
packets for pid 8190 <0x1ffe> = 24143, first = 107, last = 9761002
mpeg frames = 0, video fields = 0

what should i make of this? when i mux the output streams to a mkv container without any delays added the audio is slightly out of sync (i would say max +500ms). but how can i get the exact delay from this output?

Hi. I think I know what's going on. Will investigate.

Ron

[bergi]

D:\__encode\test>.\..\__tools\xport\xport -n test.1080p.h264.dd51.part1.ts 3 1 1
Just call xport without the -n parameter and everything should be ok.

[drmpeg]

Just call xport without the -n parameter and everything should be ok.

Good eye, I missed that.

Ron

[lupus11]

PTSCalc ? The homepage is no longer working. Is there any alternative download link ?

Thank You
lupus

[enantiomer]

Ron,
This is slightly off topic, but I was attempting to demux a VC-1 BDMV transport stream, and found that xport could not recognize the video stream. I was wondering if you could augment your xport program to handle VC-1? Or is there an option to force the demuxing of a particular PID?

Regardless, your evob_demux and xport tools have been very helpful. Thanks.

[Pookie]

lupus - http://technik.movie2digital.de/thread.php?threadid=38242

[Demone]

I noticed that during ts demux of h264 mencoder is at 250fps then at about 15% of ALL of my ts's it starts to slowdown and never ends, as of writing its at 30% and 68fps and still drops 1fps every 10 seconds.
Do you know if there's any problem with ntfs and big filesizes ?

Thanks

[Pookie]

Dunno, but you can troubleshoot by testing various builds of Mencoder. It might be time consuming to do so, however

http://oss.netfarm.it/mplayer-win32.php

http://tirnanog.fate.jp/mirror/mplayer/

Additional things to try:

demux with h264tsto

demux with xport

Increase your page file size in windows

Run a defrag on your drive

Demux to a different physical drive (this speeds things up a lot)

[protovision]

I noticed that during ts demux of h264 mencoder is at 250fps then at about 15% of ALL of my ts's it starts to slowdown and never ends, as of writing its at 30% and 68fps and still drops 1fps every 10 seconds.
Do you know if there's any problem with ntfs and big filesizes ?
Thanks

As pookie mentioned, use separate source/destination drives, and defrag your drive with a good defragger. I ran into the same issue when joining many .ts streams into one. HDTV2MPEG started at 40 MB/s, then about 50% done, it slowed to 15MB/s. My drive was fragmented, so it was jumping all over the drive when joining the pieces, and then jumping looking for somewhere to save to. Defragged with PerfectDisk8, all better now.

p.

[Pookie]

Jkdefrag . Free and good.

http://www.kessels.com/JkDefrag/index.html

[Demone]

Hi guys,
thank you very much for your help the problem is that I'm just using two different hd and I tried to invert source/destination
and tried only one hd too that is very fast, but no way.
The strangeness here is that mencoder starts very fast as I said like 250fps then slowdown at 2%/3%... for a complete movie that is 15gb it took 45 minutes to demux too much !!! :scared:
I'll try mplayer and see if its different

[Kurtnoise]

You can try also MP4Box to extract raw avc from TS files...

[Demone]

Oh thanks,
I just remember now I had problems creating mp4 from big h264 files with it so I switched to mp4creator cause it has a flag for big file support.

[Demone]

@protovision
I doubt its a fragmentation problem, when the hd is fragmented heads spin like mad while my hd is not used at all.
With 22096 build of mplayer I reached the 9gb at full speed and I was about to joy when it started to slow down again.:confused:
It reached 9gb in 10 minutes and after 30 minutes its still at 14gb.
The cpu max level is 20% during all the time there are no swap hd movements as the page file is fixed at 530mb/1.5GB so it should be some mplayer/mencoder problem.

[albatross83]

The download link on the PTSCalc site is broken. Page works, download link on the page doesn't. Anyone have an alternative source? Hardly any results from Google and I've explored them all without success.

[Schmendrick]

Hi albatross83,

following a link to the orginal german forum page where PTSCalc was presented:

http://technik.movie2digital.de/thread.php?threadid=38242

PTSCalc is available as attachement on the first post of the thread in this link and it is explained on the website of the author: http://rickman.ri.funpic.de/ which does not contain the programme itself.

Good luck

Schmendrick

[albatross83]

Not sure how I missed that attachment, but I did. Thanks!

[Cryptum]

Here is my little contribution :

Ok I have found a (even few) way to join parts files into one file (MKV file)
It's not a very clean method but that works well !

Beginning with the TSPE splitting method :

1) After the splitting operation, we need tu mux every ts part on an mkv container,
by doing this with h264tsto -> http://forum.doom9.org/showthread.php?t=117806

1.1) Then the simple way is to join all parts with AVI-Mux GUI into one MKV file.
This last step is very easy and the final resultat is an playable file.
There are no audio sync problem, no playback problem, I mean, we can passing through
the splitting points without stuttering or other problem, we can jump forward/backward
very well. But still, there are ONE problem.
The openning of this file take very long time (5-10 sec)
but after that, there are no more problem.
I don't know why this bug (maybe it's because AVI-Mux GUI makes a MKV V1 instead of V2)

2) Second method is a little similar. So we need to mux the parts with h264tsto.

2.1) We need to use MKVMerge to join all MKV parts.
Result does a playble file but the audio after the first part is unsync,
so we need to estimate this and remux another time but by adding the new sync parameter.
To do that -> Click on the second audio part -> Format specific options ->
And fill the "Delay" in with our estimation.
Same thing with the third audio part (but normaly with a higher sync value),
again with the fourth part, etc...
Check the result and correct if it needs to.

With this method (a little bit annoyning I know ..), with have an MKV file playable
and without problem (I haven't seen one until now !)

Splitting with TSsplitter :

3) It's easier because we have a GUI, we have 2 modes to split
(position in time value or Byte value).
We can use the same method at point 1.1, but we cannot use the method 2.1,
MKVMerge failed with a lot of "position" error. Can't find a solution at this time :(

4) I have not tested this but I think it will work.
So another alternative is to mux the entire TS movie with h264tsto.
4.1) Split the file with the help of MKVMerge -> Tab "Global" -> "splitting".
This is not very convenient, like TSPE, without a preview mode like on TSsplitter
but that can do it and furthermore it does that well (because joining step are doing well)
Now the last step is to join all parts with MKVMerge too and maybe we don't need
to resync audio stream (I'm not sure about that ..)

So now we have a "normal" playable file,
easy to sync subtitle whith it and more "appearence friendly" ;)

PS: Thx to Selur for the tip about AVI-Mux GUI

Da Cryptum

[Selur]

(maybe it's because AVI-Mux GUI makes a MKV V1 instead of V2)
this can be changed:
Avi-Mux Gui -> settings ->mkv structure->Matroska version ;)

[Cryptum]

Oh yes we need to uncheck the V1 before :P
Strange method lol

Thx

Da Cryptum

Edit : Hum so same problem than V1. Long long time to open the file :(

[Selur]

What filters do you use?
Doesn't take long here (Haali+CoreAVC+ffdshow for aac->ac3->spdif).

[Cryptum]

Sometimes CoreAVC+ffdshow and sometimes only CoreAVC.
And for audio, I use AC3Filter. That's it ;)

I don't think that comes from filters because my others MKV haven't this problem.

Da Cryptum

[Atak_Snajpera]

How to demux all audio streams with xport at once?

[drmpeg]

How to demux all audio streams with xport at once?
It's not possible with the current version. It was originally written for a hardware decoder many years ago, and targeted for one video and one audio stream output. Although it's certainly possible to demux all elementary streams, it won't be easy to implement with the current architecture of xport.

To speed things up, you can select a video track that's not in the multiplex for subsequent audio tracks. Like this:

xport clip.ts 1 1 1

This will demux the video and audio track 1. Then:

xport clip.ts 1 7 2

This will demux audio track 2 and create a zero byte file for bits0001.mpv (be sure to rename the first bits0001.mpv that has the demuxed video beforehand). The demux process will be much faster since there's no video to demux.

Ron

[MarkRy]

My first post! Excellent thread. Learned alot.

I have, thanks to this thread, converted an BBCHD DVB-S .ts to MP4 and AVI. As I did all the different bits, the video looked slow but I thought it was my system / codecs. Now its all complete I see I have a problem. The AVI file plays the audio at the correct speed and the video at half speed. Working backwards I go back to the ts file which in Mplayer plays the video half speed while repeating the audio bits twice.

At the start of the process, when I dump the video using mplayer, I imagine the half speed video problem is fixed into everything else I do.

I'm guessing its an interlace setting but would appreciate help. Thanks

Edit: I played the avi file in DVBViewer and it was a mirror image in the horizontal axis but it seems at the right speed! Any ideas what is going wrong?

The reason I am converting files is that I bought the Evesham iplayer to play HD / SD files. Prior a firmware update it could receive the BBC HD DVB-T broadcast. So I bought a card for my PC and recorded the satellite stream. It tries but cant play DVB-S ts off a USB Hard Drive. An analysis of the clip looks so similar to a recording of the DVB-T BBC-HD signal. But it wont play. I dont want to reencode the DVB-S stream (if I can help it) so MP4 and AVI seem a good 2nd best. I either need to find out why it wont play the DVB-s ts stream and tweak it, or create a solid MP4 or AVI stream it will. Is that asking too much!