Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > HDTV / DVB / TiVo

Reply
 
Thread Tools Search this Thread Display Modes
Old 3rd December 2008, 21:16   #1  |  Link
Ventolin
Registered User
 
Join Date: Oct 2001
Posts: 125
TSPE 0.700 Public Beta out now!

Hi folks,

After a couple years of quiet development in the darkest corners of the Internet, another public beta of TSPE is now available!

http://www.bitstreamtools.com

For those that don't know, TSPE (Transport Stream Packet Editor) is a non-linear Editor / Analyser.

So, what's new...

Supported Formats:
MPEG2 / H.264 / VC-1 / AC-3 / DDP / MLP / DTS / DTS-HD / DTS-HD MA / AAC / MPEG1 Audio / MPEG2 Audio / PCM (everything?

.TS and .M2TS support including stripping the m2ts header to make normal 188 byte .TS files.

No re-encoding of the stream to preserve maximum quality, I frame to P frame accurate, clean video/audio editing, timestamp and table correction.

You can cut commercials from captured sources, detect commercials using format changes, scan for errors, import TSPE and M2R logs, fix broken streams including timestamp wraparounds, continuity errors, timestamp gaps, realign to start codes and resync audio.

Stream stripping with table correction, PCR remapping, Demuxing, Core Audio from HD audio extraction.

Video preview is much more accurate, uses your own directshow filters, test and full edits are previewable with TSPE's own internal viewer, multi-part ts file joining with autosorting EDL (Edit decision List). Marker List to enable bookmarking of positions.

Packet Header Decode for analysis. Resizable interface.

Plenty more too, visit the website for details.

TSPE is now shareware, free trial period is 45 days. I wanted to release TSPE once I was sure it performed well enough to justify payment. It is still in beta testing, but most of the major bugs have been ironed out (I hope!). Still plenty more work to be done, eg. m2ts and vc1 support is still sketchy, but it does do the job just about.

Please let me know how you get along; of course I would like to know your feedback good or bad and especially if you have some streams which don't work properly.

Have fun,

Regards,

Vent
Ventolin is offline   Reply With Quote
Old 3rd December 2008, 23:21   #2  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,528
Those two ts samples do not work (no preview)
http://www.mediafire.com/?sharekey=f...db6fb9a8902bda

One more thing. Instead of small 'loading' window I would use window with graphic (splash screen)
Atak_Snajpera is offline   Reply With Quote
Old 4th December 2008, 01:00   #3  |  Link
Ventolin
Registered User
 
Join Date: Oct 2001
Posts: 125
Quote:
Originally Posted by Atak_Snajpera View Post
Those two ts samples do not work (no preview)
http://www.mediafire.com/?sharekey=f...db6fb9a8902bda

One more thing. Instead of small 'loading' window I would use window with graphic (splash screen)
Hi,

This is a snippit of the status log when you try to load either of the two .ts files:

Opening: X:\tmp\Batman-5.TSSplit.1-57.ts
Opening File...
Auto Finding PIDs
Unable To Find PAT

Both streams do not have a PAT. The PAT is essential for a Transport stream and there should be one every half second or so. A transport stream should have at least 1 PAT with PID 0, 1 PMT and 1 stream (which can be video or audio).

Try loading the full source into TSPE instead.

Thanks for the note about the splash screen, I haven't gotten around to doing one yet, same with an icon. These will be added in due course, once I've learnt a bit about graphic design

Cheers,

Vent
Ventolin is offline   Reply With Quote
Old 4th December 2008, 07:19   #4  |  Link
mediator
Registered User
 
Join Date: Aug 2004
Posts: 133
many real life streams don't come with PAT (e.g. captures from DVB, seems often the PAT is just stripped during the recording process). Maybe you should consider supporting them anyway, even if they don't strictly comply with ISO.
mediator is offline   Reply With Quote
Old 4th December 2008, 09:54   #5  |  Link
Ventolin
Registered User
 
Join Date: Oct 2001
Posts: 125
Quote:
Originally Posted by mediator View Post
many real life streams don't come with PAT (e.g. captures from DVB, seems often the PAT is just stripped during the recording process). Maybe you should consider supporting them anyway, even if they don't strictly comply with ISO.
PAT and PMT really are essential to a Transport stream. If a capture program strips them out of a stream, then the capture program is at fault. TSPE would then need to repair the stream to fix the poor design of the capture program. What real life streams have you come across without PAT and PMTs? If a specific capture program is stripping out the PAT and PMT perhaps the authors could fix their software?

The PAT and PMT tell the decoder what is in the transport stream. The PAT gives a list of the Programmes available in the form of PIDs for PMTs. Once the PID of the PMT is known, the PMT can be read to find out the what PIDs, types and timing information (PCR) are of the streams that make up that programme. When capturing a complete mux you may have *hundreds* of streams in the transport stream. How is anything supposed to know what streams belong to what programme if you dont have the PAT and PMTs?

Nevertheless, in the most basic mode TSPE *still* works even without the PAT and PMT. If you are able to find a working directshow filter setup (i.e. one that works in graphedit) you can then have video preview. TSPE will still allow you to set start and end points and make an edit if you turn the fix options off with or without the video preview.

Since I couldn't get any filter combination to work in graphedit, I couldn't test what happens with TSPE completely, but I did edit a section out successfully nonetheless.

Full table editing (for PAT and PMT) is already on the TODO list however. Once that is implemented it will allow TSPE to repair streams without PAT or PMTs. Perhaps I could get TSPE to launch an external viewer like MPC instead of it's internal viewer for test edits in the mean time. I could code that in if there was enough demand...

To edit these streams "blind" switch to the decode window (using the View) button. Then use the decoded PTS/DTS timestamps as a guide to the real timecode of the stream. Then you can navigate to test out positions. Make sure you turn off Fix Tables and Fix Timecodes and turn off the internal viewer from the menu options. You can then make test and full edits and preview the filename.test.ts or filename.edit.ts files generated with something like MPC. Scanning the file revealed a timestamp gap of 49 seconds or so at the start in the Batman sample; the scan dropped a marker at the end of the gap so you can navigate there easily.

Hope that helps,

Regards,

Vent
Ventolin is offline   Reply With Quote
Old 4th December 2008, 10:23   #6  |  Link
canTsTop
Registered User
 
canTsTop's Avatar
 
Join Date: Apr 2007
Location: Lithuania
Posts: 120
hello,
i have this sample http://www.megaupload.com/?d=5A68THW9 is it possible to fix it so i have audio / video synch after encode, and how?
canTsTop is offline   Reply With Quote
Old 4th December 2008, 12:48   #7  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,528
Quote:
once I've learnt a bit about graphic design
I can create something for you for free.

Another thoughts:
1) Main form should appear in center of the screen
2) Movie slider should start at first decodable key frame instead of P or B frame

I forgot to say. Excellent job!

Quote:
i have this sample http://www.megaupload.com/?d=5A68THW9
Are you joking ? This sample is like wooden wall destroyed by Minigun. Too many holes ...

Last edited by Atak_Snajpera; 4th December 2008 at 13:09.
Atak_Snajpera is offline   Reply With Quote
Old 4th December 2008, 13:34   #8  |  Link
Ventolin
Registered User
 
Join Date: Oct 2001
Posts: 125
Quote:
Originally Posted by canTsTop View Post
hello,
i have this sample http://www.megaupload.com/?d=5A68THW9 is it possible to fix it so i have audio / video synch after encode, and how?
Woah, this is a glitchy sample!

After loading into TSPE, do a scan. This will reveal:

Continuity Errors: 764, Transport Error Indicator Errors: 0
Timestamp gaps: 16, PCR gaps: 7
Total number of separate discontinuous regions: 655

The Marker List will now contain alot of entries. Looking through there are 4 points which we can use:

A. First entry is at offset 1186092 (5 seconds in marker list)
B. Last entry is at offset 5670456 (24 seconds in marker list)
Gap in the middle between:
C. offset 2977168 (13 secs)
D. offset 5000236 (22 secs)

Every other marker entry is too close to each other to be of use, the biggest gap being about 3 seconds, with not much useable inbetween. Note that the marker list time values aren't quite accurate, once the marker is double clicked a time based on the nearest timestamp in the stream will be used (assuming timestamp mode is running which it will be by default). The byte position is the only thing we should be concerned with.

So, we need to make 3 cuts:
from start to point A.
from point C to point D.
from point B to the end

So:
1. double click the first entry, then click "Set" under End Point
2. Click on EDL tab to switch to EDL view
3. Click Add under EDL view (first entry will appear)
5. Click on Markers tab
6. Scroll to point C in Markers (look for offset 2977168 (13 secs)
7. Double click entry - this moves current position to that offset
8. Click Set under Start Point
9. Scroll to point D (offset 5000236 (22 secs))
10. Click Set under End Point
11. Click on EDL Tab to switch to EDL view
12. Click Add under EDL view (second entry will appear)
13. Click on Markers tab
14. Scroll down to the bottom of the list to the last entry (offset 5670456 (24 seconds in marker list) )
15. Double click last entry
16. Click on Set under the Start Point
17. Scroll to the very end of the video with the trackbar and using >> to make sure you're at the very end
18. Click on Set under End Point
19. Click on EDL tab
20. Click on Add under EDL view - 3rd entry should appear.

Lots of steps to create a 3 entry EDL, but it's simpler once you do it.

The EDL should now have something like this:
00:00:00.00 0 00:00:03.80 1186280
00:00:15.08 2977168 00:00:21.96 5000424
00:00:29.79 5670456 00:00:46.01 10440580

If you click Edit now, you will generate a playable, non-crashing clip. The audio cuts out a bit too much though but stays in sync.

To try and improve the audio, Uncheck Menu Edit->Clean Edit Point->Clean Audio Frames. This cuts audio at byte positions instead of looking at timestamp positions from the video stream at the expense of possible broken audio packets.

Now clicking edit produces a playable stream with better audio and we're done!

I hope the rest of the stream is clean, if you have that many errors all over the stream, it's really too damaged to be of much use.

If you're going to be re-encoding from this source, you'll have to do some more experimentation, but from a playback point of view, the edited clip is in sync.

Good Luck!

Regards,

Vent
Ventolin is offline   Reply With Quote
Old 4th December 2008, 13:34   #9  |  Link
Flaarn
Registered User
 
Join Date: Jan 2003
Location: UK
Posts: 23
Thanks for this ventolin, checking it out on a couple of h264 transport streams now, bbc hd and channel 4 hd.
The big issue for me has always been frame accurate cutting so that no extra 2.0 audio frames are present if the main broadcast is 5.1 but adverts are 2.0, many conversion tools tend to throw a wobbly when there are audio changes, more so when the transport stream contains some 2.0 ac3 frames at the start, sure it can be patched when parsing, but it adds another step in the process which in my opinion is a nuisance, with this new release I'm hoping that I can bypass this step and have a clean ts to play with, thanks again for your ongoing work.
Flaarn is offline   Reply With Quote
Old 4th December 2008, 13:51   #10  |  Link
Ventolin
Registered User
 
Join Date: Oct 2001
Posts: 125
Quote:
Originally Posted by Atak_Snajpera View Post
I can create something for you for free.
Seriously? Thanks! I can create one for free too :P but if you really want to have a go, we'd be happy to look at it

Quote:
Originally Posted by Atak_Snajpera View Post
Another thoughts:
1) Main form should appear in center of the screen
2) Movie slider should start at first decodable key frame instead of P or B frame
Thanks for your suggestions, I'll think about 1) but might be confusing with multiple instances. 2) Slider should be at the very start of the stream as this is a *packet* level editor and analyser - the PAT usually is at the first packet!

Quote:
Originally Posted by Atak_Snajpera View Post
I forgot to say. Excellent job!
Thank you kindly!

Quote:
Originally Posted by Atak_Snajpera View Post
Are you joking ? This sample is like wooden wall destroyed by Minigun. Too many holes ...
LOL, but still possible to get something out, see above

Cheers,

Vent
Ventolin is offline   Reply With Quote
Old 4th December 2008, 14:01   #11  |  Link
Ventolin
Registered User
 
Join Date: Oct 2001
Posts: 125
Quote:
Originally Posted by Flaarn View Post
Thanks for this ventolin, checking it out on a couple of h264 transport streams now, bbc hd and channel 4 hd.
The big issue for me has always been frame accurate cutting so that no extra 2.0 audio frames are present if the main broadcast is 5.1 but adverts are 2.0, many conversion tools tend to throw a wobbly when there are audio changes, more so when the transport stream contains some 2.0 ac3 frames at the start, sure it can be patched when parsing, but it adds another step in the process which in my opinion is a nuisance, with this new release I'm hoping that I can bypass this step and have a clean ts to play with, thanks again for your ongoing work.
Hi,
I've found the audio switchover point to be less than accurate on the broadcaster's end which means that you might chop off the start of the audio at the start of a segment. I hate this alot as my offboard processor will mute for a second at the switchover point. I did complain to the BBC some time ago, they tried to fix it but screwed things up alot and promptly gave up.

Hope you find TSPE useful though, look forward to your feedback.

Regards,

Vent
Ventolin is offline   Reply With Quote
Old 4th December 2008, 14:11   #12  |  Link
Atak_Snajpera
RipBot264 author
 
Atak_Snajpera's Avatar
 
Join Date: May 2006
Location: Poland
Posts: 6,528
Quote:
2) Slider should be at the very start of the stream as this is a *packet* level editor and analyser - the PAT usually is at the first packet!
I'm not saying about discarding everything before first keyframe.
Slider should be rewinded to first key-frame by default. See screenshot


I see no reason to cut stream at P or B frame. Only cutting at key frame maintains audio synchronization if we want to transcode file further. This is main reason why I use this tool.

Last edited by Atak_Snajpera; 4th December 2008 at 14:15.
Atak_Snajpera is offline   Reply With Quote
Old 4th December 2008, 14:40   #13  |  Link
canTsTop
Registered User
 
canTsTop's Avatar
 
Join Date: Apr 2007
Location: Lithuania
Posts: 120
Quote:
Originally Posted by Ventolin View Post
Woah, this is a glitchy sample!

After loading into TSPE, do a scan. This will reveal:

Continuity Errors: 764, Transport Error Indicator Errors: 0
Timestamp gaps: 16, PCR gaps: 7
Total number of separate discontinuous regions: 655

The Marker List will now contain alot of entries. Looking through there are 4 points which we can use:

A. First entry is at offset 1186092 (5 seconds in marker list)
B. Last entry is at offset 5670456 (24 seconds in marker list)
Gap in the middle between:
C. offset 2977168 (13 secs)
D. offset 5000236 (22 secs)

Every other marker entry is too close to each other to be of use, the biggest gap being about 3 seconds, with not much useable inbetween. Note that the marker list time values aren't quite accurate, once the marker is double clicked a time based on the nearest timestamp in the stream will be used (assuming timestamp mode is running which it will be by default). The byte position is the only thing we should be concerned with.

So, we need to make 3 cuts:
from start to point A.
from point C to point D.
from point B to the end

So:
1. double click the first entry, then click "Set" under End Point
2. Click on EDL tab to switch to EDL view
3. Click Add under EDL view (first entry will appear)
5. Click on Markers tab
6. Scroll to point C in Markers (look for offset 2977168 (13 secs)
7. Double click entry - this moves current position to that offset
8. Click Set under Start Point
9. Scroll to point D (offset 5000236 (22 secs))
10. Click Set under End Point
11. Click on EDL Tab to switch to EDL view
12. Click Add under EDL view (second entry will appear)
13. Click on Markers tab
14. Scroll down to the bottom of the list to the last entry (offset 5670456 (24 seconds in marker list) )
15. Double click last entry
16. Click on Set under the Start Point
17. Scroll to the very end of the video with the trackbar and using >> to make sure you're at the very end
18. Click on Set under End Point
19. Click on EDL tab
20. Click on Add under EDL view - 3rd entry should appear.

Lots of steps to create a 3 entry EDL, but it's simpler once you do it.

The EDL should now have something like this:
00:00:00.00 0 00:00:03.80 1186280
00:00:15.08 2977168 00:00:21.96 5000424
00:00:29.79 5670456 00:00:46.01 10440580

If you click Edit now, you will generate a playable, non-crashing clip. The audio cuts out a bit too much though but stays in sync.

To try and improve the audio, Uncheck Menu Edit->Clean Edit Point->Clean Audio Frames. This cuts audio at byte positions instead of looking at timestamp positions from the video stream at the expense of possible broken audio packets.

Now clicking edit produces a playable stream with better audio and we're done!

I hope the rest of the stream is clean, if you have that many errors all over the stream, it's really too damaged to be of much use.

If you're going to be re-encoding from this source, you'll have to do some more experimentation, but from a playback point of view, the edited clip is in sync.

Good Luck!

Regards,

Vent
Thank You for answer.

This sample intentionally so damaged

If i correctly understand this method cuts out damaged parts off file. I did exactly as you described (with Clean Audio Frames unchecked) and it kind off worked (DGAVCIndex didn't crash). But if i again open already edited (fixed) TS file in TSPE, scan, and there is Continuity Errors: 17. It would be good to have some kind of automation of this method if for example there is 3 hours recording where we need to do ~20 cuts


Is this is only way to fix damaged TS files? Maybe something like ProjectX, it replaces damaged frames with empty frames?
canTsTop is offline   Reply With Quote
Old 4th December 2008, 15:36   #14  |  Link
Ventolin
Registered User
 
Join Date: Oct 2001
Posts: 125
Quote:
Originally Posted by Atak_Snajpera View Post
I'm not saying about discarding everything before first keyframe.
Slider should be rewinded to first key-frame by default. See screenshot


I see no reason to cut stream at P or B frame. Only cutting at key frame maintains audio synchronization if we want to transcode file further. This is main reason why I use this tool.
Lets take the following two scenarios:

TV capture:
When I capture a TV programme I like to start capturing a few minutes before the start of the actual programme itself - that way I won't miss the very start if it's broadcast early. Then when I load the captured .ts file into TSPE I look for the start of the programme. This almost never is at the first I frame, but usually a minute or two into the file. So I dont really need to look at the first I frame. However, I might be interested at looking at the PAT and PMTs which could be at the very start of the stream.

BD analysis:
Suppose I want to look at a stream from a BD. Again the first few packets are the PAT/PMT, followed by an I-Frame for the start of the video. In this situation, I am already at the point I want to edit with.

So, I think in most situations starting from the very beginning of the stream is correct?

I guess the only time I'd want the very first I frame is if I have missed the start of the captured programme, in which case it's a simple case of navigating to the first I frame.

In any event, it's certainly possible to make sure the edit always starts with an I frame so you don't have to look for it manually. I'll look into that possiblity further, thanks

Regards,

Vent
Ventolin is offline   Reply With Quote
Old 4th December 2008, 15:48   #15  |  Link
Ventolin
Registered User
 
Join Date: Oct 2001
Posts: 125
Quote:
Originally Posted by canTsTop View Post
Thank You for answer.

This sample intentionally so damaged

If i correctly understand this method cuts out damaged parts off file. I did exactly as you described (with Clean Audio Frames unchecked) and it kind off worked (DGAVCIndex didn't crash). But if i again open already edited (fixed) TS file in TSPE, scan, and there is Continuity Errors: 17. It would be good to have some kind of automation of this method if for example there is 3 hours recording where we need to do ~20 cuts


Is this is only way to fix damaged TS files? Maybe something like ProjectX, it replaces damaged frames with empty frames?
I think you are looking for similar functionality to Mpeg2Repair where bad mpeg2 frames are repaired. Whilst this is possible, it's certainly going to take some time to implement, especially as M2R is such an awesome tool and I'm still developing my c0dering skill0rz It would require a detailed knowledge of mpeg2/h264/vc1 encoding.

For playback purposes, having minor glitches in place is preferable in my opinion to missing a chunk of the programme.

TSPE actually does do some repair automatically eg. timestamp gaps, so for some glitches it is automated. It all depends on the type of glitch and severity.

Cheers,

Vent
Ventolin is offline   Reply With Quote
Old 13th December 2008, 15:43   #16  |  Link
JonRead
Registered User
 
Join Date: Sep 2002
Posts: 217
Thanks, I will test out with UK satellite broadcast HD h.264 streams and let you know if I find any bugs.
JonRead is offline   Reply With Quote
Old 18th December 2008, 19:36   #17  |  Link
robinsj
Registered User
 
Join Date: Jan 2002
Posts: 62
In your editor is there a way to manually enter a time you want to go to in a video?
robinsj is offline   Reply With Quote
Old 20th December 2008, 20:04   #18  |  Link
Ventolin
Registered User
 
Join Date: Oct 2001
Posts: 125
Quote:
Originally Posted by JonRead View Post
Thanks, I will test out with UK satellite broadcast HD h.264 streams and let you know if I find any bugs.
Hi, Please do let me know how you are getting on!

Vent
Ventolin is offline   Reply With Quote
Old 20th December 2008, 20:05   #19  |  Link
Ventolin
Registered User
 
Join Date: Oct 2001
Posts: 125
Quote:
Originally Posted by robinsj View Post
In your editor is there a way to manually enter a time you want to go to in a video?
I'm just going to add that in right now, this very minute! Well OK, maybe the next minute or two as I'm still typing

Cheers,

Vent
Ventolin is offline   Reply With Quote
Old 26th December 2008, 20:05   #20  |  Link
robinsj
Registered User
 
Join Date: Jan 2002
Posts: 62
Okay another question.... I am trying to edit a few movies that I have that my wife really likes, but she really hates hearing the F-word in movies.... So I am trying to use your software to edit them out. (not really interested in getting into an arguement about editing a movie to take out language, so please leave that alone to whomever is interested in arguing)

So what is the best way to do it..... Here is what I have tried to do... I try playing the movie in the preview window, and then I just hit pause right before he says it and mark that as the end, then add that to the edits. Then I play it again until it has passed that he said it, usually only is like half a second or so... and then mark that as the beginning, and then go all the way to the end of the movie and mark that as the end and then add that to the edits. So basically I have a very small portion that is maybe 15-25 frames long that is being cut out. Well, then when I edit it and go to play it, it seems it totally misses the swear word and cuts something else out within frames from where i was trying to cut. Does this have to do with the PID's? Or the I-Frames? I tried unselecting Fix tables and timecodes, and that seemed to work, but then all I got was digital snow instead of just a clean cut...??

Any Help?
robinsj is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 00:42.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.