Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > HDTV / DVB / TiVo

Reply
 
Thread Tools Search this Thread Display Modes
Old 7th May 2010, 02:17   #1  |  Link
dumbledore
Registered User
 
Join Date: Apr 2010
Posts: 14
tscombine: a simple tool to combine two transport streams for error reduction

hi there

i wrote this tool a few months ago and feel that it's become pretty usable now, so i thought i would post it here. i wrote it because my clear-qam cable captures had annoying glitches in them here and there, and it was very difficult for me to get an error-free stream capture. likewise, over-the-air atsc had the same problem. however since both streams were from the same source, i was able to write a tool to detect and correct what errors it could at the transport stream level, resulting in a (potentially) bit-exact replication of the source broadcast.

to use the tool:
take two recordings of the same showing (the tool will cut off any leading and trailing parts which don't occur in both recordings), and run "tscombine.exe file1.ts file2.ts out.ts". it will provide information regarding progress as it processes the files.

i'm also going to include the source code, in the hopes that if someone else finds this useful, they can develop it further. i've made a few comments in the source as far as a few ideas of my own for improvements, but as it servers my current needs fine already i'm unmotivated to actually code them . my only request is that if you do change the code to something better, that you post the code changes in this thread so others can benefit as well.

one final note: it should be possible to recompile this into linux without too much difficulty (i originally developed it on an ubuntu box).
Attached Files
File Type: zip tscombine.zip (19.1 KB, 254 views)
dumbledore is offline   Reply With Quote
Old 7th May 2010, 14:15   #2  |  Link
Gser
Registered User
 
Join Date: Apr 2008
Posts: 415
I actually had this idea a while ago too. Brilliant minds think a-like. Will this work with h.264/avc? And how does it detect errors? I dont suppose it could be retro-fitted to detect macro-blocking too.
Gser is offline   Reply With Quote
Old 7th May 2010, 15:41   #3  |  Link
dumbledore
Registered User
 
Join Date: Apr 2010
Posts: 14
it uses a couple methods to detect errors. firstly, the transport stream multiplexer is _supposed_ to set a bit in a packet indicating if there was a transmission error. however it seems this doesn't always happen, so additionally it detects if there's a mismatch between the two streams, in which case it will choose the stream with the least amount of missing packets until it can re-sync them.

[edit] also, since this works on the transport stream level, it supports any codec which you can mux into a .ts. so mpeg2, avc, ac3, etc.
dumbledore is offline   Reply With Quote
Old 7th May 2010, 18:49   #4  |  Link
b66pak
Registered User
 
b66pak's Avatar
 
Join Date: Aug 2008
Location: The Land Of Dracula (Romania - EU)
Posts: 934
thanks a lot...
_
__________________
if you ask a question and somebody give you the correct answer don't forget to leave a "thank you" note...
Visit The Land Of Dracula (Romania - EU)!
b66pak is offline   Reply With Quote
Old 8th May 2010, 14:19   #5  |  Link
Ghitulescu
Registered User
 
Ghitulescu's Avatar
 
Join Date: Mar 2009
Location: Germany
Posts: 5,768
Does tscombine make use of a sort of internal synch in aligning the two recordings? The real life situation says that the two recordings won't start exactly at the same microsecond, and if the studios reencode the final output before airing it, the streams won't have the same structure....
Ghitulescu is offline   Reply With Quote
Old 8th May 2010, 15:21   #6  |  Link
dumbledore
Registered User
 
Join Date: Apr 2010
Posts: 14
right, in this case a much more involved restructuring would be needed, and probably a re-encoding if you wanted the final output to be a .ts file. (not to mention would probably significantly increase the combining time...) this is for multiple recordings of the same source, such as atsc over-the-air + clear-qam rebroadcasts, or multiple over-the-air recordings of the same station. it is able to tolerate PID differences and will automatically map the differing PIDs to the correct values. one of the possible improvements i think i mentioned in the code would be for if the packets were aligned differently, or if they were multiplexed in a different order. fortunately in my tests it seems cable companies don't do this.
dumbledore is offline   Reply With Quote
Old 25th September 2010, 04:40   #7  |  Link
dumbledore
Registered User
 
Join Date: Apr 2010
Posts: 14
i've been working on this bit by bit... one thing i changed was that it seems sometimes the error flag isn't marked on a packet that had an error, so in the version i posted here i hardcoded something like every time it sees a packet with errors, it drops 2 packets before and 3 after, to be safe... however with some streams this isn't enough, so i added a cmdline parameter to specify how big a buffer size you want.

one thing i keep getting though is that sometimes it syncs to the incorrect position in the files... all i can guess is that it's finding that some generic dummy packets match up before the actual bad spot is finished, and this causes problems... i was thinking of making it match at least 2 in a row before determining that the error region is finished. unless anyone has a better idea?
dumbledore is offline   Reply With Quote
Old 26th September 2010, 02:04   #8  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,867
Very cool idea

I don't personally have use for it, but cheers for releasing this

Derek
__________________
These are all my personal statements, not those of my employer :)
Blue_MiSfit is offline   Reply With Quote
Old 26th September 2010, 15:58   #9  |  Link
Ghitulescu
Registered User
 
Ghitulescu's Avatar
 
Join Date: Mar 2009
Location: Germany
Posts: 5,768
A TS is supposed to be tolerant to transmission errors. First thing to do is to reorder the packets in their native order, only then perform the "puzzling" (the compare function should work, at least better than before).
Ghitulescu is offline   Reply With Quote
Old 28th September 2010, 07:24   #10  |  Link
dumbledore
Registered User
 
Join Date: Apr 2010
Posts: 14
Quote:
Originally Posted by Ghitulescu View Post
A TS is supposed to be tolerant to transmission errors. First thing to do is to reorder the packets in their native order, only then perform the "puzzling" (the compare function should work, at least better than before).
this would essentially mean demux the ts first, correct? the problem i see with doing that is you might lose audio sync, since i think the order of packets in the ts determines sync points... on the other hand, testing the data for decodability would be a more accurate indicator of the presence of errors. (and a lot more work, as then it would need to know how to decode codecs generally muxed into ts)
dumbledore is offline   Reply With Quote
Old 20th October 2010, 21:41   #11  |  Link
dumbledore
Registered User
 
Join Date: Apr 2010
Posts: 14
i added additional checks to make sure it only copies streams with pid entries in the program map table. (sometimes muxers screw up and throw in random streams apparently). also i put in the part about matching up to k packets (k being some user inputted number) to determine a resync instead of just one. i would upload the new code but there's still something going horribly wrong sometimes, where it resyncs and copies over the data from one stream, but then somehow ends up copying the second streams data after that (it should simply skip over it), and i haven't been able to find where the problem is. (if someone here would be willing to take a look at it, i could upload the buggy code). if not then i'll upload it when i get around to fixing the problem
dumbledore is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 22:08.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.