Sync with TS subtitles and dropped frames

[alex22]

Hi,
I'm using DGIndex/DGDecode (through AutoGK) on MPEG2 transport streams coming from a Dreambox. This works very well: thanks !

Now I also extract subtitles from the TS, using Dave Chapman's dvbtextsubs. This tools decodes the teletext substream, and extracts the page holding the subtitles. The timestamps found in these packets are output (substracting the initial value) in the sub file. Hence, the subtitle times are relative to the initial broadcast program's timeline.

Now, if there is any frame-dropping done by DGIndex/DGDecode, the final AVI's timeline no longer matches the initial program's. This can be witnessed as subtitles becoming late through the length of the movie (by discrete increments corresponding to spots of frame-dropping).

Is there any way to:
(1) use the information in the .d2v to know where frame-dropping occurs
(2) or extend it with video/audio timestamp data in order to deduce the frame-dropping spots (and their size) on the time axis
?

Thanks in advance for any insight.

Best regards,

-Alex

[neuron2]

DGMPGDec does not drop frames.

Would you care to clarify your question?

[alex22]

> DGMPGDec does not drop frames.

Well, the assumption was based on len0x's remarks in a similar thread in the AutoGK forum:

http://forum.doom9.org/showthread.php?s=&threadid=88748

> Would you care to clarify your question?

Sure: some frames are invalid because of corrupted TS packets (the satellite signal is not perfect). In fact, the corrupted packet may even lack the TS sync byte (0x47), hence the routine which is reading TS in DGIndex must skip them, looking for resync.

The problem is, in the original program these were actual video and audio data. Hence, the initial and final timelines do not match. Hence nobody can currently get a perfectly timed subtitle out of a tansport stream. Sure, we can fix it manually with Sub Workshop et al, but it is slow and difficult: even a careful tuning with 2 sync points still shows abrupt drift around the drop points.

However, there's hope: audio and/or video data in a TS come with timestamps. Hence a TS-scanning tool like DGIndex could detect the discontinuities, and (say) output them in a separate text file. This would solve the whole problem in an elegant way !

[neuron2]

Can you please directly and specifically state what you are asking for?

Demux subtitles?

Dump PTS values?

What???

[Cyberia]

I think he's asking for a way to keep subtitles in-sync even if DGIndex drops/skips a frame (for any reason).

Since subtitles are displayed acording to timestamps, missing frames desync the subtitle stream. So, to resync the streams, he somehow needs information about what frames are missing.

But can frames go missing? Wouldn't that throw off the audio also? Do we make audio 'adjustments' if it does?

DGIndex could demux subtitles. It would be the right place to do it. But subs are a hornets nest of types and flavors. Do you want to do this (yet, ever?)

[neuron2]

I know what he wants. I was just trying to brush him off. :p

He needs a program that demuxes everything together and drops everything together between transport sync loss and regaining. Think of subtitles as just another kind of audio. :eek:

His problem is that I have zero interest in subtitles. Sorry, Alex!

Exactly the same thing can happen with audio. That is why I have that issue listed on the work list (Enhancements 10 b). But I don't see adding subtitle demuxing any time soon.

As long as we can track the program clock reference (PCR), we can detect missing video and audio and we can add extra video and audio to keep the sync. I'm more inclined to write a stand-alone program to fix up an errored stream like that (for all PIDs, including subtitles) than to add this to DGIndex.

[alex22]

>He needs a program that demuxes everything together and drops
>everything together between transport sync loss and regaining.
>Think of subtitles as just another kind of audio

No, dropping the subtitles won't help. In most cases there is no subtitle event in the dropped part, while the time shift induced by the audio/video dropping affects all subsequent subtitles.

> But I don't see adding subtitle demuxing any time soon.

Nor was I asking for it ! Dvbtextsubs does the job pretty well, it only needs to know the "time discontinuities" that DGIndex observes on the PTS of audio and video data. Hence I'm not asking DGIndex to demux anything more than it already does, that's the point !

>As long as we can track the program clock reference (PCR), we can
>detect missing video and audio and we can add extra video and audio
>to keep the sync. I'm more inclined to write a stand-alone program
>to fix up an errored stream like that (for all PIDs, including
>subtitles) than to add this to DGIndex.

*This* would solve the problem by restoring the original timeline, but from a developer's standpoint it looks like an overkill. But you choose, of course:)

(Alex) Just add to DGIndex a printf() to a secundary output file every time missing audio/video are detected.

(Donald) Write an external program that demuxes a TS like DGIndex, extracts PTSs like DGIndex, finds missing frames, fills them with dummy data (e.g. repeated packets ? or zero-motion vectors and silence ?), and remuxes them all, for DGIndex to demux again...

Verdict ?

-Alex

[neuron2]

DGIndex is not architected to easily do what you ask for. If it was, I would not have proposed the alternative. Believe it or not, I am very familiar with MPEG transport and the internals of DGIndex.

Verdict: The source code is there if you think I am wrong.

[ydobon]

Hello,

I'm also interested in extracting subtitles from a .ts recording. I'm lost about most of what you discuss (I haven't looked at the internals of a .ts) so forgive my naivety.

Wouldn't it be enough information if (in this "enhancement 10b") DGIndex would write a log of what repairing it had to do? I think with that the original subs could be fixed.

Or does anybody know of any other tool that already does that?


Regards,

[neuron2]

DGIndex does not do any repairing.

[alex22]

Originally posted by neuron2
DGIndex is not architected to easily do what you ask for.
Verdict: The source code is there if you think I am wrong.

Okay, I'm looking at the code, and I don't see any kind of audio/video resync beyond the global offset (delay). Am I right ?

Let's rephrase the question (since I know you'll ask me to anyhow :)
Consider a transport stream with the following structure:

V1V1V1V1A1A1A1A1V2V2V2V2A2A2A2A2V3V3V3V3A3A3A3A3

here the Ai are the audio frames associated with the video frames Vi.
Now suppose there's corruption, and we get

V1V1V1V1A1A1A1A1xxxxxxxxA2A2A2A2V3V3V3V3A3A3A3A3

Can you explain what DGIndex does in this case ?
Or does the above example yield a desync (the audio being late by 4 frames starting with A3) ?

[neuron2]

You will get a desync.

DGIndex began as a tool for backing up DVDs. It is not architected for the repair and recovery you are quite rightly looking for. Can it be added? With great difficulty. But a complete rewrite would be much more sensible. You are better off looking for support from tools that were designed from the ground up for dealing with errored transport streams.

[alex22]

Okay, thanks for the clear statement.
Do you have a suggestion fo such other tools ? (PVAS is closed-source)

But let me just remind you that data corruption is a fact of life with transport streams, and that the desync occurs ouside any concern with subtitles... hence you're bound to encounter "normal" users complaining about apparently random audio desync one day or the other.

Unless, of course, you brush them off by saying:

> Your problem is that I have zero interest in transport
> streams. Sorry, dude !

:devil:

[neuron2]

Originally posted by alex22
Okay, thanks for the clear statement.
Do you have a suggestion fo such other tools ? (PVAS is closed-source) I don't have any suggestion for you. Sorry.

But let me just remind you that data corruption is a fact of life with transport streams, Hey Alex, do you think I forgot since the last few posts to this thread? Or are you just being condescending?

I write software for stream parsing for all the major providers, DirecTV, Dish, etc. Do you think I'm an idiot and don't know all about practical issues? Here, I'll put it in bold for you:

DGIndex is not the approriate base for an error-tolerant transport stream parser. This does not mean that I think error-tolerant parsing is not important.

and that the desync occurs ouside any concern with subtitles... hence you're bound to encounter "normal" users complaining about apparently random audio desync one day or the other. The normal use of DGIndex is for uncorrupted streams.

Unless, of course, you brush them off by saying:

> Your problem is that I have zero interest in transport
> streams. Sorry, dude ! Now you are putting words in my mouth that I did not say and verging on total disrespect for me. I have said that I am considering developing another tool to address the issues.

I suggest that you consider adjusting your attitude a little bit.

[alex22]

>> But let me just remind you that data corruption is a fact of life
>> with transport streams,

>Hey Alex, do you think I forgot since the last few posts to this
>thread? Or are you just being condescending?

Not at all, I wanted to point out that "focusing on uncorrupted streams" is just like "assuming your LAN never has a single collision". The transport stream structure was designed specifically for robustness to transmission errors (yes I know you know and am not insulting you), hence it is a strange positioning for a TS-reading tool to ignore the case.

Yes, you acknowledged that the function is needed and would best fit into a separate tool. I'm just saying that, then, it would be nice to warn the users that TS support is meant only for crystal-clean transports.

>I write software for stream parsing for all the major providers,
>DirecTV, Dish, etc. Do you think I'm an idiot and don't know all
>about practical issues?

Not at all. But maybe you underestimate the average error rate on TS in the whole population of your users. And you wouldn't deserve criticism for that until statistics are available. Since TS input is fairly recent in DGIndex's history, I guess it's a bit early. But I hoped requests like mine would raise your interest ahead of large-scale user pressure...

> > Your problem is that I have zero interest in transport
> > streams. Sorry, dude !

> Now you are putting words in my mouth that I did not say and verging > on total disrespect for me

I beg your pardon. If you please scroll up into this thread, you'll see these were your very words, replacing 'transport streams' by 'subtitles'.

And talking about disrespect, I might have taken offense of the "don't disturb the Prince" way you handled my first posts ("...tried to brush him off"). Equal score ;-)

All in all, thanks for the piece of insight you ended up giving me, despite the emotional cost.

[neuron2]

Originally posted by alex22
I beg your pardon. If you please scroll up into this thread, you'll see these were your very words, replacing 'transport streams' by 'subtitles'. Which means that you have just acknowledged putting words into my mouth that I did not speak.

Hey, Alex, do you know that this stuff is hard? Do you know that I don't get paid to do it? Do you know that I don't owe you or anybody else here a damn thing? If you want a tool and it doesn't exist, do you think it is the right approach to go and start bullying someone that is already giving all he can to free tools? Hey, I'm sorry that the freeware tool support isn't yet perfect according to Alex the Great. But, you know what, I'm not going to lose any sleep over it.

[Cyberia]

Ok this has gone on long enough. I am closing this thread.