Problem with understanding packet PTS vs. frame PTS in M2TS files

[Unforgettable]

Dear all,

I am currently playing around with Bluray / M2TS to MKV conversion, where I am especially interested in generating MKV files with the help of external timestamp files. To create those timestamp files, I need to extract the PTS for every video and audio frame from the M2TS files.

Accidentally, I used something like the following command to do that:

ffprobe -show_entries packet=stream_index,pts,duration -of compact=p=0:nk=1 input.m2ts > timestamps1.txt

Then I thought about it and came to the conclusion that I probably did it wrong, because I obviously extracted PACKET timestamps instead of FRAME timestamps, while I actually assumed that a packet may contain more than one frame of the same stream. However, the MKV file that I created using the timestamps in timestamps1.txt was correct in every respect.

Nevertheless, I tried to correct my mistake and used something like the following command:

ffprobe -show_entries frame=stream_index,pts,duration -of compact=p=0:nk=1 input.m2ts > timestamps2.txt

That is, I now actually extracted the frame PTS instead of the packet PTS.

I then was baffled: Apart from the fact that this took an extremely long time, the timestamps in timestamps2.txt were the same as the timestamps in timestamps1.txt (at least the audio and video timestamps; let's put subtitles aside for now).

I would appreciate help with understanding this. Until now, I was assuming that a M2TS file basically consists of packets, and that a packet basically consists of an arbitrary number of frames that belong to arbitrary streams, and notably, that a packet may contain multiple frames that belong to the same stream.

But obviously, the result described above implies that there is no more than one frame of each stream in a packet. I would be grateful if somebody could explain in simple words whether it is pure random that the M2TS files I have tested so far contain no more than one frame per stream in each packet, or whether this is a general property of a M2TS file.

Thank you very much in advance!

[cubicibo]

Transport Streams are indeed a chain of packets ("Transport Packets" – TP). But a Transport Stream is meant to robustly carry data from one point to the other. It is not meant for instantaneous decoding and presentation, just transmission. There's no concept of PTS at the transport packet abstraction level.

The TPs that contain audiovisual data carry payloads. These are generally small pieces of data of a larger "PES Packet". A PES packet contain the PTS of the frame/audio/subtitle.
Essentially, transport streams are used to transmit PES packets, in small chunks. TS encapsulation is robust to noise and clock drift. PES packets are not, but contain complex data suited to the target decoder.
Your Blu-ray player/TV essentially reads transport packets and put the payloads in the respective (video, audio, subtitle) elementary buffers. When it has processed enough TPs, a PES Packet containing (video, audio, subtitle) data is ready for decoding and presentation. FFmpeg does the same, it concatenates the TP payload to produce PES packets and get the PTS from those, and that is the only thing it can do to get the PTS.

By the way, with H.264, the PTS may not be monotonic: the order of the frames in the stream is not linear and depends on frame prediction mode of the encode. I do not know if FFmpeg reorders them or not when you dump the timestamps.

[Unforgettable]

Thank you very much for the in-depth explanation!

Let me summarize: A PES packet consists of TS packets. A PES packet has an associated PTS, while a TS packet hasn't. Hence, if I execute

ffprobe -show_entries packet= ...

, it outputs the PES packet timestamps.

But then I have one question left:

If I execute

ffprobe -show_entries frame= ...

, the process takes at least 10 times longer, but in my case, the outcome seems to be the same (except for subtitles, which are a problem on their own). What is the difference between -show_entries frame=... and -show entries_packet= ...? Does one PES packet always contain no more than one audio, video or subtitle frame?

Regarding the last paragraph in your reply: ffprobe does not reorder the video frame timestamps by default (but I don't know whether there is a command line option to do that).

[cubicibo]

I do not know what FFmpeg is doing; I would assume with packet it just gets the PESpacket PTS and does not look at the content, while with frame it is forced to parse the entire data to produce the frame itself. I suspect frame will always output the correct frame order/PTS timeline, while packet can suffer from out of order presentation timestamps.