How to extract and calculate the framerate of the video in a mp4 container

[TEB]

hi, afaik, the mp4 container doesnt have a absolute moov atom describing the actual framerate of the video within, but its rather calculated based on the PTS/DTS timing values.
Are there tools, pref opensource, that can show the PTS/DTS values and do this calculation?
I know mediainfo, ffprobe etc can do it, but i want to understand where in the mp4 container this data resides.. ;)

[butterw2]

maybe https://github.com/nu774/mp4fpsmod
Tiny mp4 time code editor. You can use this for changing fps, delaying audio tracks, executing DTS compression, extracting time codes of mp4.

[videoh]

The frame rate can be calculated from the number of samples and the duration:

u_int32_t mp4_numSamples = MP4GetTrackNumberOfSamples(mp4File, trackId);
MP4Duration duration = MP4GetTrackDuration(mp4File, trackId);
double msDuration = (double)(MP4ConvertFromTrackDuration(mp4File, trackId, duration, MP4_MSECS_TIME_SCALE));
double mp4_Frame_Rate = mp4_numSamples * 1000LL / msDuration;

It may also be specified in the video ES.

[benwaggoner]

The frame rate can be calculated from the number of samples and the duration:

u_int32_t mp4_numSamples = MP4GetTrackNumberOfSamples(mp4File, trackId);
MP4Duration duration = MP4GetTrackDuration(mp4File, trackId);
double msDuration = (double)(MP4ConvertFromTrackDuration(mp4File, trackId, duration, MP4_MSECS_TIME_SCALE));
double mp4_Frame_Rate = mp4_numSamples * 1000LL / msDuration;

It may also be specified in the video ES.
And that's the problem. It can be specified in multiple places, and there's no enforcement mechanism to make sure they even match.

The ES is generally what players go by, so it'd go for that. The MP4 headers can be wrong, and often give the average fps for variable frame rate content, which isn't that helpful.

[SeeMoreDigital]

The ES is generally what players go by, so it'd go for that. The MP4 headers can be wrong, and often give the average fps for variable frame rate content, which isn't that helpful.
Indeed... Even back in the old MPEG-4 Part 2 encoding days, it soon became apparent after .mp4 container support was added to hardware playback devices, that the fps data needed to be added to the elementary video stream, not the container. Same too with the aspect ratio signalling.

[benwaggoner]

Indeed... Even back in the old MPEG-4 Part 2 encoding days, it soon became apparent after .mp4 container support was added to hardware playback devices, that the fps data needed to be added to the elementary video stream, not the container. Same too with the aspect ratio signalling.
I remember with WMV that there were THREE different places that picture aspect ratio could be stored, without any clear rules as to which would take precedence.

Were I God of Video, I'd say that the elementary stream would always take precedence, and it's the job for each layer above to conform to it.

[rwill]

Indeed... Even back in the old MPEG-4 Part 2 encoding days, it soon became apparent after .mp4 container support was added to hardware playback devices, that the fps data needed to be added to the elementary video stream, not the container. Same too with the aspect ratio signalling.

Well yes and no. There is the case with fixed frame rate, then ES and container should match. Then when a video is missing frames the duration a picture might be displayed is a multiple of a picture 'tick's, this is also the case when there are transmission errors. Last case is totally variable frame rate, then timing should just be in the container.

[TEB]

Soo, how can i extract the ES AR and ES Framerate? Or rather, where exact in the ES does this info recide?

[videoh]

It depends on the video format, i.e., AVC, HEVC, MPEG2, or VC1. Did you have one in mind? ES just means what you would get if you demuxed the video. You can consult the appropriate video specification. They are readily available. I'll help if you specify your format.

Be warned: there may not be a specification in the ES for any given file.

[SeeMoreDigital]

...Last case is totally variable frame rate, then timing should just be in the container.Hmmm...

When it comes to generating encodes suitable for playing in a 'hardware playback device', you should never create files with a variable frame rate, regardless of where the frame rate information is stored.

[TEB]

It depends on the video format, i.e., AVC, HEVC, MPEG2, or VC1. Did you have one in mind? ES just means what you would get if you demuxed the video. You can consult the appropriate video specification. They are readily available. I'll help if you specify your format.

Be warned: there may not be a specification in the ES for any given file.

HEVC and AVC! ;)

[Ritsuka]

If a player ignores the container PTS/DTS, then that player is hopelessly broken.

videoh code is not entirely correct. MP4 has a feature called "edit lists", and the tkhd box duration is equal to the sum of the durations of all of the track’s edits. But the edits can use less the actual samples in a track, so you can't divide it by the number of samples of the track.

To get the actual frame rate, you would need to calculate the actual presentation timestamp of each sample included in an edit list, and them make an average.
And edit list can change the rate at which samples at played or be a "dwell" without any sample for extra fun.

Refer to the free ISO/IEC 14496-12 Fifth edition pdf for a description of all this fun little features.

[benwaggoner]

Yeah, the MPEG-4 file format is derived from the QuickTime file format, which was Turing Complete! Modern implementations don't support all of the late 90's interactive features of QuickTime, but the file format was capable of things akin to what Director or HyperCard could create.

[Ritsuka]

Yes, there was some weird stuff in there. But *edit lists* are a basic feature that should be required, but unfortunately in the real world is not always implemented.

[benwaggoner]

Yes, there was some weird stuff in there. But *edit lists* are a basic feature that should be required, but unfortunately in the real world is always implemented.
OMG, I know. I used to get .mov mezzanine files all the time where an operator trimmed out leader or something and did a Save instead of Save As. It was a coin toss whether any given video tool would read the edit list, or just grab the whole elementary stream. <shudder>.

More a technology can do same thing in different ways, the more implementations will wind up diverging. That was one nice thing about AVI as a format - it was so brain-dead simple it was really hard to get into trouble with it.