Truly variable framerate issues

[hack41]

I'm not sure that this is the correct forum, so please tell me if I'm off. Also, this isn't really a DVD-related question, but I figured that I'd get the most expert audience here. So, here's my issue:

How do I deal with truly variable framerates?

Let me explain. What I don't mean is things like mixed 24fps/30fps videos -- those are very special (since they come from cinema or TV) and lots of solutions are known for handling those. What I have in mind are captured videos from, say, 3D computer games. What happens is that the actual framerate that I capture may be completely random, and dependent solely on the performance of my graphics card. I'd like to see if anyone has experience with that. Some specific questions:

Are there any video codecs or containers that can deal with variable framerate? Is this a codec or a container question at all? Do I have a variable framerate, or is the concept of "frame rate" not even appropriate and I should timecode each frame individually? Is that possible?

Which codecs/containers are good for this? I know that WMV has some sort of fmv support, but I don't know if it's just faking. AVI apparently definitely has no real support for fmv but can "fake" it by taking the highest common multiple of all framerates and inserting empty frames as needed. Finally, the MKV container apparently can use custom timecodes and have framerate free videos altogether.

Which tools are available for dealing with arbitrary/no-framerate videos? Is mplayer/mencoder any good? (I can't even get mplayer to print out the running frame number.) How can I convert a fake fmv AVI or WMV to, say, a no-framerate MKV?

How do I deal with the sound and A/V sync?

I know my questions are pretty vague, but there seems to be very little information about this on the internet, so I'd be happy about any kind of information that you may have.

Thank you!

[poisondeathray]

What I have in mind are captured videos from, say, 3D computer games. What happens is that the actual framerate that I capture may be completely random, and dependent solely on the performance of my graphics card. I'd like to see if anyone has experience with that.

While your gameplay may have variable frame rates , the recorded footage usually has contant frame rates. Even if you get say, 40-200 fps in game, you might cap at a constant 30fps. I don't know of any that common game capture hardware or software that cap at non constant framerates.

Usually the only time you have sync issues is when your system encounters a bottleneck and drop frames (e.g. might be a storage IO bottleneck, or CPU bottleneck)

[hack41]

While your gameplay may have variable frame rates , the recorded footage usually has contant frame rates. Even if you get say, 40-200 fps in game, you might cap at a constant 30fps. I don't know of any that common game capture hardware or software that cap at non constant framerates.
Hm, OK, that may be true, though the cap seems to apply only "from above", and if I have lower output than the fixed framerate, I don't even know what kind of video I'll get as the result. (Will the capturer repeat frames to fill the gap? will there be a desync?)

But that's not really my point, so forget about the video games. Suppose I really just have some abstract video source with a non-fixed framerate (e.g. a novelty codec that saves bandwidth during slow scenes).

[poisondeathray]

Yes, you will get frame repeats, because the actual gameplay fps than the recording fps

For other sources, you would use timecodes to make VFR video ; and yes it is a container issue, because only a few support timecodes (e.g. wmv, mkv, flv ,mp4, might be a few others...)

[roozhou]

Yes, you will get frame repeats, because the actual gameplay fps than the recording fps

For other sources, you would use timecodes to make VFR video ; and yes it is a container issue, because only a few support timecodes (e.g. wmv, mkv, flv ,mp4, might be a few others...)
You don't need timecodes to "make" VFR video. Actually all the containers listed here natively supports VFR and stores PTS for each frame. Most capture software output only AVI, but even AVI should handle frame dropping easily by inserting null frames.

If you are going to encode captured video later, use x264(don't use avisynth!). It supports vfr natively as well.

Timecodes is only needed when you have software that doesn't support vfr in your toolchain.

[poisondeathray]

Thanks for the clarification roozhou

[hack41]

You don't need timecodes to "make" VFR video. Actually all the containers listed here natively supports VFR and stores PTS for each frame. Most capture software output only AVI, but even AVI should handle frame dropping easily by inserting null frames.

If you are going to encode captured video later, use x264(don't use avisynth!). It supports vfr natively as well.

Timecodes is only needed when you have software that doesn't support vfr in your toolchain.

Thanks, that's interesting. But I don't consider "frame dropping" as a real solution, and I would rather know of a more general solution that genuinely and truly accepts variable framerates, not just frame dropping.

Suppose then that I'm writing my own OpenGL screen grabber and run it on a 3D application. The scene complexity varies widely and the output framerate can be anything from 10fps up to maybe 60fps, with occasional spikes up to 100fps. (Just for example's sake.) My application records all the individual time codes for lack of my knowledge of how to do it better. But I do want to keep all the frames, I don't want to cut anything off arbitrarily.

Now what kind of tools can I use to encode such a video stream nicely? Does x264 have support for this (and could you give an example)? Does mencoder support such a project? I'm pretty sure that AVI does not support such vfr, all you can do is compute a highest common multiple and then "drop" frames from there, so we're back to frame dropping, which I don't consider truly vfr. Do you see what I mean?

[sneaker_ger]

I don't really get you. What's "a general solution" supposed to be? If you were to program such a capping application you'd have to program it yourself obviously. It would either have to cap to mkv, mp4 etc. or raw/avi with an additional timecode file.
And yes, x264 can support this. I can't give you an example for binding x264 into your own apps (just don't know how to do it...) but reencoding vfr material via the cli is as simple as "x264 -o output.mkv input".