Variable frame rate support

[neuron2]

[posted on behalf of Kurosu by mod]

Seeing what was reported by ErMac, a rather distant (as in: can't be used for now) point came to my mind. Variable framerate content. Today, the VfW interface limits us to constant frame rate (except) for dropped frame. Moreover I think those dropped frames are a decision of the encoder (codec), and I don't know if it can be forced from outside when the codec is able to introduce 'dropped' frames. But the upcoming of VirtualDub 2.0 (not only non-linear filtering, but I also hope access to directshow filters/codecs), the new containers such as Ogg (ogm), mp4, mcf or matroska will most probably support variable frame rate encoding. But it exists since some time, for instance in RealMedia format and what is embodied by ASF.

The use of all this:
- having 30fps and 24 fps footage (hybrid clip, no more jerky playback)
- effect of Donald's dup filter (I don't expect however to see for instance anime encoding foir which frame rate will vary from 4fps to 24)
- still sequences

I agree that this proposal may not even be in the scope of avisynth 3, is a nightmare programming-wise (at least ) and in the end unusable. But I'm curious as to how all the previous tools (Vdub, the containers and their directshow implementation) will tackle that. I fear that no solution can be found, although it would have brought only a small improvement.

I now even wonder if some ogm tool could join together a 24fps and a 30fps sequence easily...


__________________
Kurosu (Kurosu (at) inforezo dot org)
Avisynth related: http://kurosu.inforezo.org/avs
"Reality ? Is that where the pizza delivery guy comes from ?"

[Kurosu]

Thanks for moving it out of the other thread, it would otherwise have led to a difficult to follow discussion.

[neuron2]

An interesting somewhat related link:

http://broadcastengineering.com/ar/b...ate_camcorder/

My two cents is that the dropped frame solution is an ingenious but abominable hack that should be avoided like the plague. We need a new multimedia format that naturally supports variable frame rate. When such a format becomes accepted and ubiquitous, it would be the most natural thing in the world for Avisynth to move to it.

[Defiler]

The atomic unit of an MPEG-4 stream is, in theory, an arbitrarily-long group of frames, right? It seems like the MPEG-4 creators must have had something like this in mind, at least to some degree.
The first post in this thread put a big smile on my face. I'd love to see even half of those features implemented. Amazing.

[ErMaC]

The thing is we would need an MPEG4 encoder which could deal with this kind of thing properly, i.e. accept an AVISynth script with a variable framerate. Considering AVISynth currently communicates - for the most part - via the VFW interface, this is very problematic. Essentially the MPEG4 encoder would have to have it's own AVS interpreter.

A good place to start tackling this problem is - how could AVISynth communicate the idea of variable framerates to an encoder? One option I see is giving each frame a duration. At 29.97fps every frame has a duration of 33.3667ms, and then when you want to switch to a part which has been IVTC'd, those frames are given a duration of 41.7084ms (23.976fps).

Say we use a container like MCF, OGM, or even MP4. What needs to be done on AVISynth's end to make this possible?
This ties back in to the original discussion of giving frames and clips properties. A clip could have a framerate property, but that makes it impossible to cleanly do VFR. What about making duration a frame-based property?

[trbarry]

When it comes to variable frame rate I vote that Avisynth be absolutely the last utility in the world to handle it. It is a can of worms.

- Tom

[Kurosu]

VfW doesn't support VFR, so considering the interfaces as they are now designed can't help. This idea will keep being a dream if no software use DirectShow calls to read the source, and can't manage the framerate information that would be provided by avisynth. But now that I think of it, it's not really a matter of framerate (unless the codecs and different formats work that way) but rather a time at which the frame is displayed. And this property is really a frame-level property. Yet the problem remains: how today interfaces (ie mainly directshow) manage that?

But then I have mixed feelings about it: is it the job of the codec to handle the VFR tags? I believe it needs to be stored to the file, in the frame headers. I will check on the differents formats I was talking of. Maybe a VfW codec could be used: data is given to the codec, it outputs an encoded buffer, the application writes this buffer to file with the appropriate headers.

But this requires a revolution in the avisynth core, as I guess it was only designed for VfW interface. But then it wouldn't be 'avi'synth anymore.

This leads me to the point: there are future additions that should be considered in regard to today's file formats and codec interfaces. Someone posted once a link about some File Information tags reporting if the video had chroma interlacing for instance. If avisynth never evolves from VfW (which is already hard enough to deal with), then it seems obvious that, if some video stream needs avisynth-only (not usable by VfW) information to be displayed and encoded correctly (such as the chroma interlacing), many limits would be hit.

Now the pandora box is opened

[sh0dan]

"I can't bear to be Pandora. And I'm not brave enough to wait around and see the death and misery I have caused... This is my last transmission, my friend. Be careful... I think SHODAN has plans for you."

Just to quote one of the characters in System Shock 2

Anyway - there is no way for AviSynth to deliver variable FPS data, and I doubt there will ever be (as long as it's called AviSynth). The only current way is to give a framerate of the lowest common denominator, and deliver extremely many duplicate frames, which is... just stupid.

[Kurosu]

Originally posted by sh0dan
Just to quote one of the characters in System Shock 2

Ah! The mystery is no more then!

Originally posted by sh0dan
Anyway - there is no way for AviSynth to deliver variable FPS data, and I doubt there will ever be (as long as it's called AviSynth).

Indeed, that's why it was only a research topic. Anyway, here is a quick search I did about the feasibility:

MCF:
Block info
- block header: uint32 for timecode in ms from start of file (mandatory)
Seek header (mandatory)
- Timecode: ms from start of file, uint32

Timecodes are only known for:
- The first frame in a Block, and
- All Frames in Tracks which have constant Frame duration

DirectShow/DMO:
TIMECODE_SAMPLE structure:
- LONGLONG qwTick is the reference time, per 100 nanosecond units (quadword)
- struct TIMECODE: more restrictive, looks a lot like the structures in avisynth
VIDEOINFOHEADER/VIDEOINFOHEADER2 Structure:
- REFERENCE_TIME AvgTimePerFrame (LONGLONG), value that specifies the video frame's average display time, in milliseconds.

Matroska:
Information block
- TimecodeScale: Timecode scale in nanoseconds (1.000.000 means all timecodes in the segment are expressed in milliseconds).
Cluster block (uint) (always mandatory)
- Timecode: Absolute timecode of the cluster (based on TimecodeScale) (uint) (always mandatory)
- ReferenceBlock: Timecode of a frame used as a reference (B or P frame). The timecode is relative to the block it's attached to.
Track block, video
- FrameRate: Number of frames per second. Informational only. (at best optional, also not used)

I have still no clue about a possible DirectShow compliant video editor (VirtualDub 2?). I think OGM can manage variable framerate source (because as all the previous containers, it relies on timestamps, see subtitle streams for instance), but I haven't found any evidence in the oggfile specs.

Originally posted by sh0dan
The only current way is to give a framerate of the lowest common denominator, and deliver extremely many duplicate frames, which is... just stupid. [/B]

That is, 119.88fps for typical hybrid NTSC material... I would however ponder the 'stupid', alas only on often useless basis:
- it's stupid in the first place because the info 'dropped' can't be passed to the avi application, and because this application can't always force a codec to encode a frame as 'dropped'
- when dealing with delta-frame encoders (mpeg or such), a simple copied frame takes almost no place. Though, considering the huge amount of duplicates, it still amounts to a huge value. And we don't consider the CPU usage.

Anyway, this thread can't be of any interest untill a video editor actually manage variable framerate on input... And yes, the amount of rework on avisynth is maybe not worth it (together with some frame info as chroma interlacing).

[Stabmaster-Arson]

VFR is bad. I don't like it.

Make it go away, please.

[ErMaC]

OK this thread is for CONSTRUCTIVE discussion of this topic. If you're only post is going to be "I don't like VFR" then please keep it to yourself. Tom's comment actually had substance. Yours, stabmaster, does not.

Anyways, as Korosu pointed out - modern container formats DO have the capacity to deal with VFR. Ergo, eventually people will be asking AVISynth developers to include it. I know I will when I make the switch off of AVI.
While I think it is obviously premature to include VFR support even in v3.0, I do think it's a good time to start thinking ahead about what kinds of problems these new container formats will present for AVISynth's core.
Since my original post was concering additional properties for frames, I would like to reiterate - I think it's a good idea to include a Duration value for each frame. Even if in the current implementation of AVISynth the duration must be the same for each frame in a clip (i.e. constant frame rate), it will most likely save us a headache later when the times comes that we might want to add VFR support. Which is why I brought the VFR subject up in the first place. I think it'd be a good idea to plan for the future and include a duration value for frames. That's all I'm asking.

Now as for the rest of the discussion - here's a good idea. Why doesn't someone make an AVS2OGM or AVS2MCF program of some sort, or at least show a proof of concept, that would make VFR work? I'm not a programmer, I'm an encoder, so I have no idea how to do this and I'm not familiar with the other formats at all, really. This might be something good to take to the makers of the format and say, "Hey, you want people to take advantage of all your cool features? Come up with an API like VFW that AVISynth can support easily and we'll do lunch. I'll have my people call your people."

[Belgabor]

Originally posted by ErMaC
Now as for the rest of the discussion - here's a good idea. Why doesn't someone make an AVS2OGM or AVS2MCF program of some sort, or at least show a proof of concept, that would make VFR work? I'm not a programmer, I'm an encoder, so I have no idea how to do this and I'm not familiar with the other formats at all, really. This might be something good to take to the makers of the format and say, "Hey, you want people to take advantage of all your cool features? Come up with an API like VFW that AVISynth can support easily and we'll do lunch. I'll have my people call your people."

Shouldn't be too hard. Only add a function like GetFrameTime to IClip (or overload GetFrame with an additional out parameter).

Writig a new api is cumbersome and not necessary. The app only has to get a IScriptEnvironment directly from AviSynth and call Eval with the script as argument.

Cheers
Belgabor

[Kurosu]

@ErMaC

I'm currently looking at oggmux by Koepi, but I'm far for even having found the point where the timecode would be written. I think it's feasible if somewhere I can specify the timecode. My plan: take for instance the TNG sample provided by neuron2, encode the CG parts at 29.97fps into an avi, and the other at 23.976fps in another avi, and put these 2 parts together in an ogm.

About the new frame property (we're maybe talking in the wind but, still, it's an interesting stuff), I prefer to see it as a timecode (ie an absolute time since the start of the video stream) rather than a frame duration. Frame duration is more precise, but I think that a precision of under 1ms is hardly usefull (though all containers support lower resolution), and timecodes are the natural information handled by those containers.

Though, I don't know how to tackle skipping to a particular temporal point in the stream (as we don't know yet the equivalence time<->frame). I even wonder how ogm/mcf stream readers handle this (I've seen seek headers for mcf for instance). For that, I think that an additional structure (an index of frames) has to be created an one point for non-linear access to frames. And if 'avi'synth is to support VFR, it somehow has to build an index, that would be better passed to the editing tool, if DirectShow permits that. I'm confident timecodes can be passed from 'avi'synth to any video editing tool if they correctly handle DirectShow and communicate through it. But then again, it wouldn't be avisynth anymore.

@Stabmaster-Arson
Your fear of VFR either comes from the impossibility for most tools to handle it (say, avisynth or virtualdub), the fact you, as all of us, encode to a container limited to constant framerate, and the nightmare it can be to handle it (nonlinear seeking). But VFR is only a consequence of what's happening. If you consider only a succession of frames and the time at which they are displayed, it's the same think as constant FR sequences. You could use any VfW encoder to encode that flawlessly, only the container would have to tackle the VFR. And a good editing tool for that container should be able to at least retrieve the timecode, keep them and reuse them later. What I mean: don't deal with frame timecodes if you don't have to. Only the previewing is troublesome, the encoder will encode frame by frame.

[trbarry]

Tom's comment actually had substance...

Actually, even my comment was rather too flippant for a serious issue. I apologize.

I'm still against doing much about it now because it seems there is a basic design clash between a time stamp based model and the constant rate Avisynth/Vdub/VFW design model. And I was afraid that someone would take the idea and run with it making an Avisynth that was not easily compatible with existing stuff.

But I do believe that in the future most media will operate this way and that even displays will not have to have a fixed rate but instead will just display each frame at the specified time.

So even the idea of a display refresh rate (and associated judder) will maybe go away. Telecine with 3:2 pulldown is a silly idea that maybe we won't have to put up with forever.

So discussions of this are relevant. I just think other things should probably take priority for awhile.

- Tom

[ErMaC]

The problem with a timecode value is that it would have to be rewritten every time there is a shift in frames - i.e. if I insert a 3 frame span at the very start of the stream, then the timecode for every single subsequent frame must be recalculated.
Instead, if we use a duration variable, and then at the time the script is executed build a timecode index as part of the script environment for the encoder to look at, this would avoid multiple, redundant executions of the same timecode calculations.

BTW Belgabor - it's good to see you here, as having input from someone who actually does encoding program development would be very helpful. What do you think would be necessary to make VFR a reality between AVISynth, and encoder like VDubMod/a command line encoder, and a next-gen container like OGM, MCF, or Matroska?

[Kurosu]

Oggmux builds a filter graph (the way graphedit does). That put my idea of using it for these experiments far above my abilities, as it is based on DirectShow API. Anyway, I doubt there is a DirectShow filter to concatenate videostreams of different framerates to help for muxing.

I'm looking at VirtualDub_Mod, which seems to handle timecodes (timestamps), but then again...

[Suiryc]

Some information for you

OGM (based on Ogg) doesn't rely on a timestamp, but what is called a granulepos (i.e. a sample count).
As said in the specs :

absolute granule position

Note that the 'position' data specifies a 'sample' number (eg, in a CD quality sample is four octets, 16 bits for left and 16 bits for right; in video it would likely be the frame number. It is up to the specific codec in use to define the semantic meaning of the granule position value). The position specified is the total samples encoded after including all packets finished on this page (packets begun on this page but continuing on to the next page do not count). The rationale here is that the position specified in the frame header of the last page tells how long the data coded by the bitstream is. A truncated stream will still return the proper number of samples that can be decoded fully.

The current purpose of this field in OGM files is indeed to store the frame number. So the current OGM specs doens't really allow Variable FrameRate.

To be able to use Variable FrameRate this field could be used to store a timestamp (expressed in ms or any other unit; this is what is done currently for subtitles embedded in OGM).
However there is something that you won't be able to do regarding those specs and current (and future) XipH codecs. As you can read the meaning of this field is proper to the codec associated to the data stored in the track. What does that mean ? Well simply that a Vorbis track also use this granulepos field to store the number of samples that can be decoded so far. In other words your Vorbis stream shouldn't contain 'gaps' (period of times where no data are to be rendered).
Using a timestamp for data (as in MCF and Matroska specs, and maybe - in a way or another - in MP4 specs) theorically allow you to add a delay or insert gaps in your stream.


The current implementation of VirtualDub (and VirtualDubMod) doesn't allow Variable FrameRate because it is heavily based on VfW and the AVI container. For example when dealing with video (or audio) data are requested by using a sample number (index of the video frame / audio sample). Maybe next versions (VirtualDub 2 ?) will be able to handle this kind of feature, I don't know what Avery Lee plans (except to make the program better and better ).

[Kurosu]

It's up to the codec writer to decide what those 8 bytes means, but it seems that Ogg Vorbis file format won't accept discontinuous (rather, gaps or variable step in the index numbering) as a timestamp for a VFR stream would require...

I understand that the idea I'm about to present has many reasons to be rejected (first, incompatibility with older streams), but let's have a look. Once a packet is finished on the current page, stripping all the standard ogg vorbis header leaves data which is up to the reader to interpret. It could be raw data. But what about another 8bytes for timestamp (the absolute granular position is still the frame number)?

Now, why should it be definetely be dropped. There has to be some means to do that, or ogg vobis file format lacks a feature. For the time being, I'll have to see what is available for testing purpose with MCF/Matroska.

I guess I'll really have to install WinCVS...

PS: About the frame-based requesting in VDub, it's rather the mapping frame<->timestamp that is problematic, as putting the timestamp as part of the data 'solves' (rather circumvent) the problem.

[Atamido]

Originally posted by ErMaC
The problem with a timecode value is that it would have to be rewritten every time there is a shift in frames - i.e. if I insert a 3 frame span at the very start of the stream, then the timecode for every single subsequent frame must be recalculated.
Instead, if we use a duration variable, and then at the time the script is executed build a timecode index as part of the script environment for the encoder to look at, this would avoid multiple, redundant executions of the same timecode calculations.

This exact same issue came up when we were discussing how a vfw replacement should work.

Using a frame duration setting sounds good, and was even mentioned for adding to the Matroska specs, so that a frame (subtitle frame) could be displayed for only X amount of time. The problem with using duration for video frames is that you can get rounding errors when converting from a constant frame rate source. You would also get the same rounding errors when using timestamps, but they wouldn't have the cumulative effect that a duration time. When you add up several duration times that are each off by a small amount, you end up with a potential error margin of all of the errors added together. When using timestamps, you can have a margin of error on each frame, but they will never be cumulative.

As far as a vfw replacement, it was considered that you could ship the timestamp along with the frame (audio, video, whatever) as a type of unique frame-ID for that stream. As only one frame can occur at a given time in a single stream, this would work out pretty well. Now the codec wouldn't have to understand the timecode at all, as it is simply the ID that can be used to identify the frame. However, if the codec is smart enough, it could drop out frames, and change the timecodes around a bit to compensate.

How this relates to something along the lines of AVISynth, is that it would simply need to pass along the timecode with the frame, without anything actually needing to understand it. If something did understand it, it could possibly make use of it, but would otherwise just keep passing the timecode along.


Note: When refering to a vfw replacement, I was not referring to DirectShow as it apparently doesn't give the application enough ?control?, and it is not platform independant. There were some interesting discussions on this in the UCI mailing list that can be accessed here.

[trbarry]

I'm not at all an expert in this but some things like digital TV probably work better with time stamps instead of durations. This is because you can pick up a stream in the middle and still be able to sync separate parts.

It may also help in the case of dropouts from reception problems, etc.

But captured digital tv sort of starts at the point where you start capturing. There isn't really a beginning or end of a file so any control info has to be repeatedly imbedded in the stream, which it is. I'm not sure how durations would work with all this.

- Tom