Variable frame rate support

[neuron2]

[posted on behalf of Kurosu by mod]

Seeing what was reported by ErMac, a rather distant (as in: can't be used for now) point came to my mind. Variable framerate content. Today, the VfW interface limits us to constant frame rate (except) for dropped frame. Moreover I think those dropped frames are a decision of the encoder (codec), and I don't know if it can be forced from outside when the codec is able to introduce 'dropped' frames. But the upcoming of VirtualDub 2.0 (not only non-linear filtering, but I also hope access to directshow filters/codecs), the new containers such as Ogg (ogm), mp4, mcf or matroska will most probably support variable frame rate encoding. But it exists since some time, for instance in RealMedia format and what is embodied by ASF.

The use of all this:
- having 30fps and 24 fps footage (hybrid clip, no more jerky playback)
- effect of Donald's dup filter (I don't expect however to see for instance anime encoding foir which frame rate will vary from 4fps to 24)
- still sequences

I agree that this proposal may not even be in the scope of avisynth 3, is a nightmare programming-wise (at least ) and in the end unusable. But I'm curious as to how all the previous tools (Vdub, the containers and their directshow implementation) will tackle that. I fear that no solution can be found, although it would have brought only a small improvement.

I now even wonder if some ogm tool could join together a 24fps and a 30fps sequence easily...


__________________
Kurosu (Kurosu (at) inforezo dot org)
Avisynth related: http://kurosu.inforezo.org/avs
"Reality ? Is that where the pizza delivery guy comes from ?"

[Kurosu]

Thanks for moving it out of the other thread, it would otherwise have led to a difficult to follow discussion.

[neuron2]

An interesting somewhat related link:

http://broadcastengineering.com/ar/broadcasting_panasonics_variableframerate_camcorder/

My two cents is that the dropped frame solution is an ingenious but abominable hack that should be avoided like the plague. We need a new multimedia format that naturally supports variable frame rate. When such a format becomes accepted and ubiquitous, it would be the most natural thing in the world for Avisynth to move to it.

[Defiler]

The atomic unit of an MPEG-4 stream is, in theory, an arbitrarily-long group of frames, right? It seems like the MPEG-4 creators must have had something like this in mind, at least to some degree.
The first post in this thread put a big smile on my face. I'd love to see even half of those features implemented. Amazing.

[ErMaC]

The thing is we would need an MPEG4 encoder which could deal with this kind of thing properly, i.e. accept an AVISynth script with a variable framerate. Considering AVISynth currently communicates - for the most part - via the VFW interface, this is very problematic. Essentially the MPEG4 encoder would have to have it's own AVS interpreter.

A good place to start tackling this problem is - how could AVISynth communicate the idea of variable framerates to an encoder? One option I see is giving each frame a duration. At 29.97fps every frame has a duration of 33.3667ms, and then when you want to switch to a part which has been IVTC'd, those frames are given a duration of 41.7084ms (23.976fps).

Say we use a container like MCF, OGM, or even MP4. What needs to be done on AVISynth's end to make this possible?
This ties back in to the original discussion of giving frames and clips properties. A clip could have a framerate property, but that makes it impossible to cleanly do VFR. What about making duration a frame-based property?

[trbarry]

When it comes to variable frame rate I vote that Avisynth be absolutely the last utility in the world to handle it. It is a can of worms.

- Tom

[Kurosu]

VfW doesn't support VFR, so considering the interfaces as they are now designed can't help. This idea will keep being a dream if no software use DirectShow calls to read the source, and can't manage the framerate information that would be provided by avisynth. But now that I think of it, it's not really a matter of framerate (unless the codecs and different formats work that way) but rather a time at which the frame is displayed. And this property is really a frame-level property. Yet the problem remains: how today interfaces (ie mainly directshow) manage that?

But then I have mixed feelings about it: is it the job of the codec to handle the VFR tags? I believe it needs to be stored to the file, in the frame headers. I will check on the differents formats I was talking of. Maybe a VfW codec could be used: data is given to the codec, it outputs an encoded buffer, the application writes this buffer to file with the appropriate headers.

But this requires a revolution in the avisynth core, as I guess it was only designed for VfW interface. But then it wouldn't be 'avi'synth anymore.

This leads me to the point: there are future additions that should be considered in regard to today's file formats and codec interfaces. Someone posted once a link about some File Information tags reporting if the video had chroma interlacing for instance. If avisynth never evolves from VfW (which is already hard enough to deal with), then it seems obvious that, if some video stream needs avisynth-only (not usable by VfW) information to be displayed and encoded correctly (such as the chroma interlacing), many limits would be hit.

Now the pandora box is opened :D

[sh0dan]

"I can't bear to be Pandora. And I'm not brave enough to wait around and see the death and misery I have caused... This is my last transmission, my friend. Be careful... I think SHODAN has plans for you."

Just to quote (http://www.radisol.com/shock2/characters/polito/pandora.htm) one of the characters in System Shock 2 ;)

Anyway - there is no way for AviSynth to deliver variable FPS data, and I doubt there will ever be (as long as it's called AviSynth). The only current way is to give a framerate of the lowest common denominator, and deliver extremely many duplicate frames, which is... just stupid.

[Kurosu]

Originally posted by sh0dan
Just to quote (http://www.radisol.com/shock2/characters/polito/pandora.htm) one of the characters in System Shock 2 ;)


Ah! The mystery is no more then!

Originally posted by sh0dan
Anyway - there is no way for AviSynth to deliver variable FPS data, and I doubt there will ever be (as long as it's called AviSynth).

Indeed, that's why it was only a research topic. Anyway, here is a quick search I did about the feasibility:

MCF (http://mukoli.free.fr/mcf/mcf.html):
Block info
- block header: uint32 for timecode in ms from start of file (mandatory)
Seek header (mandatory)
- Timecode: ms from start of file, uint32

Timecodes are only known for:
- The first frame in a Block, and
- All Frames in Tracks which have constant Frame duration

DirectShow/DMO (http://www.microsoft.com/Developer/PRODINFO/directx/dxm/help/ds/ref/structs/struct_TIMECODE_SAMPLE.htm):
TIMECODE_SAMPLE structure:
- LONGLONG qwTick is the reference time, per 100 nanosecond units (quadword)
- struct TIMECODE: more restrictive, looks a lot like the structures in avisynth
VIDEOINFOHEADER/VIDEOINFOHEADER2 Structure:
- REFERENCE_TIME AvgTimePerFrame (LONGLONG), value that specifies the video frame's average display time, in milliseconds.

Matroska (http://matroska.sourceforge.net/specs/):
Information block
- TimecodeScale: Timecode scale in nanoseconds (1.000.000 means all timecodes in the segment are expressed in milliseconds).
Cluster block (uint) (always mandatory)
- Timecode: Absolute timecode of the cluster (based on TimecodeScale) (uint) (always mandatory)
- ReferenceBlock: Timecode of a frame used as a reference (B or P frame). The timecode is relative to the block it's attached to.
Track block, video
- FrameRate: Number of frames per second. Informational only. (at best optional, also not used)

I have still no clue about a possible DirectShow compliant video editor (VirtualDub 2?). I think OGM can manage variable framerate source (because as all the previous containers, it relies on timestamps, see subtitle streams for instance), but I haven't found any evidence in the oggfile specs.

Originally posted by sh0dan
The only current way is to give a framerate of the lowest common denominator, and deliver extremely many duplicate frames, which is... just stupid. [/B]
That is, 119.88fps for typical hybrid NTSC material... I would however ponder the 'stupid', alas only on often useless basis:
- it's stupid in the first place because the info 'dropped' can't be passed to the avi application, and because this application can't always force a codec to encode a frame as 'dropped'
- when dealing with delta-frame encoders (mpeg or such), a simple copied frame takes almost no place. Though, considering the huge amount of duplicates, it still amounts to a huge value. And we don't consider the CPU usage.

Anyway, this thread can't be of any interest untill a video editor actually manage variable framerate on input... And yes, the amount of rework on avisynth is maybe not worth it (together with some frame info as chroma interlacing).

[Stabmaster-Arson]

VFR is bad. I don't like it.

Make it go away, please.

[ErMaC]

OK this thread is for CONSTRUCTIVE discussion of this topic. If you're only post is going to be "I don't like VFR" then please keep it to yourself. Tom's comment actually had substance. Yours, stabmaster, does not.

Anyways, as Korosu pointed out - modern container formats DO have the capacity to deal with VFR. Ergo, eventually people will be asking AVISynth developers to include it. I know I will when I make the switch off of AVI. :)
While I think it is obviously premature to include VFR support even in v3.0, I do think it's a good time to start thinking ahead about what kinds of problems these new container formats will present for AVISynth's core.
Since my original post was concering additional properties for frames, I would like to reiterate - I think it's a good idea to include a Duration value for each frame. Even if in the current implementation of AVISynth the duration must be the same for each frame in a clip (i.e. constant frame rate), it will most likely save us a headache later when the times comes that we might want to add VFR support. Which is why I brought the VFR subject up in the first place. I think it'd be a good idea to plan for the future and include a duration value for frames. That's all I'm asking.

Now as for the rest of the discussion - here's a good idea. Why doesn't someone make an AVS2OGM or AVS2MCF program of some sort, or at least show a proof of concept, that would make VFR work? I'm not a programmer, I'm an encoder, so I have no idea how to do this and I'm not familiar with the other formats at all, really. This might be something good to take to the makers of the format and say, "Hey, you want people to take advantage of all your cool features? Come up with an API like VFW that AVISynth can support easily and we'll do lunch. I'll have my people call your people."

[Belgabor]

Originally posted by ErMaC
Now as for the rest of the discussion - here's a good idea. Why doesn't someone make an AVS2OGM or AVS2MCF program of some sort, or at least show a proof of concept, that would make VFR work? I'm not a programmer, I'm an encoder, so I have no idea how to do this and I'm not familiar with the other formats at all, really. This might be something good to take to the makers of the format and say, "Hey, you want people to take advantage of all your cool features? Come up with an API like VFW that AVISynth can support easily and we'll do lunch. I'll have my people call your people."

Shouldn't be too hard. Only add a function like GetFrameTime to IClip (or overload GetFrame with an additional out parameter).

Writig a new api is cumbersome and not necessary. The app only has to get a IScriptEnvironment directly from AviSynth and call Eval with the script as argument.

Cheers
Belgabor

[Kurosu]

@ErMaC

I'm currently looking at oggmux by Koepi, but I'm far for even having found the point where the timecode would be written. I think it's feasible if somewhere I can specify the timecode. My plan: take for instance the TNG sample provided by neuron2, encode the CG parts at 29.97fps into an avi, and the other at 23.976fps in another avi, and put these 2 parts together in an ogm.

About the new frame property (we're maybe talking in the wind but, still, it's an interesting stuff), I prefer to see it as a timecode (ie an absolute time since the start of the video stream) rather than a frame duration. Frame duration is more precise, but I think that a precision of under 1ms is hardly usefull (though all containers support lower resolution), and timecodes are the natural information handled by those containers.

Though, I don't know how to tackle skipping to a particular temporal point in the stream (as we don't know yet the equivalence time<->frame). I even wonder how ogm/mcf stream readers handle this (I've seen seek headers for mcf for instance). For that, I think that an additional structure (an index of frames) has to be created an one point for non-linear access to frames. And if 'avi'synth is to support VFR, it somehow has to build an index, that would be better passed to the editing tool, if DirectShow permits that. I'm confident timecodes can be passed from 'avi'synth to any video editing tool if they correctly handle DirectShow and communicate through it. But then again, it wouldn't be avisynth anymore.

@Stabmaster-Arson
Your fear of VFR either comes from the impossibility for most tools to handle it (say, avisynth or virtualdub), the fact you, as all of us, encode to a container limited to constant framerate, and the nightmare it can be to handle it (nonlinear seeking). But VFR is only a consequence of what's happening. If you consider only a succession of frames and the time at which they are displayed, it's the same think as constant FR sequences. You could use any VfW encoder to encode that flawlessly, only the container would have to tackle the VFR. And a good editing tool for that container should be able to at least retrieve the timecode, keep them and reuse them later. What I mean: don't deal with frame timecodes if you don't have to. Only the previewing is troublesome, the encoder will encode frame by frame.

[trbarry]

Tom's comment actually had substance...

Actually, even my comment was rather too flippant for a serious issue. I apologize.

I'm still against doing much about it now because it seems there is a basic design clash between a time stamp based model and the constant rate Avisynth/Vdub/VFW design model. And I was afraid that someone would take the idea and run with it making an Avisynth that was not easily compatible with existing stuff. ;)

But I do believe that in the future most media will operate this way and that even displays will not have to have a fixed rate but instead will just display each frame at the specified time.

So even the idea of a display refresh rate (and associated judder) will maybe go away. Telecine with 3:2 pulldown is a silly idea that maybe we won't have to put up with forever.

So discussions of this are relevant. I just think other things should probably take priority for awhile.

- Tom

[ErMaC]

The problem with a timecode value is that it would have to be rewritten every time there is a shift in frames - i.e. if I insert a 3 frame span at the very start of the stream, then the timecode for every single subsequent frame must be recalculated.
Instead, if we use a duration variable, and then at the time the script is executed build a timecode index as part of the script environment for the encoder to look at, this would avoid multiple, redundant executions of the same timecode calculations.

BTW Belgabor - it's good to see you here, as having input from someone who actually does encoding program development would be very helpful. What do you think would be necessary to make VFR a reality between AVISynth, and encoder like VDubMod/a command line encoder, and a next-gen container like OGM, MCF, or Matroska?

[Kurosu]

Oggmux builds a filter graph (the way graphedit does). That put my idea of using it for these experiments far above my abilities, as it is based on DirectShow API. Anyway, I doubt there is a DirectShow filter to concatenate videostreams of different framerates to help for muxing.

I'm looking at VirtualDub_Mod, which seems to handle timecodes (timestamps), but then again...

[Suiryc]

Some information for you :)

OGM (based on Ogg) doesn't rely on a timestamp, but what is called a granulepos (i.e. a sample count).
As said in the specs (http://www.xiph.org/ogg/vorbis/doc/framing.html) :
absolute granule position

Note that the 'position' data specifies a 'sample' number (eg, in a CD quality sample is four octets, 16 bits for left and 16 bits for right; in video it would likely be the frame number. It is up to the specific codec in use to define the semantic meaning of the granule position value). The position specified is the total samples encoded after including all packets finished on this page (packets begun on this page but continuing on to the next page do not count). The rationale here is that the position specified in the frame header of the last page tells how long the data coded by the bitstream is. A truncated stream will still return the proper number of samples that can be decoded fully.
The current purpose of this field in OGM files is indeed to store the frame number. So the current OGM specs doens't really allow Variable FrameRate.

To be able to use Variable FrameRate this field could be used to store a timestamp (expressed in ms or any other unit; this is what is done currently for subtitles embedded in OGM).
However there is something that you won't be able to do regarding those specs and current (and future) XipH codecs. As you can read the meaning of this field is proper to the codec associated to the data stored in the track. What does that mean ? Well simply that a Vorbis track also use this granulepos field to store the number of samples that can be decoded so far. In other words your Vorbis stream shouldn't contain 'gaps' (period of times where no data are to be rendered).
Using a timestamp for data (as in MCF and Matroska specs, and maybe - in a way or another - in MP4 specs) theorically allow you to add a delay or insert gaps in your stream.


The current implementation of VirtualDub (and VirtualDubMod) doesn't allow Variable FrameRate because it is heavily based on VfW and the AVI container. For example when dealing with video (or audio) data are requested by using a sample number (index of the video frame / audio sample). Maybe next versions (VirtualDub 2 ?) will be able to handle this kind of feature, I don't know what Avery Lee plans (except to make the program better and better ;)).

[Kurosu]

It's up to the codec writer to decide what those 8 bytes means, but it seems that Ogg Vorbis file format won't accept discontinuous (rather, gaps or variable step in the index numbering) as a timestamp for a VFR stream would require...

I understand that the idea I'm about to present has many reasons to be rejected (first, incompatibility with older streams), but let's have a look. Once a packet is finished on the current page, stripping all the standard ogg vorbis header leaves data which is up to the reader to interpret. It could be raw data. But what about another 8bytes for timestamp (the absolute granular position is still the frame number)?

Now, why should it be definetely be dropped. There has to be some means to do that, or ogg vobis file format lacks a feature. For the time being, I'll have to see what is available for testing purpose with MCF/Matroska.

I guess I'll really have to install WinCVS... :(

PS: About the frame-based requesting in VDub, it's rather the mapping frame<->timestamp that is problematic, as putting the timestamp as part of the data 'solves' (rather circumvent) the problem.

[Atamido]

Originally posted by ErMaC
The problem with a timecode value is that it would have to be rewritten every time there is a shift in frames - i.e. if I insert a 3 frame span at the very start of the stream, then the timecode for every single subsequent frame must be recalculated.
Instead, if we use a duration variable, and then at the time the script is executed build a timecode index as part of the script environment for the encoder to look at, this would avoid multiple, redundant executions of the same timecode calculations.
This exact same issue came up when we were discussing how a vfw replacement should work.

Using a frame duration setting sounds good, and was even mentioned for adding to the Matroska specs, so that a frame (subtitle frame) could be displayed for only X amount of time. The problem with using duration for video frames is that you can get rounding errors when converting from a constant frame rate source. You would also get the same rounding errors when using timestamps, but they wouldn't have the cumulative effect that a duration time. When you add up several duration times that are each off by a small amount, you end up with a potential error margin of all of the errors added together. When using timestamps, you can have a margin of error on each frame, but they will never be cumulative.

As far as a vfw replacement, it was considered that you could ship the timestamp along with the frame (audio, video, whatever) as a type of unique frame-ID for that stream. As only one frame can occur at a given time in a single stream, this would work out pretty well. Now the codec wouldn't have to understand the timecode at all, as it is simply the ID that can be used to identify the frame. However, if the codec is smart enough, it could drop out frames, and change the timecodes around a bit to compensate.

How this relates to something along the lines of AVISynth, is that it would simply need to pass along the timecode with the frame, without anything actually needing to understand it. If something did understand it, it could possibly make use of it, but would otherwise just keep passing the timecode along.


Note: When refering to a vfw replacement, I was not referring to DirectShow as it apparently doesn't give the application enough ?control?, and it is not platform independant. There were some interesting discussions on this in the UCI mailing list that can be accessed here. (http://article.gmane.org/gmane.comp.video.uci.devel/320)

[trbarry]

I'm not at all an expert in this but some things like digital TV probably work better with time stamps instead of durations. This is because you can pick up a stream in the middle and still be able to sync separate parts.

It may also help in the case of dropouts from reception problems, etc.

But captured digital tv sort of starts at the point where you start capturing. There isn't really a beginning or end of a file so any control info has to be repeatedly imbedded in the stream, which it is. I'm not sure how durations would work with all this.

- Tom

[Bidoche]

Internally in avisynth, we don't care if it's smarter for codecs to use timestamps, durations are definitely smarter, and we could always regenerate timestamps on output...

And for precision loss, we can choose the precision we want, even push to quantic levels if we want. I hope it would be enough. :)

[Belgabor]

We probably dont need to go too far, just convert fps to timestamps and derive durations from that.

[Atamido]

Doing

fps -> timestamp -> duration

would probably be fine, just a little redundant. Just don't do

fps -> duration -> timestamp

Personally, I don't see a good reason to use durations when timecodes are better for so many of the cases. On the other hand, I'm not AVISynth coder, and if it outputs timestamps fine, then I guess I shouldn't care what it uses internally.

[Belgabor]

Durations make juggeling around parts of the video much easier, you don't need to keep track of timestamps through the editting process.

[ErMaC]

Exactly - when using timestamps, everytime you did an (un)aligned splice or trim, you'd have to recalculate timestamps.

[Kurosu]

The splices don't have such a nasty effect. Sure it would not be immediate to recalculate timestamps, but what time is needed to even update an index of 5,000,000 uint32 entries? All in all, none should be able to notice the delay due to that calculus. And it would be settled once and for all. In the end, it's timestamps that are written (in mcf/matroska).

But again, what is the drawback of a frame duration? Not enough precision. Storing duration in nanoseconds in an uint32 (allowing for frames duration of up to 4s...) is sufficient: the maximum error per frame is a ns. You can hardly reach the ms, in a program made for editing files, and an environment also processing files. For instance, a 8H long videao at 120fps almost contains 3,5 milions frames, with a [U]maximum[U] error of 3,5ms. And it needs to always be the same error...

Though... How the editing program and 'avi'synth should communicate? Either the program natively support avisynth, or it supports DirectShow. Unless a new API appears, and if 'avi'synth wants to be usable by a maximum of programs, let's keep DirectShow:
- average frame duration is stored in ms: the maximum error per frame is 1ms, quite more sensitive
- timestamp is stored in 100ns unit...
Maybe this thinking is wrong, but I don't have the OggDS source to see what structures are filled.

Another problem: how do we skip to a particular point of time? Using timestamps, you have to scan through an index to see what frame is the closest to the timerequested. Using frame duration, you have to sum up all durations untill the requested time is reached.

[ErMaC]

To keep the thread alive :)

Well AVISynth does not have any sort of internal timestamp I don't think - it does everything by frame #, hence the addition of frame durations or frame timestamps would not affect the internals of an AVIsynth script at all. Trim() will work on a VFR sequence just fine if it has durations, since you'll just cut out a portion of frames but the sections with a specific FPS will remain the same. Using timestamps would make this more difficult.

As for seeking, with timestamps you can just use a search algorithm (either sequential from the start or binary from the middle) to find the right timestamp, or with durations simply start from the beginning and add the duration of each frame until the next frame would take you over the requested time, but since AVISynth would probably want to generate a timestamp index at script execution anyways for the reasons listed earlier, you could use the same method as outlined above for seeking.

But remember, seeking probably wouldn't be the job of AVIsynth - it will be the job of whatever program is accessing AVISynth's internal structure, so for the scope of AVISynth's implementation this is a moot point as long as the information required for seeking is provided to the application reading it.

[Atamido]

Would it create a problem to replace the framenumber with a timecode? I think that this would resolve the issue of variable framerate support, and reduce the amount of change needed in the code.

It should be pretty simple as you would be passing floating units instead of intergers. There are two potential problems I can see with this. The code would require a major rewrite to handle floats instead of intergers in the framenumber (ala frame-ID). And the code may not handle getting the numbers in a non-interger sequence (1, 2, 3, 4,... 367, 368, 369...)

Can anyone that is familiar with the AVISynth code comment on why this might be bad or good?

[ErMaC]

I think this would be a very bad thing because it would break one of the cardinal rules that the AVISynth developers have always followed - maintaining backwards compatability. Making the change to dealing with timecodes instead of frame numbers would suddently render all previously written AVISynth scripts, filters, and applications invalid.

Not to mention the fact that writing scripts in timecode would SUCK. I would not want to write scripts with full timecodes as parameters. It would be much more difficult to read, and far less easy to write.

Besides I think you will always want to have a frame # associated with each frame because that's the way that various container formats store frames. Each frame may have a timecode associated with it, but I don't think it's ever the other way around.

[Bidoche]

There is no issue with variable framerate support, at least no issue for how to do it, it's easy (mark videoinfo fps as variable and tag frames with durations)
The issue is if it should be done.

And I agree with ErMac, working with timecodes is a pain in the ...
We may want to expose a clip with timecodes for an external API, but not working with them internally.

If you need them in your own filter, you just have to convert timestamps to frames..
I guess I could provide both get frames throught frame # and timestamps, but if we go vfr, all clip will have to maintain their own seeking code and members (for efficiency it will have to cache many results) which I don't want.

[Kurosu]

OK, avisynth seems to work more efficiently with frame duration. Is it possible to convert this data into timestamps when it's an application (software) that request the frame, and not a filter?

I'm rather waiting for beta tools and libs (they already exist, but I think everyone is already doing his best to finish those tools) for the upcoming containers, like matroska. Ogg (Vorbis) file format could probably be adapted, but it's still not time for the developpers (I only know of Tobias) to spend time on it for now.

Besides, regarding 'avi'synth, it will need someone used to DirectShow (or someone from the UCI codec interface API development team) so that such info is passed to the application.

[ErMaC]

Instead of interfacing via DirectShow or UCI or what not.. why not do what was mentioned earlier and just have the application interface directly with AVISynth and fetch the frame structures directly. Isn't this what AVS2AVI does?
There's no need to write support for a whole new API unless there's already applications that support it, and I don't know of any encoding programs other than Graphedit which supports DirectShow encoding :)
It's just as simple to have an application fetch the IScriptEnvironment from AVISynth and interface directly with that, as Belgabor suggested.

[Bidoche]

In a different register, it came to me that I can probably make width and height variable too (internally).
What do you think of that ?

[ErMaC]

An interesting idea, but how would it be implemented in a player application? I can think of instances where it would come in handy (say in things which change aspect ratio or where you only need a certain section of the screen for a short part of the movie like the subtitles in Hunt for Red October).

[Kurosu]

Instead of interfacing via DirectShow or UCI or what not.. why not do what was mentioned earlier and just have the application interface directly with AVISynth and fetch the frame structures directly.

Why not? If someone is willing to add an other output than avi.
I was referring to DShow/UCI, because many application have the ability to read DShow source (like Premiere, RealMedia Producer). This interface is commonly used, and implementing it in avs would make available in a natural way the needed information for variable framerate.

@Bidoche
Why not? Though I wonder how it will be used. For instance, no codec is able to change the framesize on the fly. Moreover if you encode separately the different portions that needs different crop/resize, then you can also directly make an avisynth script that puts everything back to the wanted frame dimension.

[Bidoche]

I didn't not mean to export with variable width/height but to have it possible internally.
It may make some things easier/possible like Animate effects on the size.

[sh0dan]

I hope you take into consideration that all temporal filters in principle would have to be rewritten, to compensate for the varying time delays.

[Kurosu]

To me, the frame droping should occur at the very last moment (or before a resize), to make the better decision as what should be dropped, therefore after any cleaning. After all, that's the same problem (now on the temporal side) as having dimension restrictions in filters (MOD8,...): do it at a point where it's allowed/usefull, or don't do it.

Maybe a flag in the environment could be usefull, so that the filter knows if it may be fed with vfr content (and maybe minimum and maximum frame duration, though I don't see the use for it).

[Bidoche]

@sh0dan

I am not saying I am doing it, just thinking about it, so it may be implemented in the future at minimum cost.

[bilu]

I've been making waves instead of reviving this ***OLD THREAD*** and by doing so waking up everybody in it! ;)

How about an Avisynth fork or extender for Variable Framerate?
http://forum.doom9.org/showthread.php?s=&threadid=63290

@VFW/DShow experts: VFR in Avisynth could be good news for Matroska
http://forum.doom9.org/showthread.php?s=&threadid=63541

Practical cases on the encoding scene nowadays

RV9 Animation DropDupe Pre-filter
http://forum.doom9.org/showthread.php?s=&threadid=56564

Speed anime encoding a lot plus some bitrate gain.

Proved benefits:
1) Anime. You drop a duplicate but double the play time of the original frame, keeping the framerate.

2) Hybrid movies. You don't drop frames in the video parts (pure 30 fps) but drop duplicates in the film part (telecined 24fps -> 30 fps) to achieve pure 24 fps, and on this segment you DON'T want to keep the original framerate.

My idea, would it be doable?
Scheme
======
VFR: AVS -> VFW + frame duration stream -> VFR DShow splitter -> VFR rendering

CFR: AVS -> VFW + frame duration stream(a) -> CFR DShow splitter(b) -> CFR rendering

(a) frame duration stream would be optional and activated by a flag in Avisynth. If not available,
even the VFR DShow splitter would do CFR rendering;

(b) if the CFR DShow splitter received such a stream it would ignore it;

My idea is to transmit frame duration through another VFW stream (if possible, something like an audio stream) containing the frame duration info. A DShow splitter developed for this "VFR-Avisynth" would grab that info and apply it to frames by changing its rendering time, frame by frame. (doable?)

This would mean both VFR previews and encodings.

For VFR codecs: transcoding back to CFR capacity (optional)
Frames wouldn't need to be dropped, just set to 0 ms and referenced to the former frame. Then if you want to come back to CFR again (AVI,MPEG-2?) it should require another DShow splitter (or the same?) to recreate the original structure: a VFR IVTCed movie would become a Telecined CFR movie after Telecide(), waiting for Decimate()... ;)

Such references to the former frame (as many as needed to recreate CFR ) with duration set to 0ms would still be smaller and faster to encode than any CFR dup frame is now.


I still have hope in seeing a VFR Star Trek hybrid clip :)

Best regards,
Bilu

[Blight]

What needs to be done for this to work properly is:

1. Replace AviSynth with "SynthCore", a core library based on timestamps that can handle the various AVISynth functions and plugins.
2. Recreate AviSynth ontop of the "SynthCore" so you maintain support for all current application.
3. Create a DirectShowSynth filter to support any VFR codecs, the filter would be a source filter accepting AVS files and churning out time-stamped RAW images/audio.

The biggest problem I see are the filter that require a frame rate to work (Decimate and the likes). Most plugins would probably need a minor rewrite and some would need a major rewrite.

[bilu]

@all

Please take a look at my post here:

http://forum.doom9.org/showthread.php?s=&postid=389270#post389270

I think it's a great idea for VFR, much more compatible, easier to develop.

Please give your feedback. Thanks.


Bilu

[Atamido]

Originally posted by Blight
1. Replace AviSynth with "SynthCore", a core library based on timestamps that can handle the various AVISynth functions and plugins. Couldn't you just add a property for each frame called TimeCode?3. Create a DirectShowSynth filter to support any VFR codecs, the filter would be a source filter accepting AVS files and churning out time-stamped RAW images/audio.I couldn't agree more. It is the only interface that could take full advantage of the benefits this would offer.

[bilu]

Please continue discussion here:

@VFW/DShow experts: VFR in Avisynth could be good news for Matroska
http://forum.doom9.org/showthread.php?s=&threadid=63541

Bidoche has been participating there too, and it has been a very productive thread until now.I couldn't agree more. It is the only interface that could take full advantage of the benefits this would offer. Only for closed source non-modified VFR muxers and for AVS previewing. Modified VFR muxers can get the timestamp through an AVS log, as seen in the mentioned thread.


Bilu

[Atamido]

I think I have a solution to this that wouldn't require any modification to AVISynth. Simply create a DirectShow filter that utilizes AVISynth. You create a graph in GraphEdit that looks like this:

[some source] --> [decoder] --> [AVISynth4DS] --> [muxer] --> [file writer]

The theoretical [AVISynth4DS] DirectShow filter would take an uncompressed frame and the timecode from the decoder. Then AVISynth4DS would call an AVISynth script using a special plugin as the input for the script. The frame would have a frame number and be manipulated in the normal fashion. At the end of the script the manipulated frame would be passed back to AVISynth4DS and AVISynth4DS would attach the timecode back to it.

Using this method, AVISynth would not need to alter its structure in any way to accomodate VFR. Only two things would need to be developed to get this working. The AVISynth4DS filter, and the AVISynth plugin for it to input to.

[bilu]

Don't forget that sometimes the frame is dropped and never comes back - IVTC.

Bilu