How about an Avisynth fork or extender for Variable Framerate?

[bilu]

Variable framerate has its advantages, namely:

- hybrid processing without blending film to 30 fps or decimating video to 24 fps;

- RV9's Dropdupe-like filtering for anime, helps encoding;


I know Avisynth is based on AVI structure which makes it constant framerate. I also know that even if a hack was feasible it would break compatability with plugins and all the apps that currently support Avisynth.

But given the recent birth of variable framerate container like RealVideo or Matroska, I was wondering if the creation of an Avisynth-like framework for variable framerate would be possible.


Bilu

[ChristianHJW]

You guys should definitely talk to the developers of VFRE (http://corecodec.org/projects/vfre), i guess the 2 projects were made for each other, and DaveEL could act as a consultant as he has maybe some use for that also, with respect to avs2matroska ... one question tough :

Is it really necessary to fork ? Cant AVISynth be extended somehow, or is its structure to much based on VfW/AVI limitations ?

[bilu]

Originally posted by ChristianHJW
one question tough :

Is it really necessary to fork ? Cant AVISynth be extended somehow, or is its structure to much based on VfW/AVI limitations ?

Some resources:

Variable frame rate support
http://forum.doom9.org/showthread.php?s=&threadid=47203

VFR in Avisynth 3.0?
http://forum.doom9.org/showthread.php?s=&threadid=53028

3.0 Wip
http://forum.doom9.org/showthread.php?s=&threadid=48940


Just searched, will checkout now.

EDIT: It seems very difficult, but maybe something can be learned from RV9's Dropdupe filter approach :)

Bilu

[bilu]

I think I had a great idea... :D


Seems to me that there are only two uses for variable framerate (but important ones):

1) Dup decimate -> Lower the framerate of original frame to keep the timestamp; for anime

2) Dup decimate -> Keep the resulting framerate on frame; for hybrid IVTC


As seen on

RV9 Animation DropDupe Pre-filter
http://forum.doom9.org/showthread.php?s=&threadid=56564

neuron2's Dup is recommended before using this filter.
There are other great Avisynth filters out there.

For an hybrid IVTC use one could use a flag like Telecide's (debug=true) to get info to external programs.


My idea is: why not create an "Avisynth extender" that receives flags as well as the frameserved video to handle it on a variable framerate approach? Maybe Matroska-based structure would be nice and it would work with the current Avisynth without changes! :)


Bilu

[Bidoche]

I guess I could have somehow VFR in 3.0 by using the property system to tag each frame with a duration. (with huge precision to get rid of rounding errors)

But what's the point it's just internal ?
As long as avisynth is exporting through an API who doesn't support VFR, it can't reasonably support VFR.

[bilu]

@Bidoche

But what do you think about creating another frameserver, Avisynth-like, that could grab the AVS script as a CFR source?

I'm sure the knowledge put into Avisynth and its filters is more than just the AVI structure.

Having some sort of MatroskaSynth (or any other VFR container that could serve this purpose) that supports VFR plugins, an entire new project based on the Avisynth roots, could be the way to go.

By enabling the use of AVS scripts as source the transition between both projects could run smoother.


Also it may be feasible to produce CFR from VFR by establishing a maximum framerate and repeat frames as needed to fill that framerate.

Delays by repeating more than needed ( ex: repeat 4fps to make 30fps couldn't be 7,5 repetitions, should be 7 or 8) could be compensated by repeating +- 1 frame as needed. Still better for hybrid movies than blending film to 30 fps or decimating video to 24 fps.... ;)


EDIT: What a nonsense :rolleyes: . 24 fps to 30 fps means repeating one frame in every four frames. CFR means every frame plays for the same time, so it would just look like a film only filtered with Telecide() :rolleyes:

Bilu

[Sirber]

Hum... IMO It would be easy to implement it for AVISynth using a filter that drop unwanted frames on AVISynth output.

Maybe I'm wrong (I can bet on that :D)

[Bidoche]

But what do you think about creating another frameserver, Avisynth-like, that could grab the AVS script as a CFR source? Well, you don't seem to realize the amount to work it represents. By myself (if nobody helps) I may not even complete 3.0 in 2004.

Having some sort of MatroskaSynth (or any other VFR container that could serve this purpose) What you need to export to is an API, not a container.

Also it may be feasible to produce CFR from VFR by establishing a maximum framerate and repeat frames as needed to fill that framerate Possible, but it will need a smart algorithm, not just a common multiple. Because users will come up with cases where the multiple is 1000 or even more...

[Bidoche]

@Sirber

What you can do, is interface to avisynth directly, pass a script that repeat frames so you can join, and test frames for equality at the time of encoding.

[bilu]

Originally posted by Bidoche
But what's the point it's just internal ?
As long as avisynth is exporting through an API who doesn't support VFR, it can't reasonably support VFR.

What API are you referring to? Do you know any possible replacement?


This already serves as inspiration :)
http://forum.doom9.org/showthread.php?s=&threadid=63286


Bilu

[Bidoche]

Avisynth exports to VFW.
When an avs is opened, avisynth creates an IAVIFile COM object (from VFW) with the expected behavior.
It don't really mimic an avi file, but reading from an avi file.

As for another API, the problem is I don't know any.
Sometimes ago, I heard about some work in progress about an API destined to replace VFW, but no more news... (and I forgot the name)

[bilu]

Probably this one?

http://uci.sourceforge.net/

I'm sure Christian will remember it ;)

EDIT: Check Transor API in

http://www.matroska.org/

still in early states.


Bilu

[bilu]

( I hope the Matroska guys give me some help here or else this thread is dead :D )

[ChristianHJW]

Well, i really hope this thread will not become as dead as UCI is :( .... we gave up on this project, its lead developer, Alex Stewart, has meanhwile shown again, but preferred to work on the MCF specs together with the MCF founder ( you know matroska is a fork of MCF ), Lasse 'Tronic' Karkainen, but he is obviously not interested in UCI anymore.

We have meanwhile founded a replacement for it, called CoreAPI (http://corecodec.org/projects/coreapi), and we hope to be able to take the API from VFRE as a basis for it, and can build from there.

There are a couple of real experts in the FFMPEG environment who could do such an API in no time, and with a great result, but these guys prefer to make their own container format, because they mislike the size of our library and feel our specs are too powerful ( = 'bloated' ), made by 'DirectShow kiddies' ....

.... sorry i dont have any better news, for the time being neither UCI nor CoreAPI can be of much help for your project i fear :( ...

[bilu]

Originally posted by ChristianHJW
There are a couple of real experts in the FFMPEG environment who could do such an API in no time, and with a great result, but these guys prefer to make their own container format, because they mislike the size of our library and feel our specs are too powerful ( = 'bloated' ), made by 'DirectShow kiddies' ....

What container format is this?


EDIT: Is this the one (libavformat) ?

http://ffmpeg.sourceforge.net/ffmpeg-doc.html#SEC29

7.1 API
libavcodec is the library containing the codecs (both encoding and decoding). See `libavcodec/apiexample.c' to see how to use it.
libavformat is the library containing the file formats handling (mux and demux code for several formats). See `ffplay.c' to use it in a player. See `output_example.c' to use it to generate audio or video streams.

7.2 Integrating libavcodec or libavformat in your program
You can integrate all the source code of the libraries to link them statically to avoid any version problem. All you need is to provide a 'config.mak' and a 'config.h' in the parent directory. See the defines generated by ./configure to understand what is needed.

You can use libavcodec or libavformat in your commercial program, but any patch you make must be published. The best way to proceed is to send your patches to the ffmpeg mailing list.

http://www1.mplayerhq.hu/cgi-bin/cvsweb.cgi/ffmpeg/libavformat/?cvsroot=FFMpeg

Bilu

[ChristianHJW]

:) .... libavcodec/libavformat are the very basis for ffdshow/ffvfw and comprise probably the most complete collection of container and compression format libraries, like for AVI, MPEG ES, MPEG TS, MPEG PS, ASF, etc. .... ;) ....

NUT is not in the ffmpeg CVS yet i guess, but is coded from a dev who is also a core dev of mplayer and thus hosted on the mplayer CVS.

[esby]

mmmm i'm not expert in this field

BUT...

if you want to have variable framerate,
and if avisynth is limited to avi spec,

How about having an avi imitating variable frame rate first,
then trying to do it with avs...

Let me explain,
an avi can contain several video streams,
if you assign a framerate of 29.76 fps to the first video stream,
one of 23.976 fps to the second,
using both streams correctly you can have something
that imitates the vbr aspect.
Of course, frames that are at the same 'position' in both streams should be dropped in one of them. Which one, i dunno actually...

For avisynth, that means avs must be able to return two video streams, to begin with.

esby

[bilu]

@Christian

http://www1.mplayerhq.hu/cgi-bin/cvsweb.cgi/main/ .... and ? :)


Please point us there :D


Bilu

[bilu]

@Bidoche, Karl, Christian

Could this be used in Avisynth and Matroska?
http://forum.doom9.org/showthread.php?s=&postid=386944#post386944

Bilu

[Bidoche]

I gave it some tought, and I finally think that a complete and comprehensive VFR support is far too complicated to be considered.
It requires huge changes : changing random access from frame nuùbers to timecode, lots of filter logic to adapt...

Besides that's not really what you want, you only want to merge 24 and 30 fps clips (and some likewise cases).
This can be done by a layer above avisynth (like avs2matroska) which will generate the VFR content, either by dropping duplicates from a 120fps stream, or using duration flags setup by a VFRSplice function.

[ChristianHJW]

Originally posted by bilu Please point us there :D - Bilu

nope :P

Its a format written from techs for techs, nothing we would have to worry about IMHO. They didnt listen to their potential users when making the specs, something thats very normal if techs make something ;) ....

[bilu]

Originally posted by Bidoche
This can be done by a layer above avisynth (like avs2matroska) which will generate the VFR content, either by dropping duplicates from a 120fps stream, or using duration flags setup by a VFRSplice function.

@Christian

Then point us here :D



http://forum.doom9.org/showthread.php?s=&postid=386944#post386944

Originally posted by karl_lillevold
As a matter of fact is already completely open source. Anyone interested can check out the source code for our inverse telecine and de-interlacer, and improve it! The location is
producersdk/plugins/transform/videoprogressive
It is checked out as part of building producer from source code.


I was thinking about the use of this, but then again Bidoche's proposals seem much more reasonable, especially a duration flag that could be grabbed by VFR encoders or filters like DropDupe.

If such a flag could be exported then Avisynth filters like Decomb could begin using that flag (humm... 0 ms for IVTC dups? :) ) while still providing constant 30 fps stream to the VFW interface.

The Dup filter could also use that flag and information would be sent to DropDupe, making its life easier :)

Seems good enough for XVID, Matroska, MP4, Real, Avisynth, hybrid movies and anime ;)

Real would also benefit from not spending their few resources (a 4 programmers team) on filters.


The key question is: Can such a flag be exported through the VFW interface? :sly:

If so life can be easier to everybody :D (except AVI and OGM, only CFR structured)


Bilu

[karl_lillevold]

Originally posted by bilu
Real would also benefit from not spending their few resources (a 4 programmers team) on filters.
Absolutely, if your suggestion is possible. With currently just two engineers, myself included, doing all the video codec research, porting, and development, any assistance on improving the existing filters, or a framework that allows Avisynth filters to be used, would be most appreciated. Note that Producer does not actually read its input files via VFW, but via DirectShow, and DirectShow provides the VFW interface.

[bilu]

@Christian

(if you want to take down AVI for good, now is the time :D )

Shh... don't tell anyone ;)


Bilu

[bilu]

Using a flag through VFW would also be useful when previewing an AVS script through a player: if the player supports VFR it would grab the flags in every frame and act accordingly.

Unless it poses a similar problem in previewing AVS that the libavcodec has setting aspect ratio on Windows players for MPEG-4... :sly: (IIRC VFW flags started only after DShow i think)

EDIT: It's a problem for aspect ratio on AVI because AR info is only sent after the first frame has been decoded. Only on Windows, not on Linux.See http://forum.doom9.org/showthread.php?s=&postid=216624#post216624 for details.

Originally posted by karl_lillevold
Note that Producer does not actually read its input files via VFW, but via DirectShow, and DirectShow provides the VFW interface.

It all makes sense to me now: the VFW interface is estabilished after the DShow graph rendering, so any information that has to pass only after the first frame has been decoded. May not be a problem because initial(or maximum??? :confused: ) framerate is sent on the first frame.

But a question to those who know the VFW API: is it possible to use some sort of flag that is continuously sent (and may change) through the VFW interface?


Bilu

[bilu]

Idea: DShow AVS filter that grabs that info. If VSFilter can grab subtitles through out the movie, so could this filter grab framerate info ! :)

Filter source could also be used for VFR encoders.

Bilu

[bilu]

Originally posted by Bidoche
Avisynth exports to VFW.
When an avs is opened, avisynth creates an IAVIFile COM object (from VFW) with the expected behavior.
It don't really mimic an avi file, but reading from an avi file.


Originally posted by Bidoche
Besides that's not really what you want, you only want to merge 24 and 30 fps clips (and some likewise cases).
This can be done by a layer above avisynth (like avs2matroska) which will generate the VFR content, either by dropping duplicates from a 120fps stream, or using duration flags setup by a VFRSplice function.

@Bidoche

Have I misunderstood you when I suggested sending duration flags over VFW to the encoder on my last posts?

Now re-reading I got the impression that you suggested its use on a layer above Avisynth, so the flag info couldn't be started inside Avisynth itself :confused:

Bilu

[Bidoche]

What i did not mention was the 'through VFW' part, as I have no idea of if it's possible (or not)

What I imangined was the flag to be setup inside Avisynth, and the upper layer to direcly interface with avisynth in order to read it.

But if it can be passed through VFW, it's all good.

[bilu]

I've PMed Gabest and Blight about this thread.

Opinions from DShow/VFW experts sure is welcome on this subject :)


Bilu

[bilu]

Even if the VFW interface doesn't let you send information through the Video stream continuously, you probably could generate another stream just like you do with audio right now. A compatible DShow splitter would grab the streams, and probably do the frame cut / frame delay itself.

I'm not a programmer, so I don't know if this is a valid suggestion :confused:

Bilu

[Blight]

DirectShow supports non-fixed frame rate (see mixed MPEG2 24/30fps content). Each sample is given a timestamp and is synched as such. I don't really have that much low-level understanding of the subject matter to help further.

I think WMV supports mixed frame rate content, not sure about others.

[AS]

Originally posted by Blight
I think WMV supports mixed frame rate content, not sure about others.

It indeed does, I had some anime raws in wmv which were consisting of 24 fps for OP and main, and a whopping 60 fps for the ed.

[bilu]

@all

Please take a look at my post here:

http://forum.doom9.org/showthread.php?s=&postid=389270#post389270

I think it's a great idea for VFR, much more compatible, easier to develop.

Please give your feedback. Thanks.


Bilu

[bilu]

Sorry for the cross-posting, but this project has to be monitored both by the container/muxer community and the Avisynth community :rolleyes:

http://forum.doom9.org/showthread.php?s=&postid=389761#post389761

@Bidoche
I'd really like your feedback on these, here or in the other thread.
At least until someone works on it and unites the threads :)


A Summary until now:

VFR DShow filter
==========
- reads frames and timestamps and increases the frame duration as needed;
- for unmodified VFR muxers;
- VFR AVS preview;

VFR Muxer
=======
- add support for timestamp log reading;
- ignore timestamps = 0ms when they exist;

Avisynth VFR producing
===============
- add a VFR flag with three states: no VFR, compatibility (a), complete timestamp (a);
- a timestamp flag with the frame duration for each frame;
- a frame skipping process when timestamp=0ms (a);
- frame dropping in the end of the script when timestamp=0 (a);
- a timestamp log writer;

Avisynth VFR reading (especially to convert VFR -> CFR)
=============
- add support to read timestamp logs;
- ability to copy same frame for the number of times defined in the original frame;
- ability to copy same frame again if "framenr 1235 referenced to frame 1233, duration 0 ms" but not included in the duration of the original frame in the VFR timestamp - this means it is a IVTC duplicate;

if log not available:
- calc max framerate;
- divide (frame duration/max framerate) and copy frame that number of times;

it would work on anime, but never in hybrid IVTC. Log is allways necessary in this case.


(a) The Avisynth VFR process
==================
1) Complete Timestamp mode: used to generate a timestamp log that includes the dropped frames (0 ms).
Useful for VFR -> CFR conversions and VFR reading in Avisynth (needed for VFR -> CFR).

This method should be compatible with VFR-prepared filters only.
IVTC filters mark frames with 24fps duration, 30fps duration or 0ms. No frame is dropped.
Dup filters mark frames as (nr dups + 1) frame duration or 0ms. No frame is dropped.

All the VFR-compatible filters skip frames with 0ms, but don't drop them. This is to keep the complete timestamp integrity.

In the end of the script the complete timestamp log is written and frames are dropped before exporting to VFW.

2) Compatibility mode: used to generate a timestamp log without the dropped frames(0ms).
Useful whenever you need to use non VFR-compatible filters in the script.

This method should be compatible with all filters.
IVTC filters mark frames with 24fps duration, 30fps duration and drops duplicates.
Dup filters mark frames as (nr dups + 1) frame duration and drops duplicates.

Filters don't need to skip any frames based on timestamp, hence the increased compatibility.
Timestamp log only contains non-dropped frames.

In the end of the script the timestamp log is written and all frames are exported to VFW.


Note: the "framenr 1235 referenced to frame 1233, duration 0 ms" notation in the timestamp log serves to define the original frame, so that when reading for VFR->CFR in Avisynth you know which frame to copy.



Best regards,
Bilu

[bilu]

As it as been discussed in the other thread, what seems to be needed in the Avisynth core is:

- frame duration property;
- export to a timestamp log file, containing framenumbers and frame durations (maybe a filter?);

This are the top priority developments that can enable the development of VFR AVS filters, DShow filter and VFR muxer.

Bottom priority developments are:

- using VFR sources in Avisynth, translating timecodes into frame duration;
- a filter that copies frames for VFR -> CFR conversions;

@Bidoche

I'd like your opinion about this. First steps into VFR should come from Avisynth, including the timestamp log format.

Can the top priority developments be scheduled? They look like simple hacks to Avisynth to me, and they would enable development on other teams.

I still don't know if you agree with developing on this feature.


Best regards,
Bilu

[Atamido]

Originally posted by Blight
DirectShow supports non-fixed frame rate (see mixed MPEG2 24/30fps content). Each sample is given a timestamp and is synched as such. Most MPEG-2 DS filters actually produce b0rked stream without proper timecodes, but DS is pretty forgiving and the filters are able to render the frames at the proper time. One of the Matroska developers is working on an MPEG-2 reader that would give out the MPEG-2 frame with a timecode. This could then be made into a DS source filter that would pass MPEG-2 frames with a timecode so that they could be stored in almost anything.

[bilu]

@Pamel

Would that help VFR encoding ? :confused:


Bilu

[Mango Madness]

another advantage that I thought of is when dealing with anime (especially DBZ) is that a lot of the sequences have still frames (which has already been discussed thoroughly) but also have video sequences that repeat. Notice in any of the battle scenes many times the animators just get lazy or whatever and display the same pattern over and over (rocks moving back and forth; a bolt of energy that fluctuates between 3-5 frames; etc.). So if newer decoding methods existed, there would be no need to encode these sequences multiple times but rather encode it one time and have some sort of loop system that would say "play these 7 frames 10 times then break out of the loop and continue". Anyways, this was just a thought I had.