Wrapper element for AviSynth filters for GStreamer - GSoC project

[Wilbert]

It turns out someone will be making a great project for GStreamer and for us!

The goal is to make some AviSynth video filters available as GStreamer pipeline elements. This would also bring AviSynth v3.0 vapor-ware closer to reality.

The project (http://socghop.appspot.com/student_project/show/google/gsoc2009/uraeus/t124025048258) consists of:
1) Implementing frame number information extraction in demuxers
2) Implementing frame-based seeking
3) Implementing a video ringbuffer
4) Implementing GstAVSynth wrapper library
5) Implementing GstAVSynth headers/libraries for AviSynth filters to build against

more info: http://gstreamer.freedesktop.org/wiki/AviSynth

So, post all your ideas about it here. I will also ask LRN to post here!

IanB (i hope you don't mind posting this) gave him the following advice:

You may want to consider using Avisynth as a black box script rendering engine rather than reinvent the whole wheel. You would just need to write a GStreamer_Internal_Source() module and Initialise the Last variable to an instance of this in your IScriptEnvironment, then Import() a script file or Eval() a string buffer containing script verbs.

[LRN]

Actual project is more...modest, i should say. Read the wiki page, it should contain more or less actual info. I'll set up a git repo right after i write something useful :) And i'll try to keep wiki updated.

[bilboed]

Hi Wilbert and others,

Just thought I'd introduce myself. I'm the GStreamer mentor for this avisynth project and originally proposed this project (deeeply need it for the PiTiVi video editor, as well as other GStreamer apps).

Just to avoid any confusions, and maybe the wiki isn't clear enough on that point, the goal is to be able to use as many AVISynth filters as possible through GStreamer with minimal filter code rewrite. It's not meant to replace/rewrite the script engine... yet :)

Theoretically, after this SoC, we should have:
* a GStreamer element base class for AVISynth filters
* a 100% API backward-compatible avisynth.h with a little bit of gobject/gst-ification in it and a new way to expose internal information to make it even easier to use through GStreamer (like detailed properties (name, description, type, limits), number of frames needed to output one, ...)
* a collection of filters ported to the GstAVSynth API, along with documentation of how to make filter code a bit more portable accross multiple platforms.

That should enable people to use avisynth filters in any GStreamer pipeline, with any other existing gstreamer plugins (decoders, encoders, (de)muxers, network transport, RTP, editing plugins, gst-ffmpeg, ....) and any existing GStreamer application.

The goal for this SoC is not to have a auto-magic wrapper than can expose existing binary .dll in GStreamer ... although it should bring us much closer. On Linux, with the help of Wine (for which there's another accepted GStreamer SoC project) and this SoC project we should be able to do that (although the GstAVSynth-enhanced ones will work slightly better) eventually.

[LRN]

A bit of news:
This thing actually works now. The wrapper doesn't know how to handle events (such as seek) and queries yet, but a simple pipeline with SimpleSample filter (from AviSynth tutorial) seems to be working, and it doesn't even leak too much memory.
Code is available at http://github.com/LRN/gst-avsynth

[LRN]

Something you may want to hear, if you're interested in AviSynth development.
When i was deciding which parts of AviSynth i should implement, and which i should not, i've picked frame buffer recycling to be one of the latter. I decided that it just not worth the trouble and that performance gain from that feature is negligible.
Today i came around to actually test this speculation on AviSynth itself.
I've built two versions of AviSynth 2.58 - one with frame buffer recycling disabled (patch is attached, unified diff) and another without any changes (for reference).
The patch in question is rather crude, it just forces ScriptEnvironment::GetFrameBuffer2() to clean up unreferenced frames and allocate a new frame each time it is called (normally it performs a cleanup only when it runs out of memory and it always tries to re-use some old frames instead of just allocating them). The patch does not remove any upkeep introduced by LinkedFrameBuffer and friends, since that would require changes in a lot of places (such as cache manager), and i'm too lazy for that. And VideoFrame objects are still recycled in both versions.

I've used two scripts - a fast script and a slow script.
Fast script:

AVISource("d:\big_buck_bunny_480p_stereo.avi", audio=false)
trim(0,500)
crop( 20, 20, -20, -20)
LanczosResize(600,480)


Slow script:

AVISource("d:\big_buck_bunny_480p_stereo.avi", audio=false)
trim(0,500)
crop( 20, 20, -20, -20)
LanczosResize(600,480)
fft3dfilter(sigma=2.00,bw=32,bh=32,oh=16,ow=16,bt=3)


Each script was tested with 3 different settings - no encoding/no writing, encoding/no writing, encoding/writing to disk:

ffmpeg -i d:\big_buck_bunny_480p_stereo.fast.avs -vcodec copy -f null f:\bbb1.mkv
ffmpeg -i d:\big_buck_bunny_480p_stereo.slow.avs -vcodec copy -f null f:\bbb1.mkv
ffmpeg -i d:\big_buck_bunny_480p_stereo.fast.avs -vcodec mpeg4 -b 1000k -f null f:\bbb1.mkv
ffmpeg -i d:\big_buck_bunny_480p_stereo.slow.avs -vcodec mpeg4 -b 1000k -f null f:\bbb1.mkv
ffmpeg -i d:\big_buck_bunny_480p_stereo.fast.avs -vcodec mpeg4 -b 1000k -y f:\bbb1.mkv
ffmpeg -i d:\big_buck_bunny_480p_stereo.slow.avs -vcodec mpeg4 -b 1000k -y f:\bbb1.mkv


Each configuration was executed 3 times and the execution time was measured (i've used standard Windows %time% variable for that).

Results are attached (ISO/IEC 26300:2006 spreadsheet, packed into 7z/lzma).

Conclusion: in this particular test case (and with this particular method of recycling removal) video frame buffer recycling provided from 1.6% (slow script, with encoding, but without writing to disk) to 6.4% (fast script, without encoding, without writing to disk) performance increase. The difference could be larger on complex scripts. However it is, in my opinion, negligible when compared to the delay introduced by slow filters (deinterlace/denoise) and slow encoders.

P.S. GstAVSynth goes fine. Yesterday it managed to process a real-life AVI/XviD video file with SimpleSample filter from the beginning, skipping the middle, to the end without segfaulting. That's a good omen.

[IanB]

@LRN,

Try this scriptSetMemoryMax(256)
AVISource("d:\big_buck_bunny_480p_stereo.avi", audio=false)
StackVertical(Last, Trim(100,0))
Trim(0, 500)

VFB reuse is for 2 reasons primarily :-

1. This is how the cache works, it tracks those not in use, ready to be recycled VFB's and rescues them if ever needed (and they are still available).
2. To prevent heap fragmentation and combing.

The cache is only useful when a given calculated frame is needed twice in a script. Both your example scripts are purely linear, so you are testing the NULL set.

[LRN]

With that script (tested only on null encoding without writing to disk):
Without recycling:
101.03 seconds
100.95 seconds
101.25 seconds

With recycling:
100.90 seconds
101.05 seconds
101.21 seconds

If you look at the patch, you'll see that i was testing the speed impact of the recycling of allocated buffers, which appears to be insignificant. If speed was not the goal of buffer recycling - well, fine (as for heap fragmentation and combing, i can't really comment on it - never considered it a problem). And this isn't really related to caching, or maybe i just don't understand how it is possible to recover a particular calculated video frame from a list without knowing exactly where did that frame came from (never tried to dig deeper into the cache manager, maybe it does track that somehow). If the same list was used for both recycling and caching (which is my impression from reading your last post), it seems that disabling buffer recycling does not affect frame caching (if it did, i would have experienced a dramatic performance drop).

[LRN]

Today i've got Telecide() from Decomb working (somewhat, first run was without any parameters -> results were not spectacular). And bilboed managed to compile Depan on Linux, you can take a look - here (http://github.com/bilboed/depanestimate/tree/master) . At the moment there are plans for moving each ported filter into separate repository. Don't know yet how that would work out.

But, there's a fly in the ointment. A few days ago a decision was made to make GstAVSynth filters depend on GLib. Which means that for filter developing it would be necessary to have GLib headers at the least, and GLib import libraries at the most (i'm assuming that you do have GLib binaries along with GStreamer and GstAVSynth). A filter may or may not use GLib functions, but it is necessary for cross-platform compatibility to use GLib types (integer types mostly). For those of filter developers who still cling to MSVS, adding GLib dependence shouldn't be too much trouble (and there's no need to build the stuff - all necessary binaries, import libraries and headers are available at GTK+ site, in download section).

As IanB have predicted, there were some problems with assembler. It wasn't too bad with Decomb, because it comes with C alternatives, but could be a problem with asm-exclusive filters. GStreamer runs on may platforms, and it would be required to port any asm code to *each* platform separately for it to continue to work. Which makes sense, because if an asm code is not using platform-dependent features to run faster, it's typically *not* running [much] faster than C equivalent.
As for Decomb's MMX-optimized asm routines, i've used asm converter to transform these to be GCC-compatible. Still had to fix some things manually. And i'm still not sure i've fixed everything correctly (will know later, after i test Decomb further).

[LRN]

Yeah, Telecide() seems to be working 100% correctly. And because i didn't hack Decomb itself too deep, it is safe to assume that FieldDeinterlace() works too (although it's still untested).

If you have a filter or a source file in mind that, in your opinion, would be a good test for GstAVSynth - don't hesitate, name it (or give it to me).

[buzzqw]

yadif ?

it's a nice and fast deinterlacer

.. and btw.. i haven't fully understood this project
the aim is to create a gstreamer interface to avisynth ?

i love it! since i run on linux

any ready to test build (and relative howto) ?

thanks

BHH

[LRN]

No builds yet (btw, if you're on Linux, you should be capable of building it yourself; source code is available and bilboed made sure it's Linux-compatible...mostly...).
I will definitely write AviSynth->GstAVSynth porting HowTo somewhere around August 17th. And will, maybe, release a tarball. And probably a set of Windows binaries.

It's not a "gstreamer interface to avisynth", it's "gstreamer interface to avisynth filters". Which means that you can take a source code of a filter, change a few relatively minor things (which things - will be explained in HowTo) and recompile it against gstavsynth_sdk.h and GLib headers. If you don't use any GLib functions in a filter, you will only need GLib headers, no import libraries are necessary (at least at this moment, this may still change). And then you will be able to use this compiled filter with GStreamer the same way you're using normal GStreamer elements. Here's a hypothetical example of trans-coding a DVD video to H.264 via gst-launch:
gst-launch-0.10 rsndvdsrc device="c:/A_DVD_directory_containing_VIDEO_TS_directory" ! decodebin2 ! ffmpegcolorspace ! avsynth_Telecide post=2 vthresh=41 ! ffmpegcolorspace ! x264enc ! filesink location="c:/transcoded_dvd_video.264"
It's hypothetical, because it's somewhat hard to read MPEG2 video from DVD without going through title menu. And reading directly from VOBs may make GStreamer report wrong video duration.

[buzzqw]

yup.. interesting... i hope for a quick and dirty already build compilation... :D

BHH

[LRN]

yadif ?

it's a nice and fast deinterlacer

yadif is an AviSynth port of MPlayer's deinterlacer. That is, it is originally written in GCC-compatible assembler. Fizick ported it to AviSynth, that is - to cl.exe. Porting it *back* again would be foolish (it would have been better to port it from MPlayer if i wanted to). BTW, GStreamer does have enough deinterlacers for now, i think (recently GstAVSynth-ported Decomb + native GstDeinterlace which comes with "Greedy", "Linear", "Bob", "TomsMoComp" and some other stuff).

[Fizick]

No, I used GCC, not cl.exe, so porting "back" is not so foolish ;)

[LRN]

Fizick, you've just ruined my day :)

Here i was thinking that no one uses AviSynth C plugin interface (so i did not implement it at all). And *BAMPF* - here you are, with yadif which DOES use it. Probably the only AviSynth plugin out there that does (at least i was not successful in finding such plugins).

That, of course, is not an obstacle for me. Since C API is a wrapper around C++ API, it wasn't hard to implement C API (just copied the original and changed a few things).
The obstacle is the fact that you've linked yadif.dll *directly* to avisynth.lib. Because AviSynth C API is designed improperly. You've probably noticed that C++ API passes all the necessary stuff via objects that implement I*blah-blah-blah* interfaces. C API does no such thing. So you've linked yadif to avisynth. You shouldn't have done that, really. Instead you should have filed a bug pointing out that C API is b0rked. Now i have to think of something (and i don't feel like linking to gstavsynth - that would violate the whole idea of a *plugin*).
I have an idea...let's hope it works...because if it doesn't, i'll have to just tell people that C plugins are not supported (which is kinda weird, since GStreamer itself is pure C and supports C plugins quite well).

[Fizick]

Avisynth C API existed in several versions. I use latest from avisynth 2.5.7, not from Kevin. I remember the Kevin avisynth.lib was smaller.
Once upon a time there was the Yadif in MPlayer. I take it and make an Avisynth plugin. It is C-plugin because I do not know GCC assembler and how to convert it to NASM or MASM.
My Yadif works OK. Therefore, C-API works OK.
I am poor plugin writer only. Do not ask me too hard questions. :)

I hope your Idea works. If not, I can recompile yadif with any updated API you provide :)

[Gavino]

AviSynth C API is designed improperly.
You may be right, but do you have a proposal for a better one?

[LRN]

Yep. The idea is to pass function pointers with AVS_ScriptEnvironment. Something akin to C++ function table, but explicit.

Also, i've found something that confuses me. It's all over avisynth_c.cpp, avs_new_video_frame_a() in particular:
extern "C"
AVS_VideoFrame * AVSC_CC avs_new_video_frame_a(AVS_ScriptEnvironment * p, const AVS_VideoInfo * vi, int align)
{
p->error = 0;
PVideoFrame f0 = p->env->NewVideoFrame(*(const VideoInfo *)vi, align);
AVS_VideoFrame * f;
new((PVideoFrame *)&f) PVideoFrame(f0);
return f;
}
As far as i can tell, the idea is to call C++ method NewVideoFrame, get the new frame copy it, place the pointer to the copy into AVS_VideoFrame pointer (casting it to AVS_VideoFrame*), and return this pointer.
So, following that logic, new((PVideoFrame *)&f) PVideoFrame(f0); should call the PVideoFrame constructor (to make a copy) and write the resulting (PVideoFrame *) pointer into f (casting f as (PVideoFrame *)).

However, MSDN (http://msdn.microsoft.com/en-us/library/kewsb8ba%28VS.71%29.aspx) says otherwise. It says that this (new(foo) bar(baz);) construction calls the constructor bar with argument baz and, if the constructor throws an exception, it calls bar::operator delete(void *, typeof(foo)). In this case that would mean PVideoFrame::operator delete (void *, PVideoFrame *). MSDN also says that if such operator is not defined the memory allocated by the constructor will not be deallocated, i.e. it will leak. GCC, naturally, reports this as an error.

So...what's up? What was the intention behind this line? Am i right in assuming that this was intended as an assignment?

[IanB]

PVideoFrame is a smart pointer to a VideoFrame. When the PVideoFrame goes out of scope it automatically decrements the use count on the contained VideoFrame, when it gets to zero it is deleted.

Kevin Atkinson's code is a sneaky way to hold onto the smart reference. Note the new is for a pointer to the smart pointer using the copy constructor for the smart pointer as the initializer. There is no actual memory allocation because pointers are base types and handed around by value.

So yes it is an assignment that defeats the smart reference mechanism. The user is honour bound to call avs_release_video_frame to enact the smart reference mechanism when they have finished with the VideoFrame.extern "C"
void AVSC_CC
avs_release_video_frame(AVS_VideoFrame * f)
{
((PVideoFrame *)&f)->~PVideoFrame();
}
Have a look at the generated assembler for _avs_new_video_frame_a@12 it may help you understand the horrid convoluted code path involved.

[Gavino]

Yep. The idea is to pass function pointers with AVS_ScriptEnvironment. Something akin to C++ function table, but explicit.
That's what I was thinking, but I wondered if you had got as far as a concrete specification for a new API. ;)
(ie a revised avisynth_c.h)

[IanB]

With exported API entries you can call these from VB, Delphi, etc... With function pointers you cannot. :(

[Myrsloik]

With exported API entries you can call these from VB, Delphi, etc... With function pointers you cannot. :(

You can't? I'm convinced I can make it work in Delphi/freepascal without even having to use the more obscure compiler directives.

[IanB]

@Myrsloik,

Okay, yes I could also deceive Delphi into using stray numbers as foreign pointers, but one really needs to know what one is doing, where as with exported API entries you only sort of need to know what you are doing. :D (6 == 0.5 Dozen)

[SEt]

I also think that Avisynth C API is horrible. Ideally it should look like DirectX API where you call the same functions and use the same headers in C and C++.

C++ API isn't great too - too much is embedded into each plugin while i think all logic should be in the Avisynth itself. Sure it will be a bit slower to call short functions from Avisynth through pointer than to inline them at compilation stage, but this cost is nothing compared to actual frame processing and would allow extending and changing the way Avisynth internally works (uhm, cache) without breaking compatibility with plugins.

[MfA]

Well, that is one of the 2.6 goals :)

[Myrsloik]

@Myrsloik,

Okay, yes I could also deceive Delphi into using stray numbers as foreign pointers, but one really needs to know what one is doing, where as with exported API entries you only sort of need to know what you are doing. :D (6 == 0.5 Dozen)

Stray numbers? Last time I checked Delphi could do just about everything C can in terms of function and type declarations. Would be no more stray numbers than when using GetProcAddress after LoadLibary...

Why not make a clean C API and then make a separate wrapper for it to turn it into "real" exported symbols? Seems like a good compromise since most languages that can't link with C-ish stuff directly usually aren't suitable for plugins. On the other hand they could use avisynth as a library for manipulating video with no problems...

[sl1pkn07]

is possible use Xine or Phonon instead of gstreamer (or other framework)?

IMO, gstreamer is trash

greetings

(for linux plataform. of course)

[MfA]

It's a gstreamer SoC project, what do you think?

[sl1pkn07]

ops. sorry :S

[LRN]

OK. Check the git repo.
Basically i had to rewrite some parts of GstAVSynth in C (not a pure C, since i had to use C++ compiler), including video frame cache, and also implement Clip-based class system in C (with inheritance!) so that C filters can be written in the same manner as C++ filters (by deriving from GenericVideoFilter). However, the whole memory management for C and C++ filters is still separate, and you still have your C++ sdk header full of code, with only parts of it being internally mapped to the corresponding C API functions/structures (whenever you see a 'virtual' keyword in a function declaration in C++ sdk header, it means that this function is internally overridden by GstAVSynth), most notable examples being AVSValue, PClip and PVideoFrame.
Previously i've tried to implement C API as a wrapper around C++ API, but that didn't work because i already had some C++ API classes with virtual functions, which broke all my attempts to copy memory from C++ object into a C structure.

This is a kind of bad news for C plugin writers, because new C API is much more different from original than my C++ API implementation was different from original C++ API. But it's better (i think) :)

Check C sdk header for some instructions (grep for "Example"). And i've added some explanatory comments to yadif port.

[LRN]

As per bilboed's request i've marked all non-trivial parts of the code that were taken from AviSynth. You can find them by greping for "FROM_AVISYNTH_BEGIN" and "FROM_AVISYNTH_END". Didn't place these in sdk headers, because they are completely copied from AviSynth (well, maybe except for C API header, since i've changed much there...).
If you find other code pieces worth marking as borrowed - tell me.

[LRN]

Here are two separate repos for Decomb and Yadif ports:
http://github.com/LRN/decomb
http://github.com/LRN/yadif
Subsequently i've removed them from the main GstAVSynth source tree (only test-filter is left, since it does not deserve its own separate repository).

These separate repositories come with simple makefile-based build system that does no require autotools. That is, you can build Decomb and Yadif with just MinGW, GNU Make and cmd.exe (i'm still thinking on how to provide GstAVSynth and GStreamer binaries and dev files in the most convenient way).

Also i've started writing the porting guide. You can read the draft in the repo (http://github.com/LRN/gst-avsynth/blob/5936b4cfdfad3072b85c27a36eb4e0190d1a53a7/docs/porting_c.txt). Also a C++ version (http://github.com/LRN/gst-avsynth/blob/5936b4cfdfad3072b85c27a36eb4e0190d1a53a7/docs/porting_cpp.txt).