vsFilterScript: writing C++ plugins like python scripts (WIP)

[feisty2]

update:
Transpose is now replaced by the more interesting nnedi3_rpow2 (https://github.com/IFeelBloated/vsFilterScript/blob/master/Examples/nnedi3_rpow2.hxx) example. center shift correction and other features that require resampling are however not implemented for the sake of simplicity.

[feisty2]

update:
removed hasattr, it seems to fail on some corner cases.

[feisty2]

new functionality: calling python functions, the syntax is the same as invoking filters (MATLAB function call syntax)
new functionality: Frame and Function can now be map items, all VSPropTypes are now supported.
new functionality: PeekFrameFormat() for map items, if you need to convert a map item to a Frame and the PixelType of the Frame cannot yet be determined, use this function to peek the format before type conversion.

a new ModifyFrame (https://github.com/IFeelBloated/vsFilterScript/blob/master/Examples/ModifyFrame.hxx)example is added to demonstrate the use of Function, a second order example for ModifyFrame (https://github.com/IFeelBloated/vsFilterScript/blob/master/Examples/ModifyFrameExample.vpy) is also provided.

[feisty2]

update:
new functionality: support for filters that return multiple clips, this also affects SelfInvoker. SelfInvoker now returns an std::vector<Clip> if the filter returns multiple clips, otherwise, it still returns a Clip. Console.Receive() now accepts a Clip or a sequence of clips stored in any iterable container.
issue: support for calling external filters that return multiple clips is not yet implemented, this breaks the static type system of C++ since it is only known at runtime if an external filter returns a Clip or an std::vector<Clip>, I haven't figured out an elegant approach to deal with this issue.

new example: Palette (https://github.com/IFeelBloated/vsFilterScript/blob/master/Examples/Palette.hxx), this is a simple source filter showing you how to write filters that return multiple clips. it accepts an array of N numbers, and returns N blank clips, each filled with a shade of gray that corresponds to a number in the array.

x, y, z = core.test.Palette([0, 0.5, 1])
# x is filled with black, y is filled with gray, and z is filled with white

[feisty2]

can someone do a quick test and see if this thing compiles with the latest version of clang?
msvc support will soon be possible with the implementation of abbreviated templates in msvc 16.8

[feisty2]

update:
new functionality: support for calling external filters with multiple outputs.
support for filters that return multiple clips is now complete, to receive every clip returned by external filters with multiple outputs, you have to call ConfigureForMultipleOutputs(), otherwise it is assumed that the filter only has one output and only the first returned clip will be fetched.


auto ffms2 = Core["ffms2"]["Source"].ConfigureForMultipleOutputs();
auto composite = ffms2("source", "something_with_alpha_channel.mov");
auto& content_clip = composite[0];
auto& alpha_channel = composite[1];


is equivalent to, in Python


content_clip, alpha_channel = core.ffms2.Source("something_with_alpha_channel.mov")

[feisty2]

update:
revamped: various syntactic sugar for stuff dealing with multiple outputs, this affects both SelfInvoker() and the calling syntax of external filters.

for SelfInvoker():
a) if RegisterVideoInfo() returns a VSVideoInfo scalar, it is assumed that the filter has only one output, and SelfInvoker() returns a Clip.
b) if RegisterVideoInfo() returns a VSVideoInfo sequence stored in a container of dynamic length (something like std::vector), it is assumed that the filter has multiple outputs, and the number of outputs could not be determined at compile time, in this case SelfInvoler() returns an std::vector<Clip>.
c) if RegisterVideoInfo() returns a VSVideoInfo sequence stored in a container of constexpr length N (something like std::array), it is assumed that the filter has N outputs and SelfInvoker() returns an std::array<Clip, N>, in this case you can unpack the returned clip array using a structural binding:

auto [clip_1, clip_2, ..., clip_n] = SelfInvoker(...);

d) if RegisterVideoInfo() is not defined, it is obvious that SelfInvoker() will never be called and thus it doesn't matter what it may return.

for external filters:
a) if the filter is obtained by Core["namespace"]["filter"], it is assumed that the filter has only one output, and filter() returns a Clip.
b) if the filter is obtained by Core["namespace"]["filter"].ConfigureForMultipleOutputs(), it is assumed that the filter has multiple outputs, and the number of outputs could not be determined at compile time, in this case filter() returns an std::vector<Clip>.
c) if the filter is obtained by Core["namespace"]["filter"].ConfigureForMultipleOutputs(N), where N is a compile time constant, it is assumed that the filter has N outputs and filter() returns an std::array<Clip, N>, in this case you can unpack the returned clip array using a structural binding:

auto [clip_1, clip_2, ..., clip_n] = filter(...);

it is safe to specify an N that is not equal to the actual number of returned clips M. if N > M, the surplus items left in the returned array would simply be empty Clips. if N < M, only the first N clips will be fetched.



the ffms2 example above could now be rewritten in a more pythonic manner:

auto ffms2 = Core["ffms2"]["Source"].ConfigureForMultipleOutputs(2);
auto [content_clip, alpha_channel] = ffms2("source", "something_with_alpha_channel.mov");

[feisty2]

@Are_
could you compile the latest version and run a speed test for core.test.GaussBlurFast? I added a new direct access mode for maximum performance.

[Are_]

This looks really good:
Convolution:
Output 100000 frames in 9.29 seconds (10769.29 fps)

GaussBlur:
Output 100000 frames in 27.54 seconds (3631.56 fps)

GaussBlurFast:
Output 100000 frames in 9.16 seconds (10917.06 fps)

[feisty2]

update:
revamped: you may now also directly pass arbitrary number of clips to Console.Receive() as its arguments, no need to manually wrap them in a container.
removed: Buffer, it's not really that relevent and I might add it back later with various new utility stuff.
new functionality: direct frame access. when you're done sketching the prototype of your filter and ready to optimize it for speed, you should enable direct access mode by specifying the DirectAccess template parameter whenever your filter reads a frame. this exposes raw pointer access to every row of every plane of a frame. out-of-bound access detection, automatic padding and the ability to create views will all be disabled and you have to manually take care of the corner cases. with direct access mode on, vsFilterScript provides only zero cost abstraction and the execution speed of your filter will be no different from filters written directly with the low level C API.

A GaussBlurFast (https://github.com/IFeelBloated/vsFilterScript/blob/master/Examples/GaussBlurFast.hxx) example is provided to demonstrate how to write filters with direct frame access, you can compare it to the earlier GaussBlur (https://github.com/IFeelBloated/vsFilterScript/blob/master/Examples/GaussBlur.hxx) example which utilizes indirect access features like automatic padding and views for cleaner and more elegant code at the cost of slower execution speed.

[feisty2]

new functionality: full integration of C++ exceptions. with exceptions, you no longer have to manually handle any of the following errors:
a) failing to invoke an external plugin (plugin does not exist)
b) failing to invoke an external filter
c) failing to invoke a python function
d) failing to invoke SelfInvoker
... and possibly many more. SelfInvoker is now allowed to throw exceptions so the earlier restriction requiring SelfInvoker to always be successfully evaluated has been removed. Any of these errors will transparently pass through your filters and propagate to a root caller like Create() (https://github.com/IFeelBloated/vsFilterScript/blob/master/include/Interface.vxx#L48) which automatically handles any error.
To you, it would be like the error does not exist so you NEVER have to worry about errors. It's now one step closer to python scripts.

Initialize() has been replaced by normal constructors because with exceptions, it is no longer required to return a value to introspect if the filter has been successfully constructed.

[feisty2]

I have nearly finished rewriting this whole thing for API v4, and I'd like to do a survey
does any one of you, including anyone that uses the vaporsynth API in external applications, use the following functions?

createCore/freeCore (implemented)
addMessageHandler/removeMessageHandler (implemented)
getFrameAsync (implemented)
createFunc (implemented)
queryCompletedFrame (deprecated)
releaseFrameEarly (deprecated)

[Myrsloik]

I have nearly finished rewriting this whole thing for API v4, and I'd like to do a survey
does any one of you, including anyone that uses the vaporsynth API in external applications, uses the following functions?

createCore/freeCore
addMessageHandler/removeMessageHandler
getFrameAsync
createFunc
queryCompletedFrame
releaseFrameEarly

createFunc/createCore/freeCore - will be used in applications that manually build graphs
addMessageHandler/removeMessageHandler - vspipe/vsedit/just about all consumers of vsscript will be using it
getFrameAsync - vspipe/vsedit/just about all applications retrieving frames
queryCompletedFrame - deprecated and 0 known uses in the wild
releaseFrameEarly - 0 known uses

[feisty2]

I have implemented the wrapper for createFunc() in the form of Core.Mount() (https://github.com/IFeelBloated/vsFilterScript/blob/master/include/Core.vxx#L114), but I'm not sure if the implementation is correct because this function is rarely used and certain cython functions relating to it seem "fishy". you're welcome to examine the code of Core.Mount or play with it if you're familiar with createFunc().

Core.Mount() could be used to mount any callable entity (function, function pointer, lambda expression, or any type that overloads operator()) compatible with any of the following signatures:

auto()
auto(auto Arguments)
auto(auto Arguments, auto Core)
// the return value may be of any type supported by VSMap, or an iterable container of any type supported by VSMap, or void

into the VS core, and the mounted function could be called in any context. for instance, you may mount a C++ function and assign the mount point to a frame property, and call the mounted function in a vpy script. the following example works as expected.


// C++ code, inside getFrame()
auto f = [](auto args) {
auto msg = static_cast<std::string>(args["msg"]);
vsapi->logMessage(VSMessageType::mtWarning, msg.data());
throw RuntimeError{ "hello world" };
};
ProcessedFrame["test"] = Core.Mount(f);


#Python script
clip = core.test.Test(clip)
clip.get_frame(0).props['test'](msg = 'aaaaa') # prints 'aaaaa', then prints 'hello world' in the error message.


you cannot currently mount non-void functions even though this functionality has been implemented, because:
a) the cython function Func.__call__ does not seem to handle the return value of the mounted function properly, the return value seems to be discarded in the process.
b) the type deduction required by Core.Mount() is so complex that it actually crashes GCC10 if you specify a non-void function to it (internal compiler error: in cp_get_fndecl_from_callee, at cp/cvt.c:1000). I'll see if this problem has been fixed in GCC11 and file a bug report if not.

[feisty2]

I updated my toolchain and the compiler bug mentioned above has been fixed in GCC trunk. I did some quick tests and Core.Mount() properly mounts any supported function-like entity, with a void or non-void return value. Any mounted function can be properly invoked in C++ programs, however you still cannot call non-void mounted functions in python and that's something only @Myrsloik can fix. The corresponding cython code has been broken since 2014 and still hasn't been fixed.

[Myrsloik]

I updated my toolchain and the compiler bug mentioned above has been fixed in GCC trunk. I did some quick tests and Core.Mount() properly mounts any supported function-like entity, with a void or non-void return value. Any mounted function can be properly invoked in C++ programs, however you still cannot call non-void mounted functions in python and that's something only @Myrsloik can fix. The corresponding cython code has been broken since 2014 and still hasn't been fixed.

YES! Your software is at my mercy! MUAHAHAHA

[feisty2]

well, I don't really have a problem with this thing being partially broken since it's rarely used anyways...
I simply wanna do some tests to verify if my implementation is correct. I'll just move on to the next function in line if there's no plan to fix this any time soon

[feisty2]

I have implemented getFrameAsync() as Node::ExpectFrame() (https://github.com/IFeelBloated/vsFilterScript/blob/master/include/Node.vxx#L90). This is yet another function that I'm not 100% sure if I have implemented correctly since I don't have a standalone VS application like vsedit to test it. Based on my quick (and evil) test which invokes this thing inside a filter's constructor, it seems to work properly. After reading the internal code of getFrame()/getFrameAsync(), I realized it's actually a lot simpler than I originally thought. I don't need to mess with threads and locks, getFrameAsync() already renders the frame in a new thread and I simply need to return a future and fulfill the promise inside the callback.

a call to Node::ExpectFrame() instantly starts rendering the requested frame in a new thread and returns a future that lets you retrieve the frame some time later when needed.

auto f = Clip.ExpectFrame(n); // instantly start rendering frame n in a new thread
... // do something else while waiting for the rendering completes
auto frame = f.get(); // now retrieve frame n, if the rendering hasn't finished yet, block the current thread until the frame is available and retrieved.

[feisty2]

I have finally implemented support for custom message loggers by year 2021
the mutex thing really got me good... (https://github.com/IFeelBloated/vsFilterScript/blob/master/include/Logger.vxx#L45)

you may send a message to all registered loggers by calling Core.DebugPrint/Print/Alert/CriticalAlert/Abort(Message), or send a message to a particular logger x by calling x.DebugPrint/Print/Alert/CriticalAlert/Abort(Message)

Core.TetherLogger() tethers a logger function to VS and returns a corresponding logger object. the logger function may be any callable entity compatible with any of the following signatures:

auto(auto Message)
auto(auto MessageType, auto Message)

"MessageType" may be of type auto/MessageTypes/VSMessageType/int, auto assumes MessageTypes
"Message" may be of type auto/const char*/std::string_view, std::string or anything constructible from const char*, auto assumes std::string_view

the logger object is reference counted and the corresponding logger function will be automatically removed from VS when the reference count drops to 0. alternatively, you may call logger x's member function x.Untether() to remove its corresponding logger function early, or x.Detach() to release the ownership of its corresponding logger function. The detached logger function will be permanently installed into VS and is removed only when the VS process terminates (API v3) or when the core is freed (API v4). you may directly add a permanent logger by calling Core.InstallLogger() which is equivalent to TetherLogger().Detach()

[feisty2]

soon this thing will have 100% coverage of the C API, once I fill in the last missing piece of the puzzle (createCore/freeCore)
yay!

[feisty2]

100% coverage (excluding 2 deprecated functions) of the C API achieved! finally!

[feisty2]

@Are_
could you run a speed test for GaussBlur and GaussBlurFast again? I applied more optimizations, mostly tags for more aggressive inlining and less indirection (so probably less cache miss)

[Are_]

Ran them three times and seemed consistent.
Old GaussBlur:
Output 100000 frames in 27.22 seconds (3674.02 fps)
Old GaussBlurFast:
Output 100000 frames in 5.92 seconds (16891.87 fps)
New GaussBlur:
Output 100000 frames in 31.10 seconds (3215.69 fps)
New GaussBlurFast:
Output 100000 frames in 5.90 seconds (16959.92 fps)

[feisty2]

it just occurred to me that I actually changed the default padding function to "reflect" in the current version, it's a bit slower but better preserves the structure of the image, and thus GaussBlur also got a bit slower. however GaussBlurFast was not affected by such change and it is indeed now a bit faster, that's enough to prove that the optimizations do work as expected :-)

[vcmohan]

The filter skeleton generator link does not exist. Get 404 error.

[lansing]

Just curious, will this works with making filter loader plugins like Vdubfilter? I tried with C API but got lost really quick.

[feisty2]

The filter skeleton generator link does not exist. Get 404 error.

it is recommended to use the stable branch for now, you can find the skeleton generator here: https://github.com/IFeelBloated/vsFilterScript/blob/stable/include/Interface.vxx

[feisty2]

Just curious, will this works with making filter loader plugins like Vdubfilter? I tried with C API but got lost really quick.

something like std.LoadPlugin? presumably yes, I can write an example if necessary

[lansing]

something like std.LoadPlugin? presumably yes, I can write an example if necessary

Like avs.LoadPlugin

[feisty2]

this thing is now feature complete and I renamed it vapoursynth++

things left to do:
code cleanup
documentation
replace header files with C++ modules

[feisty2]

I will make a few final updates to the current API v3 branch and I will then freeze the v3 branch and move on to v4

[feisty2]

MSVC support should be ready soon, considering MSVC is finally feature complete for C++20