AviSynth+ 3.5: Linux, macOS, and BSD support

[pinterf]

And avspmod and ffmpeg

[MeteorRain]

Had a very quick testing on avs+ with x265 under Debian buster, and it worked.

I'm working on bring a few other stuff to my repo, and when that's done, I'm gonna start porting common filters.

[pinterf]

Which are common filters? I have some as well, no need to work twice on the same topic.

[stax76]

c++ != c

[qyot27]

c++ != c
Which is this in response to?

x264:
https://code.videolan.org/videolan/x264/-/blob/master/input/avs.c#L49

FFmpeg:
http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavformat/avisynth.c;h=55a2efd884e287e8b3aacc8f339235059049a37a;hb=0830e9116f786572865a9c800a9156d0c4294f27#l38

Or better yet, actually look in FFmpeg's compat/avisynth directory: only avisynth_c.h, not the C++ avisynth.h header. (http://git.videolan.org/?p=ffmpeg.git;a=tree;f=compat/avisynth;h=58fb122b62b3cde47d4d629ec16154ecf5663bc0;hb=0830e9116f786572865a9c800a9156d0c4294f27)

And all the AviSynth library functions that get invoked are the avs_func_name type (and AVSC_DECLARE_FUNC/LOAD_AVS_FUNC) that only the C interface exposes.

[stax76]

Thanks for the links, I'll have a look.

Response to saying megui using C instead of C++ interface.

https://sourceforge.net/p/megui/code/HEAD/tree/AvisynthWrapper/trunk/AvisynthWrapper.cpp#l433

I've built my own wrapper that is similar but I have wondered if I could use avs and vs from .NET directly without building a wrapper, .NET has first class C and COM interop support for sure but maybe it's not worth the trouble and the interop code would be longer and more complicated than building a wrapper with a very simple interface, it's difficult to know what is better without actually trying to build it. MysteryX wrote some vs code but for avs only the vfw route has been taken before.

https://github.com/staxrip/staxrip/tree/master/FrameServer

https://github.com/staxrip/staxrip/blob/master/General/FrameServer.vb

https://github.com/staxrip/staxrip/blob/master/General/VideoRenderer.vb

So far reading the colorspace is the only thing I used the wrapper for which could not be done by vfw.

[MeteorRain]

Which are common filters? I have some as well, no need to work twice on the same topic.

Unless you plan on packing Debian packages?

If you get your plugins working on any Linux environment, I'll just need to write some meta files (debian packer, docker file, etc) and shouldn't be twice of work.

[Groucho2004]

Response to saying megui using C instead of C++ interface.I stand corrected. I probably got it mixed up with something else...

[MeteorRain]

Seems like Linux version doesn't have a default autoload. Maybe we can follow VapourSynth convention and autoload $libdir/avisynth?

For anyone who wants to try x265 and avs+ under Debian buster and Ubuntu bionic: https://yuuki-deb.x86.men/

[pinterf]

Unless you plan on packing Debian packages?

If you get your plugins working on any Linux environment, I'll just need to write some meta files (debian packer, docker file, etc) and shouldn't be twice of work.
Packaging is O.K. and welcomed, I meant that the source itself needs sometimes heavy refactoring, mainly because of SIMD processor level separation. I experienced it on avs+ source, took quite a lot of time, big codebase takes its toll. So my aim in general to be able to build them on clang and gcc as well, once compilation works I'm finished with my part.

[pinterf]

Seems like Linux version doesn't have a default autoload. Maybe we can follow VapourSynth convention and autoload $libdir/avisynth?

Good idea, let's see what qyot27 says, we were just happy to make manual AddAutoloadDir working in the final rush.

[qyot27]

My general assumption was that when autoloading by default was enabled, it would be $libdir/avisynth. When you build with MinGW, that's already where I'd made the plugins (ConvertStacked, ImageSeq, TimeStretch, and VDubFilter) get installed, even if it's sort of pointless on Windows.

What magiblot had mentioned in the plugin loader discussion was having it set up to use an external config file, as is usual on Linux, etc. That is the most portable way to do it (heck, even Windows could use something like that for portable builds).

On Linux, the two most logical places it could go are /etc (for the system, maybe in /etc/avisynth if we want to add more configuration files in the future) and $HOME/.config/avisynth (for a single user). The presence of one in $HOME/.config/avisynth would need to take precedence over the one in /etc, because it would be the one a normal user can override, add to, and edit as they like (it also would be up to the user to create the one in $HOME/.config/avisynth, but we need to be able to read from it first if it exists). The one in /etc would need to restrict itself to just the basic $libdir/avisynth directory, because having it installed with CMake would mean getting overwritten every time we build and install the library. Name-wise, 'avisynth.conf' probably makes the most sense.

The config file would have an autoload_directory= field that can hold a list of directories to autoload from (with the default being $libdir/avisynth, $libdir having been populated by CMAKE_INSTALL_PREFIX when AviSynth+ was built).

[MeteorRain]

/etc/avisynth
$HOME/.config/avisynth
'avisynth.conf'
with the default being $libdir/avisynth


All make sense.

[qyot27]

The good news: it didn't take a whole lot to get Clang 9 to compile the source on Ubuntu 19.10.

The not-so-good news: it didn't fix macOS or FreeBSD issues with Clang, even if I grab Clang 9/llvm-9 from pkg or Homebrew rather than trying to use the default versions.

[MeteorRain]

info.h needs to #include <iomanip> to compile under gcc.

[pinterf]

Thanks. Good news if that's all. I'm cherry-picking mt and other fixes from Neo branch, so this is only a temporary happyness :) I have almost cherry-picked a whole tree, no, a whole orchard, picked lines depend on other things, if I pull them they needs quadruple times other things; even the merge is huge work, and he created those changes from scratch.

[MeteorRain]

Yea I tried to pack git~d438bd5 and failed. Adding that include fixed the issue -- at least for compiling. Very appreciate if someone can push a quick fix to that repo.

Meanwhile I was thinking where to put AVS+VS dual interface filters. I'm thinking about a dedicated directory (like /usr/lib/ARCH/avisynth+vapoursynth/) for those filters and then create sym links to both avisynth and vapoursynth directories. Any thoughts?

[pinterf]

It's there.

[real.finder]

since there are work on autoload and AddAutoloadDir can we have fix for these? :) https://github.com/pinterf/AviSynthPlus/issues/11

[pinterf]

I haven't closed the ticket so yes.

[qyot27]

Yea I tried to pack git~d438bd5 and failed. Adding that include fixed the issue -- at least for compiling. Very appreciate if someone can push a quick fix to that repo.

Meanwhile I was thinking where to put AVS+VS dual interface filters. I'm thinking about a dedicated directory (like /usr/lib/ARCH/avisynth+vapoursynth/) for those filters and then create sym links to both avisynth and vapoursynth directories. Any thoughts?
I've not thought a lot about it in the case of things that are only plugins (LSMASHSource would have been one such test-case, but they're separate as currently implemented).

For normal libraries that happen to have plugin interfaces (like FFMS2) it makes sense to just install them into the normal $libdir and then symlink into the two directories based on whether the plugin interfaces were activated (and which one, if they can be toggled separately).

[MeteorRain]

I have a few filters that I equipped with dual interface. I'm planning to test compile lsmash works under Linux as one piece. I'm not a fan of having dedicated purpose libraries dangling in lib dir, so I'm heading for libdir/avisynth+vapoursynth if no one objects.

[MeteorRain]

Pinter, I'm not sure if it's only needed for GCC or should be included for all non-MSVC compilers or all compilers.

You can check the iomanip (https://www.cplusplus.com/reference/iomanip/) header and it seems it should be included regardless.

[MeteorRain]

Some updates:
Latest AVS+ compiles. I have AviSynth+ 3.5 (r3100, HEAD, x86_64) with me.
HolyWu LSmashWorks compiles with a few tweaks.
I managed to encode using x265-yuuki with lsmashsource into mp4 output, it went smoothly.

I still have to AddAutoloadDir("/usr/lib/x86_64-linux-gnu/avisynth/") because that piece of code hasn't reached there yet.
I ended up putting the filter inside /usr/lib/x86_64-linux-gnu/dualsynth/ and sym linking it to both directories.

[qyot27]

With 3.5.1, several things have been fixed:

FreeBSD and macOS can use AddAutoloadDir, although in FreeBSD's case it's required to first mount procfs. FreeBSD does have a different way of gleaning the information needed, but I'm not sure if that's applicable to the other *BSDs as well. In NetBSD and DragonFly BSD, the procfs thing should work, although like FreeBSD, it's not mounted by default and treated as deprecated. OpenBSD out-and-out removed procfs, so I'm not sure what we're going to do there.

I've abandoned the config file to manage autoloading idea for now, and instead hardcoded the two plugin autoload directories: $libdir/avisynth (determined when the core is built, doesn't change) and $HOME/.avisynth. Users will need to create $HOME/.avisynth if they wish to put anything there, but that means that even if some plugin installs don't go to the same /avisynth subdirectory that the core puts ConvertStacked in, an .avsi file using AddAutoloadDir in $HOME/.avisynth can probably remedy them loading without needing it in every script. Provided AVSI autoloading functions.

Native Clang 8 on FreeBSD 12.1 and AppleClang on Catalina can now build AviSynth+, although there are a lot of warnings. High Sierra and Mojave, for those still using them, need to keep using GCC-9 due to the lack of <filesystem> support in the base libc++ from Apple. Even llvm-9 from Homebrew seemed to get hit by this, from what I remember, but I haven't tested it thoroughly.

SONAME and SOVERSION are now set, and yes, it looks weird. The base library bears the release version, so that multiple versions can be installed simultaneously, while the SOVERSION symlink points to AVISYNTH_INTERFACE_VERSION (the closest thing to the ABI compatibility indicators SOVERSIONs are meant to relay). So libavisynth.so.3.5.1, libavisynth.so.7 symlink, and the regular libavisynth.so symlink.

-fpermissive is no longer required.

[qyot27]

The AviSynth+ switchover patches have landed in upstream FFmpeg, removing the need for using the avsplus_linux2 branch on my dev repo.

[stax76]

Some tools seem to load avisynth from the path env var enabling portable setup, other tools don't.

works:

av2pipemod
x264


does not work:

mpc be
mpv
ffmpeg


Might there be security concerns or is it simply a missing feature?

[Groucho2004]

other tools don't.Can you elaborate? What's the error message?

[stax76]

mpv.net:

starts delayed without showing an error or doing anything


mpc be:

cannot render the file


ffmpeg:

D:\Samples\aaa_temp\aaa.avs: Unknown error occurred

[Groucho2004]

D:\Samples\aaa_temp\aaa.avsCan you post that script?

Edit - Also, how did you install Avisynth?

[stax76]

Only source filter, same result with ffms2 and l-smash.

[Groucho2004]

Only source filter, same result with ffms2 and l-smash.Explicit or implicit load? Again - how did you install Avisynth?

[stax76]

Explicit load, I've not installed but uninstalled avisynth. :)

I just modify the path env var.


If Not WasInitialized Then
Package.Python.AddToPath()
Package.AviSynth.AddToPath()
Package.VapourSynth.AddToPath()
WasInitialized = True
End If


That was all to make it work for staxrip, av2pipemod and x264, child processes inherit the path.

[qyot27]

http://git.videolan.org/?p=ffmpeg.git;a=blob;f=compat/w32dlfcn.h;h=bc9bb8c9f51e8a78627bbeabe61348cb42999729;hb=85553b42f92457a581e13edbd3e2c2e6136931eb

28 * Safe function used to open dynamic libs. This attempts to improve program security
29 * by removing the current directory from the dll search path. Only dll's found in the
30 * executable or system directory are allowed to be loaded.

Everything you listed as being affected uses libavformat. In such a case, say you've put AviSynth.dll in the same directory as mpv.com and mpv.exe and a simple Version() script named testver.avs,

In cmd.exe you can do:
mpv.com testver.avs
And it'll spit back the video window with the output of Version().

In bash or PowerShell, however, this will fail because those shells do not silently interpret the current directory as being on the PATH, and while you can execute mpv as a program (because Windows' filesystems and permissions systems don't map to the same execution bits that *nix does, it treats all files as executables), it won't load any other .dlls unless they're in a directory on the $PATH. But when you explicitly call the program's executable path:
./mpv.com testver.avs
It works.

[stax76]

Thanks for the clarification qyot27. I don't know what I should think about overwriting Windows default behavior, appears to be over-zealous. If somebody made a feature request to enable path search via switch, do you think there is a chance it gets accepted?

If I put the tools in the same directory I confirm that it's working.

[qyot27]

Thanks for the clarification qyot27. I don't know what I should think about overwriting Windows default behavior, appears to be over-zealous. If somebody made a feature request to enable path via switch, do you think there is a chance it gets accepted?
I highly doubt it.

However, there is a workaround. Just create symlinks in the ffmpeg/mpv/mpc directories using the Windows mklink command.

[stax76]

I've never used mklink even though it appears to be a useful command. Unfortunately I'm not able to create a hardlink in a cmd admin console.

C:\Users\frank\Desktop\bin>mklink /h .\Apps\Support\mpv.net\AviSynth.dll .\Apps\Support\AviSynth.dll
The system cannot find the file specified.

Same command without /h works.

[qyot27]

You don't want hard links, so /h not working isn't an issue.

[stax76]

I successfully created soft links for AviSynth.dll and DevIL.dll in the mpv.net folder but mpv.net still does not open avs.

I think portable vapoursynth has worked in mpv.net even without soft links.

Even if I can get it work it's probably not useful for staxrip users or are soft links portable?

I can offer a portable version putting everything in the same directory, it's just not clean.

[ChaosKing]

Even if I can get it work it's probably not useful for staxrip users or are soft links portable?



Soft links yes, hard links no. But you can't have relative soft links I think!? So not really portable then.

[stax76]

If I could get it work with soft links staxrip could recreate the soft links whenever it notices that it was started from a new location.

[qyot27]

I checked mpv.net portable x64 5.4.6.0 just now with AviSynth.dll in a subdirectory and a symlink to it next to mpvnet.com|exe. No issues opening a Version() script, or with one loading a local FFMS2.dll and invoking FFVideoSource.

[ChaosKing]

@qyot27 Does it affect the symlink if you copy the whole folder elsewhere?

[stax76]

And avisynth was not located in system32? Here it's not working. :(

I tested now with vapoursynth, renamed the installed core folder to core- so the installed version cannot work. mpv.net opens vpy using portable vapoursynth (checked in Process Explorer). Even ffmpeg opens vpy using portable vapoursynth, but not mpc be (who cares...).

[qyot27]

@qyot27 Does it affect the symlink if you copy the whole folder elsewhere?
Yeah, in that case Windows follows the symlink and does a full copy of the DLL.

[qyot27]

And avisynth was not located in system32? Here it's not working. :(

I tested now with vapoursynth, renamed the installed core folder to core- so the installed version cannot work. mpv.net opens vpy using portable vapoursynth (checked in Process Explorer). Even ffmpeg opens vpy using portable vapoursynth, but not mpc be (who cares...).
What's in system32 is a symlink to E:\Programs\AviSynth+\AviSynth64.dll, so all I have to do to disable it is rename E:\Programs\AviSynth+\AviSynth64.dll and the link can no longer find the DLL, resulting in the same error as if it wasn't installed.

[stax76]

Thanks for helping! I don't know what I could miss so have to give up for now.

[chros]

Soft links yes, hard links no. But you can't have relative soft links I think!? So not really portable then.

I use hardlinks in script on Win8/10 as a normal user, but I can't use softlinks as normal user.

[stax76]

It's working now after a reboot. :o

The German Wikipedia article on symlinks says that Windows soft links support relative paths, I hope that this is portable across machines.

In mpv.net I added a workaround.


bool wasAviSynthLoaded;

void LoadLibrary(string ext)
{
if (!wasAviSynthLoaded && ext == "avs")
{
WinAPI.LoadLibrary("AviSynth.dll");
wasAviSynthLoaded = true;
}
}


It's working, vapoursynth works without such workaround, reason unknown.


edit:

it's probably not portable:

https://superuser.com/questions/756039/can-mklink-make-portable-links

[Groucho2004]

bool wasAviSynthLoaded;

void LoadLibrary(string ext)
{
if (!wasAviSynthLoaded && ext == "avs")
{
WinAPI.LoadLibrary("AviSynth.dll");
wasAviSynthLoaded = true;
}
}

Did you leave out the error handling or is that the actual function?

Edit - The reason I'm asking: Is there some built-in error handling in .net when for example WinAPI.LoadLibrary("AviSynth.dll") fails?