ffmpeg using .vpy scripts directly one day, like avisynth ?

[hydra3333]

Do you think there's any chance of ffmpeg one day building with something like --enable-vapoursynth so that it can consume .vpy scripts directly, sort of like avisynth ? Is there a feature in the works of any kind ?

It could mean that VS may need to be amenable to cross-compilation, so I figure the chances are remote. What do you reckon ?

Yes there's vspipe (video, no audio unless I'm mistaken ?) and HDR can be piped per this https://forum.doom9.org/showthread.php?p=1831137#post1831137

This person had a start at having a go with what appears to be limited success
'vapoursynth': { # not cross compiling ( yet ) and
'vapoursynth_libs': {
https://github.com/DeadSix27/python_cross_compile_script/blob/master/cross_compiler.py

[Selur]

sure it's possible in theory, but whether someone is motivated enough to implement it is a totally different thing :)
-> either implement it yourself and send a patch to the ffmpeg dev team or create a feature request through the official channels and hope for the best

The repository you link there doesn't seem to aim to add Vapoursynth support to ffmpeg.
It's compiles for example MPV on Linux for Windows.

[Myrsloik]

That must be a very old comment about cross compiling, I know people who've cross compiled it many times to compare gcc vs msvc for speed. Time to write the code I guess...

[hydra3333]

That must be a very old comment about cross compiling, I know people who've cross compiled it many times to compare gcc vs msvc for speed.
Oh. Would you be able to name someone here, or maybe post or a github link or something ?

Time to write the code I guess...
:thanks: :D

[jackoneill]

The core library and the included plugins (minus ocr) can be cross-compiled pretty easily. You pass --host=x86_64-w64-mingw32 to configure.

The Python module is the annoying part, because you must also cross-compile Python, and I think the Python developers don't make it easy.

But this is all irrelevant, because VapourSynth has a C API, so you can just use the official binaries compiled with MSVC. You don't have to cross-compile it for ffmpeg.

Correction: VapourSynth has a reasonable C API, unlike Python.

[Selur]

Also instead of cross compiling you could also compile it on Windows using media-autobuild_suite (https://github.com/jb-alvarado/media-autobuild_suite).
(Cross-Compiling won't allow ffmpeg to .vpy files so this might not be what you are looking for.)

[hydra3333]

Also instead of cross compiling you could also compile it on Windows using media-autobuild_suite (https://github.com/jb-alvarado/media-autobuild_suite).
(Cross-Compiling won't allow ffmpeg to .vpy files so this might not be what you are looking for.)
Thank you. Yes I've used jb's under Windows - and may still resort to that if cross-compilation of a static ffmpeg and its dependencies (3rd party libraries) is out of the question.

My objective was to cross-compile a static ffmpeg for windows whilst as far as possible keeping a minimalist Windows PC environment free from bits needed for compilation (eg stuff as needed for jb's approach and/or visual studio bits for vapoursynth). There are various linux cross-compilation approaches (eg zeranoe's which is a widely used build, rdp's, deadsix27's) eg using "free" ubuntu in a clean VM. The market appears to be out there.

The core library and the included plugins (minus ocr) can be cross-compiled pretty easily. You pass --host=x86_64-w64-mingw32 to configure.Ah, good oh. I guess one needs to do the usual things like setup the same folder structures as compiling it under windows, with sources from https://github.com/pinterf/AviSynthPlus/tree/MT and https://github.com/sekrit-twc/zimg/releases

The Python module is the annoying part, because you must also cross-compile Python, and I think the Python developers don't make it easy.
But this is all irrelevant, because VapourSynth has a C API, so you can just use the official binaries compiled with MSVC. You don't have to cross-compile it for ffmpeg.
Correction: VapourSynth has a reasonable C API, unlike Python.Oh. Time for some newbie googling, to see if anyone's cross-compiled python libraries in a way acceptable to an ffmpeg build process.

I suppose setting up for cross-compiling would need to take into account the "correct" way to do these these things, to result in a cross compiled Win build:

From the current windows build instructions

Needs Visual Studio 2017 and vsnasm (https://github.com/ShiftMediaProject/VSNASM)
It also needs both 32 and 64 bit Python 3.6 series with recent setuptools, cython, sphinx and sphinx-intl installed
Inno setup is needed to create the installer (default installation path assumed)
7-zip is needed to compress the portable version (default installation path assumed)
-
Clone VapourSynth
Clone zimg v2.7 branch into the VapourSynth dir (https://github.com/sekrit-twc/zimg)
Clone avs+ pinterf mt branch into the VapourSynth dir (https://github.com/pinterf/AviSynthPlus/tree/MT)
Compile 32 and 64 bit release using the VapourSynth solution
Run cython_build.bat to compile the Python modules
Run make_docs.bat to compile the documentation


I had to make "novice's home notes" while experimenting building vapoursynth under windows on a very old and exhausted "spare" PC ... so I suppose these would also need to be catered for as a part of cross compilation

A. Under Windows, I found that for other dependencies like
cython, sphinx and sphinx-intl this seemed to not crash:
1. install 64 bit python (3.6 in this case) as I only need 64bit stuff
2. start an Admin CMD dos box (or it can't copy files into protected folders) then use the pip command to install the packages, which apparently gets installed with python:
pip uninstall cython
pip --no-cache-dir install cython
pip uninstall sphinx-intl
pip --no-cache-dir install sphinx-intl
# uninstall and then install, to get the latest versions

B. Under Windows I also got this error building avscompat
Quote:
LINK : fatal error LNK1181: cannot open input file 'gdi32.lib'
which is apparently a known error per http://stackoverflow.com/questions/3...file-gdi32-lib If anyone has this problem on Visual Studio 2017, it is an issue with the Windows 10 Creators Update.
A workaround is to select the "Windows 10 SDK (10.0.15063.0) for Desktop C++ x86 and x64" under the "Desktop development with C++" workload in the Visual Studio Installer.

C. Other source to clone before building, into EXACTLY the right folders
1. For vapoursynth to build, the avs+ mt branch placed in the root of vapoursynth folder tree must be called "AviSynthPlus". Source: https://github.com/pinterf/AviSynthPlus/tree/MT
2. and the zimg v2.7.1 branch placed in the root of vapoursynth folder tree must be called "zimg". Source: https://github.com/sekrit-twc/zimg/releases

Looking at that, I'm agreeing with you, jb's Windows-based build method looks like it may be the only way for a newbie to get something up and going once the VS coding etc is done. Unless someone has already done it and is willing to share the process in detail for others to learn :)

[jackoneill]

VapourSynth's Avisynth(+) compatibility module must be compiled with MSVC, because it uses the Avisynth(+) C++ API, and Avisynth(+) itself is compiled with MSVC. If you manage to compile the Avisynth(+) compatibility module with clang and its -fms-extensions parameter, that may also work, but I don't think anyone tested it.

[Myrsloik]

VapourSynth's Avisynth(+) compatibility module must be compiled with MSVC, because it uses the Avisynth(+) C++ API, and Avisynth(+) itself is compiled with MSVC. If you manage to compile the Avisynth(+) compatibility module with clang and its -fms-extensions parameter, that may also work, but I don't think anyone tested it.

Clang can compile that too now after avs+ actually made the headers follow the c++ standard... I think it works but didn't test it that much.

[DJATOM]

I spent some time on making x264 support vpy scripts internally. Patch here (https://pastebin.com/raBWpFY8).
The ideas and some code taken from ffmpeg demuxer proposed here, avs.c input from x264 and NVEnc's vpy input module.
Not sure if that's faster than piping, but looks like complex scripts uses all CPU cores.
Code will definitely compile well with icl 2018 or gcc 7.3.0. I didn't tried another compilers.

[hydra3333]

I spent some time on making x264 support vpy scripts internally. Patch here (https://pastebin.com/raBWpFY8).
The ideas and some code taken from ffmpeg demuxer proposed here, avs.c input from x264 and NVEnc's vpy input module.
Not sure if that's faster than piping, but looks like complex scripts uses all CPU cores.
Code will definitely compile well with icl 2018 or gcc 7.3.0. I didn't tried another compilers.

Thank you !


Separately, here's a link explaining why building a static ffmpeg with vapoursynth can't be done.
https://forum.doom9.org/showthread.php?p=1840531#post1840531

[TheFluff]

http://lists.ffmpeg.org/pipermail/ffmpeg-devel/2018-April/229125.html

[hydra3333]

OK, if that could be done and for example that meant that if the ffmpeg exe was run in the same folder as "portable VS" and it worked then that could well be close enough for some purposes ( eg mine :) ).

Do you know of the current ffmpeg change proposal (per the link above) achieves that including delayed loading of python or are there more changes needed ?

Also, I wonder if the same portable VS folder structure is needed for plugins DLLs etc which I guess can't be statically linked in either, I suppose it is.

[JEEB]

It loads the script into memory, and then uses libvapoursynth's vsscript_evaluateScript() to evaluate it. Whatever libvapoursynth does in the background regarding python etc. is what matters, not really that patch.

[hydra3333]

Your own, good oh.
If I may, do you have a link to your procedure which I could look and learn from (and pinch) ?

[jackoneill]

For unknown reason I have to pass -lpython3.6m -lstdc++ as extra libs when linking with static vsscript, otherwise undefined references occur.

--enable-vapoursynth --extra-cflags="-DVS_CORE_EXPORTS" --extra-ldflags="-static" --extra-libs="-lpython3.6m -lstdc++"


You have to pass -lstdc++ because I assume ffmpeg has no C++ code, or at least the library that uses VSScript doesn't. Thus gcc is used for linking, not g++. gcc doesn't link stdc++ because it thinks it's all C code, but VSScript uses C++ internally.

And you need to pass -lpython3.6m most likely because ffmpeg's configure doesn't call pkg-config with the --static parameter. If it did, VSScript's pkg-config file would give it -lpython3.6m already.

[poisondeathray]

Will this one day be implemented in common distributed builds , such as zeranoe's ? Or the autobuild scripts ? Or will this be something you have to compile yourself ?

[JEEB]

And you need to pass -lpython3.6m most likely because ffmpeg's configure doesn't call pkg-config with the --static parameter. If it did, VSScript's pkg-config file would give it -lpython3.6m already.
--pkg-config-flags=FLAGS pass additional flags to pkgconf []

In other words, adding "--static" to this will get you that.

[hydra3333]

Here's the ffmpeg commit
https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/7074a7ccd9a4d4e445252780fd182aa0b3778b79

avformat: add vapoursynth wrapper

This can "demux" .vpy files. Autodetection of .vpy scripts is
intentionally not done, because it would be a major security issue. You
need to force the format, for example with "-f vapoursynth" for the
FFmpeg CLI tools.

Some minor code copied from other LGPL parts of FFmpeg.

I did not find a good way to test a few of the more obscure VS features,
like VFR nodes, compat pixel formats, or nodes with dynamic size/format
changes. These can be easily implemented on demand.

--enable-vapoursynth in the ffmpeg configure

[hydra3333]

It nearly works, as reported in https://github.com/DeadSix27/python_cross_compile_script/issues/61
It seems that something may be amiss per comment https://github.com/DeadSix27/python_cross_compile_script/issues/61#issuecomment-386771355

[ca18]

By the way the patch on FFmpeg mailing list seems buggy, it gives an error when trying to open vpy file. But Stephen's patch works fine for me.
Hi HolyWu, what is "Stephen's patch" and where can we find it? Thanks!

[hydra3333]

Hi HolyWu, what is "Stephen's patch" and where can we find it? Thanks!
bump ?

[DJATOM]

I still have it: https://pastebin.com/Ls1LmTyj

[ca18]

Thank you for digging that up so it can be compared now to the "official" nonworking patch. :eek:

[MysteryX]

Wait a second. VapourSynth directly executes any Python script (without any restrictions?), so it's like opening Word documents without any macro restrictions?

That means that if you have vpy files associated with a program and you click on a vpy script someone sends you via email, it could execute anything directly on your computer?

I don't like this design decision... Are there at least restrictions on what the script can do?

[DJATOM]

It can't do anything admin-related until you allow it (start with admin privileges).

[Myrsloik]

Wait a second. VapourSynth directly executes any Python script (without any restrictions?), so it's like opening Word documents without any macro restrictions?

That means that if you have vpy files associated with a program and you click on a vpy script someone sends you via email, it could execute anything directly on your computer?

I don't like this design decision... Are there at least restrictions on what the script can do?

I didn't see you complaining when you loaded dll files from strangers you found on the interwebs. Same thing. If you don't check or sandbox things it's your own fault.

[MysteryX]

There are indeed risks with downloading DLLs from unknown sources. However, one needs to do special work to run code in a DLL, it won't happen automatically, and most viruses within DLLs will be auto-detected by a good anti-virus.

Running a Python script by double-clicking on a file (for users who may not even know what VapourSynth is), however, can seriously limit the adaptation of VapourSynth. If that's the reason FFMPEG hasn't added native support for it, then I fully understand. In a business or production environment, have to be *VERY* careful where the scripts are coming from and where they are running. I don't think many people realize that. I also don't think Kaspersky will scan the Python raw script for malicious code.

I would say that this, combined with the lack of audio support, are the 2 things most limiting the adaptation of VapourSynth.

[hydra3333]

OK, any recommendations or hints (other than yes you should do this) on sandboxing under Win10x64 ?

My preference is to always use portable version of things, if that helps with advice.
If you don't check or sandbox things it's your own fault

Audio support may seem to be a bit problematic if video/audio delays are involved - at one point I had to process them separately each into a separate .mp4 container to preserve a delay, although it may have changed since then and/or I was doing something wrong ... tinkered until a process worked.

[MysteryX]

My preference is to always use portable version of things, if that helps with advice.
Running a portable version won't change anything in regards to security. The script can still execute or do anything that doesn't require admin privilege.

Audio support may seem to be a bit problematic if video/audio delays are involved - at one point I had to process them separately each into a separate .mp4 container to preserve a delay, although it may have changed since then and/or I was doing something wrong ... tinkered until a process worked.
That's for holding audio (associating it with the video track). AFAIK there's nothing in regards to audio processing.

[shekh]

...If that's the reason FFMPEG hasn't added native support for it, then I fully understand.

How is this FFMpeg problem? It does not create file associations.

[ChaosKing]

Wait a second. VapourSynth directly executes any Python script (without any restrictions?), so it's like opening Word documents without any macro restrictions?

That means that if you have vpy files associated with a program and you click on a vpy script someone sends you via email, it could execute anything directly on your computer?

I don't like this design decision... Are there at least restrictions on what the script can do?

The same applies to py files (blame python installer). That about bat or exe files? They have the same "power of potential destruction", I don't see why this is suddenly a problem now. Or do you click on every exe file send by email to you?

Edit:
If your vpy is associated with an editor, why would this be a bad thing? It is only executed only if you run it.

[hydra3333]

afaik vpy is not associated in a portable vapoursynth install ? would (portable) ffmpeg not create any associations ?
An untested assumption is to pop ffmpeg into the same folder as portable vapoursynth and then run ffmpeg with the correct commandline to open a vpy that the user specifies ?
( Assuming that one has a static vapoursynth enabled build of ffmpeg which works :( )

[ChaosKing]

I have the install version of VS and there is no auto association of vpy files.

[MysteryX]

FFMPEG doesn't create any file association. However, in a business environment, the scripts may be running in one environment while production material is being fed from employees on the network. If, for example, the server is designed to automatically batch-process all scripts in a folder, and someone manages to push a script in there, he can run anything he wants on the server. Perhaps the admins don't even know about VapourSynth and just try to read every video file in that folder thinking they are harmless.

There are lots of scenarios where we don't care about security -- but in cases where it's important, we're at a bad start.

That about bat or exe files? They have the same "power of potential destruction", I don't see why this is suddenly a problem now.

The danger of EXE and BAT files is very well known, and many steps have been taken to limit their damage. Browsers will give you strong warnings, EXE/BAT files are flagged as coming from the Internet and will ask you for confirmation to run them, and anti-viruses will scan them. The danger of Word files was also very well-known, prompting Microsoft to disable macros by default, and give strong warnings before executing any macro.

I'll also note that this isn't an issue in Avisynth.

In .NET, at least, there are sandboxing options where the code can run with limited privilege, which allows for Silverlight that could run within a browser (but then browsers dropped support for Silverlight anyway).

Indeed VapourSynth doesn't create any file association by default -- but I associated them with a program to preview the files, not knowing the risks that go with it. Of course it's not much of a personal issue on my laptop where I'm in control of everything, but in a business network, things aren't so simple.

Just to put things into perspective, FFMPEG is being used by all kinds of corporations. Many of them have batch-processing servers (and aren't even using VapourSynth). Adding native support for VapourSynth in FFMPEG would put all of these batch-processing servers and businesses at risk, as anyone who can push a file into the batch-processing could run anything on the server. The only way around that would be for server admin to explicitly forbid VapourSynth extensions -- essentially disabling FFMPEG's VapourSynth support. It certainly cannot be turned on by default.

[amichaelt]

Just to put things into perspective, FFMPEG is being used by all kinds of corporations. Many of them have batch-processing servers (and aren't even using VapourSynth). Adding native support for VapourSynth in FFMPEG would put all of these batch-processing servers and businesses at risk, as anyone who can push a file into the batch-processing could run anything on the server. The only way around that would be for server admin to explicitly forbid VapourSynth extensions -- essentially disabling FFMPEG's VapourSynth support. It certainly cannot be turned on by default.

Interesting FUD and all, but as linked to above, native Vapoursynth script support was committed to the ffmpeg git repository back in April and it's still there today.

[MysteryX]

Isn't this whole conversation because it isn't enabled by default?

[amichaelt]

Isn't this whole conversation because it isn't enabled by default?

Even if it was, it not being enabled by default has nothing to do with your FUD about Vapoursynth being a potential a malware vector. Plenty of ffmpeg features are optional and disabled by default. For example, Avisynth support is also disabled by default.

[amichaelt]

Also a malicious Avisynth plugin can be written to delete everything off your hard druve, download malware, etc. What you're talking about is what any piece of malicious arbitrary-code can do. Avisynth also has no sandboxing to prevent this. So you're really no more safe with Avisynth.

And if you use autoload, a malware-ridden plugin could have its init function called and do malicious things without you even needing to ever explicitly call it.

[MysteryX]

What about this? I haven't made anything up, but I understand their decision.
Here's the ffmpeg commit
https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/7074a7ccd9a4d4e445252780fd182aa0b3778b79

avformat: add vapoursynth wrapper

This can "demux" .vpy files. Autodetection of .vpy scripts is
intentionally not done, because it would be a major security issue. You
need to force the format, for example with "-f vapoursynth" for the
FFmpeg CLI tools.

Some minor code copied from other LGPL parts of FFmpeg.

I did not find a good way to test a few of the more obscure VS features,
like VFR nodes, compat pixel formats, or nodes with dynamic size/format
changes. These can be easily implemented on demand.


--enable-vapoursynth in the ffmpeg configure

[foxyshadis]

There are indeed risks with downloading DLLs from unknown sources. However, one needs to do special work to run code in a DLL, it won't happen automatically, and most viruses within DLLs will be auto-detected by a good anti-virus.

Running a Python script by double-clicking on a file (for users who may not even know what VapourSynth is), however, can seriously limit the adaptation of VapourSynth. If that's the reason FFMPEG hasn't added native support for it, then I fully understand. In a business or production environment, have to be *VERY* careful where the scripts are coming from and where they are running. I don't think many people realize that. I also don't think Kaspersky will scan the Python raw script for malicious code.

I would say that this, combined with the lack of audio support, are the 2 things most limiting the adaptation of VapourSynth.

In Avisynth, all you have to do is get someone to load your DLL by saying it does something cool, and it turns out that pretty much everyone would. You can't even inspect that without being an expert programmer/reverser.

[ChaosKing]

And If you are really evil you could try to overwrite files with http://avisynth.nl/index.php/ImageWriter

[MysteryX]

I see a clear difference between having security risks on a development machine for someone using Avisynth/VapourSynth and downloading various DLLs, compared to a production server using FFMPEG without any awareness of what VapourSynth/Avisynth even are.

If you take risks on your local machine, that's one thing.

If a server environment is at risk simply by using FFMPEG because it auto-loads a risky script engine (without admin awareness), that's something else.

[poisondeathray]

Using Wolfberry's FFMpeg build with vpy support , direct vapoursynth input is about 50% speed compared to vspipe to the same ffmpeg . I would have expected marginally faster for direct vpy demuxer. Script is just QTGMC . I tried specifying different ffmpeg -threads values as an input argument. Any ideas, or am I doing something wrong? Could it be a cache issue ?

https://forum.doom9.org/showthread.php?p=1866931#post1866931


vspipe --y4m input.vpy - | "ffmpeg" -f yuv4mpegpipe -i - -an -f null NUL




"ffmpeg" -f vapoursynth -i input.vpy -an -f null NUL



EDIT #1: but another test of just reading a MP4 video with lsmash (no other filters, just source filter), direct vapoursynth is about 1.4-1.5x faster than vspipe to ffmpeg

EDIT #2: tested other source filters too lsmash, ffms2, other file types (mpeg2, avc); pretty consistent observation - direct read with only source filter is faster, but as soon as you add any additional filter in script (not just QTGMC, it can be anything like a denoiser, or even a simple resize only) it becomes slower than vspipe method at about 50% speed

Is it specific to this ffmpeg build, or do other people get different results with their own build ?

[qyot27]

Post #26 in this thread linked to a copy of it on pastebin.

I think I'll probably have to switch to it also, since I could never seem to get the official implementation to cross-compile right, despite following the steps provided earlier to get the Python lib encapsulated correctly.

[poisondeathray]

Thanks Wolfberry ; does this recent one have wm4 or Stephen patch ?

[Wolfberry]

I'm sure this build uses Stephen's implementation, but I haven't done any tests.

[poisondeathray]

I'm sure this build uses Stephen's implementation, but I haven't done any tests.

Some quick tests - in terms of speed wise it seems to be working as expected

Why isn't Stephen implementation the official one ?

[Pat357]

Post #26 in this thread linked to a copy of it on pastebin.

I think I'll probably have to switch to it also, since I could never seem to get the official implementation to cross-compile right, despite following the steps provided earlier to get the Python lib encapsulated correctly.

Unfortunately I can't help you with this specific issue, but I have a request :
could you consider to make a new FFMS2 compile with the libaom decoder replaced by dav1d ?
On relative simple scripts, the just decoding from AV1 takes a lot of CPU with libaom, while dav1d is at least 4x faster on modern CPU's.

With the following videos, I can't get even real-time speed when using liboam :

https://www.elecard.com/storage/video/Stream3_AV1_4K_13.9mbps.webm

https://www.elecard.com/storage/video/Stream2_AV1_4K_22.7mbps.webm

[poisondeathray]

https://www.elecard.com/storage/video/Stream3_AV1_4K_13.9mbps.webm[/url]

https://www.elecard.com/storage/video/Stream2_AV1_4K_22.7mbps.webm


Wolfberry posted ffms2 builds with libdav1d in the other thread

https://forum.doom9.org/showthread.php?t=176198


FFMS2:

ffmpeg 4.2-dev N-93683-g163bb087f8
libdav1d 0.2.2-0d7aa95d
libopenjpeg 2.3.1
mbedTLS 2.16.1