Autoloading scripts on Linux

[zorr]

Now that the optimizer (https://forum.doom9.org/showthread.php?t=175723) is going to have Vapoursynth support I want to make sure that it also works on Linux. I have 64bit Ubuntu (Cosmic Cuttlefish) running on a virtual machine so that I can test the optimizer.

I have trouble installing additional Vapoursynth scripts (like muvsfunc) so that they will autoload. VSPipe says "ModuleNotFoundError: No module named muvsfunc" when I try to use it in a script.

Here's what I have so far:

Installed Vapoursynth from this PPA (https://launchpad.net/~djcj/+archive/ubuntu/vapoursynth). I installed packages vapoursynth and vapoursynth-extra-plugins.

I copied muvsfunc.py into /usr/share/vsscripts which contains the core scripts like havsfunc.py. I assumed it would be found from there.

I have read about the PYTHONPATH, my environment didn't have it so I tried export PYTHONPATH=/usr/share/vsscripts but that didn't help.

I have tried the method described in the official docs (http://www.vapoursynth.com/doc/autoloading.html), I added $HOME/.config/vapoursynth/vapoursynth.conf and pointed UserPluginDir to a folder where I added the scripts. Didn't help. XDG_CONFIG_HOME is not defined.

I'm out of ideas right now, does anyone have a clue as to what might be wrong? Maybe I need to change the permissions of the file?

[Are_]

Vapoursynth modules should be installed in your python site-packages directory.

One place (system-wide) can be found with this command:
python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
Another place (user-specific) with this other one:
python -m site --user-site

[Mystery Keeper]

A snippet that I use in most of my scripts:

import sys
sys.path.append('D:\\vapoursynth-plugins\\py\\')
import platform
architecture = platform.architecture()
if architecture[0] == '64bit':
vapoursynth_plugins_path = 'D:\\vapoursynth-plugins\\64bit\\'
else:
vapoursynth_plugins_path = 'D:\\vapoursynth-plugins\\32bit\\'
print('Plugins folder: ', vapoursynth_plugins_path, end='\n', file=sys.stderr)
import os
for filename in os.listdir(vapoursynth_plugins_path):
if filename[-4:] != '.dll':
continue
try:
core.std.LoadPlugin(vapoursynth_plugins_path + filename)
except Exception as e:
print('Error: ', e, end='\n', file=sys.stderr)

modules_to_reload = ['havsfunc', 'MCDenoise', 'helpers']
for module in modules_to_reload:
if module in sys.modules:
del sys.modules[module]

import havsfunc as haf
import MCDenoise as mcd
import helpers as hlp

[_Al_]

there is possibility to import modul from anywhere if you dynamically create some scripts, maybe not your case, anyway:
import importlib
import os
path = /home/user/Documents/my.py
modul_name = os.path.basename(path)
spec = importlib.util.spec_from_loader(modul_name, path)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
or even from just a string object (even if you just created that within script itself):
spec = importlib.util.spec_from_loader('my_name', loader=None)
my_name = importlib.util.module_from_spec(spec)
exec(my_string_object, my_name.__dict__)

[zorr]

Vapoursynth modules should be installed in your python site-packages directory.

One place (system-wide) can be found with this command:
python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

This one didn't quite work, I got ModuleNotFoundError: No module named 'distutils.sysconfig'. And I had to use python3 as the command, python wasn't recognized.

Another place (user-specific) with this other one:
python -m site --user-site

This one worked, result was /home/osboxes/.local/lib/python3.6/site-packages. After I created that path and moved the scripts there all was OK. Weird that it didn't find havsfunc either until I copied it there from /usr/share/vsscripts.

And thanks Mystery Keeper for that snippet, could be useful at some point.

there is possibility to import modul from anywhere if you dynamically create some scripts, maybe not your case, anyway:

Actually I am creating scripts dynamically, or rather making small modifications to a "template" script. Adding some code near the beginning of the script is something that I already do.

Thanks for the help all of you, my test turned out to be a success, it works! :thanks:

I would like to try VapourSynth-VMAF (https://github.com/HomeOfVapourSynthEvolution/VapourSynth-VMAF) next but looks like I'd need to compile it myself since there are no Linux binaries... right? Vapoursynth seems to be installed at /usr/lib/x86_64-linux-gnu/vapoursynth, should the VMAF .so file go there as well?

[cwk]

This one worked, result was /home/osboxes/.local/lib/python3.6/site-packages. After I created that path and moved the scripts there all was OK. Weird that it didn't find havsfunc either until I copied it there from /usr/share/vsscripts.


You may wish to symlink to the scripts instead of moving them, as the vapoursynth-extra-plugins PPA will update the scripts in /usr/share/scripts, but not in the other location.

[Are_]

This one didn't quite work, I got ModuleNotFoundError: No module named 'distutils.sysconfig'. And I had to use python3 as the command, python wasn't recognized.

My bad, I didn't know distutils was not part of Ubuntu's python package.


I would like to try VapourSynth-VMAF (https://github.com/HomeOfVapourSynthEvolution/VapourSynth-VMAF) next but looks like I'd need to compile it myself since there are no Linux binaries... right? Vapoursynth seems to be installed at /usr/lib/x86_64-linux-gnu/vapoursynth, should the VMAF .so file go there as well?
Yes, but just the vapoursynth plugin.

You may wish to symlink to the scripts instead of moving them, as the vapoursynth-extra-plugins PPA will update the scripts in /usr/share/scripts, but not in the other location.
How come the installed vapoursynth modules are not loadable? Isn't that too confusing for users?

[cwk]

How come the installed vapoursynth modules are not loadable? Isn't that too confusing for users?

The modules or the scripts? I have had no trouble with autoloading the modules, which are located at:

/usr/lib/x86_64-linux-gnu/vapoursynth/

It's the scripts where I needed to symlink.

As for the location of the scripts, that's a good question for the package maintainer. The place to raise an issue is here:

https://github.com/darealshinji/vapoursynth-plugins

[zorr]

My bad, I didn't know distutils was not part of Ubuntu's python package.

I installed the distutils and got the system-wide directory:
/usr/lib/python3/dist-packages

You may wish to symlink to the scripts instead of moving them, as the vapoursynth-extra-plugins PPA will update the scripts in /usr/share/scripts, but not in the other location.

That's a good idea. I noticed there is already a symlink to /usr/share/vsscripts at the system-wide library path, I wonder why it isn't working.

[Are_]

Is there inside "/usr/share/vsscripts" an empty file named "__init__.py"?

For security reason python does not load modules inside sub-directories.

If that's the case you may want to fill a bug report against the maintainer.

[zorr]

Is there inside "/usr/share/vsscripts" an empty file named "__init__.py"?

No such a file there. I tried adding it but that didn't make it load the scripts from there.