VSRepo - the official easy plugin manager

[Myrsloik]

I have created a simple repository which can now basically install havsfunc and all its dependencies with one simple command. And a pile of other plugins and some scripts too now that I've gotten help creating more packages.

Latest version is now always bundled with VapourSynth. If you want to try out development versions or report bugs you can do it at Gihub (https://github.com/vapoursynth/vsrepo)


Usage:

vsrepo.py update
vsrepo.py install havsfunc ffms2 d2v
vsrepo.py upgrade-all
vsrepo.py installed
vsrepo.py available

You can refer to plugins/scripts in several ways. The id which is long and complicated and the namespace for plugins. For scripts it's generally the lowercase module name.

You can also add the -p switch to run it in portable mode and -f to force unknown versions of plugins to be upgraded.

Files will be installed to %APPDATA%\VapourSynth\PluginsXX and %APPDATA%\Python\X.Y\site-packages

(also known as the per user autoload directory and python's per user site directory)

All installed files will by default end up in the correct paths and instantly be usable. Make sure that the portable python directory is also the working directory when running in portable mode.

HELP CREATING MORE PLUGIN DEFINITIONS WELCOME. See the files in the local folder for examples. Report any issues.

[ChaosKing]

I see you changed the RainbowSmooth url from https://raw.githubusercontent.com/dubhater/vapoursynth-smoothuv/master/RainbowSmooth.py to https://github.com/dubhater/vapoursynth-smoothuv/archive/v2.zip. I tested it with the first one and it worked well. Any reason to prefer the bigger zip file?

[Myrsloik]

I see you changed the RainbowSmooth url from https://raw.githubusercontent.com/dubhater/vapoursynth-smoothuv/master/RainbowSmooth.py to https://github.com/dubhater/vapoursynth-smoothuv/archive/v2.zip. I tested it with the first one and it worked well. Any reason to prefer the bigger zip file?

Mostly that it points to a specific release/version. Pointing at master is a horrible idea that will make the hash fail as soon as a change is made to RainbowSmooth.py. If there's no release you have to at least pick a git revision and stick to that.

[ChaosKing]

Makes sense ... But I think it would be a good idea to offer both, a stable and master (latest/beta) version, since for scripts there aren't many official releases like in plugin repos. I think less then half of the scripts on github have official releases and even then, in 99% of the time, you just pick the master. At least this is my experience.

[Myrsloik]

Maybe I'll provide master/unstable labels at some point later. This is just the first version to get things started for the most common use cases. Even debugging a script with QTGMC caused greater buttpains for me just to go on the plugin/script treasure hunt for havsfunc.

[lansing]

Will it install both 32 bit and 64 bit of the plugin to each python directory respectively?

[Myrsloik]

Will it install both 32 bit and 64 bit of the plugin to each python directory respectively?

No, by default it installs the same bitness as the python you're running it with. You can control which one is installed with "-t win32" or "-t win64"

[lansing]

Shouldn't the Vapoursynth plugin path be inside "program files (x86)/vapoursynth"? Why is it pointing to %appdata% instead?

[Are_]

Just a wild guess
Files will be installed to %APPDATA%\VapourSynth\PluginsXX and %APPDATA%\Python\X.Y\site-packages

(also known as the per user autoload directory and python's per user site directory)

[foxyshadis]

Shouldn't the Vapoursynth plugin path be inside "program files (x86)/vapoursynth"? Why is it pointing to %appdata% instead?

In default use, it won't have permissions. I have mine in a custom folder with liberal permissions, but I'm pretty sure Myrsloik's intention is having a well-known folder with plugins and scripts for better support. Customize away if you support yourself!

[Myrsloik]

In default use, it won't have permissions. I have mine in a custom folder with liberal permissions, but I'm pretty sure Myrsloik's intention is having a well-known folder with plugins and scripts for better support. Customize away if you support yourself!

1. Permissions. Default global directories need administrative rights and that's a problem.
2. Why do you feel this need to stuff everything into program files where it shouldn't be?
3. It's a horrible idea to change state of plugins for all users in a true multi-user environment. Just saying.

[Myrsloik]

Updated version posted, now has 46 plugins and scripts in total. Feel free to request what should be added next or create the definitions yourself.

[Myrsloik]

Updated to test5 in the first post, it now has various small bugfixes and simple caching to make it work faster when many things are being installed at once.

About 10 more packages since yesterday too. Keep the submissions coming and we'll have a complete package system created in under one week!

(it's already quite complete so start using havsfunc and vsTAAmbk without having to go on a treasure hunt)

[Myrsloik]

The daily update. Now we're up to 67 plugins and scripts. I'm surprised there hasn't been more feedback yet. Is it too sunny outisde?

[lansing]

There can be a command to list all available plugins and their installation status, and there can also be a command to just install all plugins.

[Myrsloik]

There can be a command to list all available plugins and their installation status, and there can also be a command to just install all plugins.

Listing all available ones, sure. Installing all plugins is just ridiculous though.

[ChaosKing]

About 10 more packages since yesterday too. Keep the submissions coming and we'll have a complete package system created in under one week!


One Problem ist that not every git repo has also a release aka binaries, only source code...

I hope this motivates people to provide some binaries as well. :)

[Myrsloik]

One Problem ist that not every git repo has also a release aka binaries, only source code...

I hope this motivates people to provide some binaries as well. :)

Yes, it really annoys me when people can't be bothered to make releases and put binaries on github.

[lansing]

Installing all plugins is just ridiculous though.

Why is that? It gives an option to people who are new or people who don't really care about customization a hassle-free installation.

[Myrsloik]

Why is that? It gives an option to people who are new or people who don't really care about customization a hassle-free installation.

It's a waste of bandwidth and a recipe for potential problems. There's a reason "apt-get install-all" is not a thing in linux...

[ChaosKing]

Install havsfunc and you have downloaded 50% of all plugins :D

[Myrsloik]

There's no reason to install more or less obsolete plugins (nnedi3 replaced by znedi3, depan replaced by mvtools, and so on). Or cuda using plugins if you have intel/amd graphics.

[DJATOM]

Is znedi3 really safe? I mean on some sources results looks not that good as on legacy nnedi3, so I think it's fine to have both. Or that was fixed with the latest version?

[Myrsloik]

Is znedi3 really safe? I mean on some sources results looks not that good as on legacy nnedi3, so I think it's fine to have both. Or that was fixed with the latest version?

All bugs should be fixed, if not report one.

[Myrsloik]

Test7 linked in the first post. I've added the "available" command to list all known package names. This version can also identify a huge number of older releases since vsrupdate.py can now grab release information from the github api AND generate new entries.

The idea is that vsrepo.py will download the latest sources.json from someplace that's regularly updated. But that's coming in a later version. Keep contributing packages (if there's anything meaningful left not already there).

Does anyone actually care about uninstalling? I mean it'd be trivial to delete a dll or two but honestly, would anyone use it?

[DJATOM]

Pretty useless unless you need uninstalling feature for resolving conflicts or so.

[lansing]

I think uninstall still deserve a spot in there. For example if I'm to uninstall havsfunc, I would like to remove all its dependencies as well. I don't want to hunt them down one by one manually.

Playing around with the install command, the success message can be little more informative like stating how many dependencies have been install from the command and stating the location of the installation paths.

[Myrsloik]

I think uninstall still deserve a spot in there. For example if I'm to uninstall havsfunc, I would like to remove all its dependencies as well. I don't want to hunt them down one by one manually.

Playing around with the install command, the success message can be little more informative like stating how many dependencies have been install from the command and stating the location of the installation paths.

The problem with uninstalling all dependencies is that then I need to keep track how things happened, not just what they are now. One of the advantages of my backwards approach is that you can simply drop all your existing things into the per user autoload directory and they'll be managed.

Uninstall would only be for individual packages if added but it's easy to do so it'll be in the next test version. Maybe an info command to show the used paths would make sense too.

So the only change you want is showing the number of installed/updated packages?

[lansing]

The problem with uninstalling all dependencies is that then I need to keep track how things happened, not just what they are now. One of the advantages of my backwards approach is that you can simply drop all your existing things into the per user autoload directory and they'll be managed.

I'm thinking of some logic like when uninstalling a package, if a plugin was a dependency of another package, then don't uninstall it. There can also be a "force" option to uninstall it anyway. Then if anything had been removed accidentally, just run the "update" command and everything would be okay again.


So the only change you want is showing the number of installed/updated packages?
Yes, message like "1 package installed, 10 dependencies installed, 1 dependencies updated" at the end.

[lansing]

I tested installing a 32 bit znedi3 plugin and got this:


$ python vsrepo.py install znedi3 -t win32
Fetching: https://github.com/sekrit-twc/znedi3/releases/download/r1/znedi3_r1.7z
Successfully installed NNEDI3 Weights r1
No binaries available for win32 in package ZNEDI3, skipping installation
Package(s) installed

The nnedi3 weights bin shouldn't be installed since znedi3 doesn't have a 32 bit version.

[Myrsloik]

I tested installing a 32 bit znedi3 plugin and got this:


$ python vsrepo.py install znedi3 -t win32
Fetching: https://github.com/sekrit-twc/znedi3/releases/download/r1/znedi3_r1.7z
Successfully installed NNEDI3 Weights r1
No binaries available for win32 in package ZNEDI3, skipping installation
Package(s) installed

The nnedi3 weights bin shouldn't be installed since znedi3 doesn't have a 32 bit version.

Will be fixed in the next version

[Myrsloik]

Test8 posted. I guess I'll start experimenting with the final critical part next, an update command that fetches a recent package list.

Notable changes:
no longer installs dependencies for packages that themselves can't be installed because there are no relevant binaries
nicer summary of what was done at the end of an operation
list command renamed to available

[lansing]

There're still problem on the checking of dependencies compatibility:

$ python vsrepo.py -t win32 install oyster
Fetching: https://github.com/sekrit-twc/znedi3/releases/download/r1/znedi3_r1.7z
Successfully installed NNEDI3 Weights r1
Fetching: https://github.com/dubhater/vapoursynth-nnedi3/releases/download/v11/vapoursynth-nnedi3-v11-win32.7z
Successfully installed NNEDI3 v11
Fetching: https://github.com/Khanattila/KNLMeansCL/releases/download/v1.1.1/KNLMeansCL-v1.1.1.zip
Successfully installed KNLMeansCL v1.1.1
Fetching: ftp://ftp.fftw.org/pub/fftw/fftw-3.3.5-dll32.zip
Successfully installed FFTW3 Library 3.3.5
Fetching: https://github.com/HomeOfVapourSynthEvolution/VapourSynth-BM3D/releases/download/r7/BM3D-r7.7z
Successfully installed BM3D r7
Fetching: https://github.com/HomeOfVapourSynthEvolution/VapourSynth-DFTTest/releases/download/r4/DFTTest-r4.7z
Successfully installed DFTTest r4
Fetching: https://github.com/EleonoreMizo/fmtconv/releases/download/r20/fmtconv-r20.zip
Successfully installed fmtconv r20
No binaries available for win32 in package MVTools (float), skipping installation
7 missing dependencies installed


This package shouldn't be install either because one of its dependencies (mvtools float) doesn't have a 32 bit version. Also it doesn't catch znedi3's incompatibility.

[Myrsloik]

There're still problem on the checking of dependencies compatibility:

$ python vsrepo.py -t win32 install oyster
Fetching: https://github.com/sekrit-twc/znedi3/releases/download/r1/znedi3_r1.7z
Successfully installed NNEDI3 Weights r1
Fetching: https://github.com/dubhater/vapoursynth-nnedi3/releases/download/v11/vapoursynth-nnedi3-v11-win32.7z
Successfully installed NNEDI3 v11
Fetching: https://github.com/Khanattila/KNLMeansCL/releases/download/v1.1.1/KNLMeansCL-v1.1.1.zip
Successfully installed KNLMeansCL v1.1.1
Fetching: ftp://ftp.fftw.org/pub/fftw/fftw-3.3.5-dll32.zip
Successfully installed FFTW3 Library 3.3.5
Fetching: https://github.com/HomeOfVapourSynthEvolution/VapourSynth-BM3D/releases/download/r7/BM3D-r7.7z
Successfully installed BM3D r7
Fetching: https://github.com/HomeOfVapourSynthEvolution/VapourSynth-DFTTest/releases/download/r4/DFTTest-r4.7z
Successfully installed DFTTest r4
Fetching: https://github.com/EleonoreMizo/fmtconv/releases/download/r20/fmtconv-r20.zip
Successfully installed fmtconv r20
No binaries available for win32 in package MVTools (float), skipping installation
7 missing dependencies installed


This package shouldn't be install either because one of its dependencies (mvtools float) doesn't have a 32 bit version. Also it doesn't catch znedi3's incompatibility.

Actually it does work. I fetch nnedi3_weights.bin from the znedi3 release archive and nothing else so no problem there.

Oyster being installable is the result of dependencies not being treated as required when no binaries exist. Not really a big deal. Ff it bothers you a lot simply annoy whoever didn't realease 32bit binaries into doing so.

[lansing]

Actually it does work. I fetch nnedi3_weights.bin from the znedi3 release archive and nothing else so no problem there.

Oyster being installable is the result of dependencies not being treated as required when no binaries exist. Not really a big deal. Ff it bothers you a lot simply annoy whoever didn't realease 32bit binaries into doing so.

I don't use 32 bit anymore myself but I just feel like the error checkings should be more robust on different scenarios.

[Myrsloik]

I don't use 32 bit anymore myself but I just feel like the error checkings should be more robust on different scenarios.

I'll just leave that case for now as it'd require a huge number of changes and be of little use. I did fix another related edge case though when binaries exist but in an older release.

[Myrsloik]

Test9 posted. Now it's basically feature complete. I guess. Have fun.

Changes:
Added update command which fetches the latest package definitions
Added uninstall command
Mark upgradable packages in a clearer way
Install binaries from an older version if the most recent version doesn't have it

[lansing]

After I installed a bunch of packages for testing, I got an error about missing "libfftw3-3.dll" when opening vs editor. I think it has to do with the dfttest package. But I do have that file in my global plugin64 folder in program files.

[Myrsloik]

After I installed a bunch of packages for testing, I got an error about missing "libfftw3-3.dll" when opening vs editor. I think it has to do with the dfttest package. But I do have that file in my global plugin64 folder in program files.

Doh, I didn't add it to the fftw3 package since I thought everything uses the si gle precision version. Will fix.

[Myrsloik]

Test10 available. Even closer to done now.

Changes:
On upgrade the currently installed version is first uninstalled to avoid leftover files
upgrade all is now upgrade-all
Nice progress bars if tqdm (https://pypi.org/project/tqdm/) is installed
Use sha256 instead of sha1

Go test it some more, as usual there are a few more packages.

[ChaosKing]

Works great so far.
I think an info msg where plugins and scripts are being installed would be nice. (and to avoid questions like "where are my plugins" in the future)
Maybe even add an parameter for that? like vsrepo paths or vsrepo locations

It would be nice if I could specify a target folder in portable mode (-p). If possible for scripts and plugins separately. Currently scripts are downloaded in the root dir of vsrepo which is a bit annoying.

[Myrsloik]

Sure, I could add an override for the locations easily. Maybe listing the default paths would be good too.

I don't think there's anotjer guaranteed search path for portable python. Correct me if I'm wrong.

[ChaosKing]

You can specify search paths in python36._pth in portable python (default is only root folder & python3X.zip). But I actually want it also for my installed version since I use a custom scripts folder.

[Myrsloik]

Test11 posted

Changes:
Plugin and script install paths can be specified
There's now a paths command to show where things will be installed

[ChaosKing]

SangNom2 Plugin Ported for VapourSynth. Beat by James1201 on https://bitbucket.org/James1201/vapoursynth-sangnom/
I guess it should be changed to this git repo then. Currently this one is used https://github.com/HomeOfVapourSynthEvolution/VapourSynth-SangNomMod

[ChaosKing]

I just re-checked and noticed that "identifier": "com.mio.sangnom" is actually from SangNom from https://bitbucket.org/James1201/vapoursynth-sangnom and not from the github version (it has a different identifier!)
What would be the prefered way now? Replace/correct sangnom.json or create a new json file? what about the filename? You knew this day will come ;)

Here's the corrected file
{
"name": "SangNom",
"type": "Plugin",
"description": "SangNom is a single field deinterlacer using edge-directed interpolation but nowadays it's mainly used in anti-aliasing scripts.",
"website": "https://bitbucket.org/James1201/vapoursynth-sangnom",
"doom9": "https://forum.doom9.org/showthread.php?t=173752",
"category": "Deinterlacing",
"identifier": "com.mio.sangnom",
"namespace": "sangnom",
"releases": [
{
"version": "r40",
"win64": {
"url": "https://bitbucket.org/James1201/vapoursynth-sangnom/downloads/libsangnom%20r40%20win64.7z",
"files": [
"libsangnom r40 win64/gcc/libsangnom.dll"
],
"hash": {
"libsangnom.dll": "cd62d7e54f086e5752965072b07cc9e90b168517768472c76e754007763de9ae"
}
}
}
]
}

[Myrsloik]

Correct it with the most up to date version I guess.

[Myrsloik]

Test12 posted. Fixes some downloads failing if the total size isn't known.

Adds some more packages as usual.

[Myrsloik]

Don't forget to test the update functionality. I've updates the package list with all the recent additions.

We're also getting close to a final release. All I want to do is add the possibility to install files into subdirectories (a very stupid limitation of the current code).

[ChaosKing]

Just for fun... In avisynth world there is no identifier or namespace equivalent, correct? So if one would transform vsrepo to an avisynth package management (avsrepo), the only "unique" thing would be the filter name or filter filename!?