VSRepo - the official easy plugin manager

[Myrsloik]

Just for fun... In avisynth world there is no identifier or namespace equivalent, correct? So if one would transform vsrepo to an avisynth package management (avsrepo), the only "unique" thing would be the filter name or filter filename!?

Exactly, there's no unique identifier or namespace there. That's why I added it to avoid most of the mess. FILENAME IS NOT UNIQUE.

Btw, if your transformations are minor consider not forking the codebase. I mean all you have to do is basically change how the plugin and script paths are read from the registry...

[ChaosKing]

Btw, if your transformations are minor consider not forking the codebase. I mean all you have to do is basically change how the plugin and script paths are read from the registry...

I just tested it with masktools2 and it works very well. Also with custom paths -b -s no changes are necessary to the vsrepo codebase, like you said.

Theoretically Avisynth could be also updated via vsrepo easily.

[Myrsloik]

I just tested it with masktools2 and it works very well. Also with custom paths -b -s no changes are necessary to the vsrepo codebase, like you said.

Theoretically Avisynth could be also updated via vsrepo easily.

Sure, in theory it could work. However the other nice key part vsrupdate generally can't. Most Avisynth developers haven't heard of github or even version control. It'll be a lot messier and in the process you'll most likely end up having to host a pile of stuff yourself.

Anyway, not my problem anymore....

[Myrsloik]

RC1 released. See the first post.

Changes:
Files can now be installed into subdirectories, waifu2x added as a result of this
vsrupdate.py now does a lot more verification to ensure no typos are in the package files

Getting close to done. Update and test as usual. Maybe we can reach 100 packages before the final release... is there still stuff left to add anyway?

The only remaining enhancements are to fix a few corner cases and convenience things in vsrupdate so for all other purposes consider this version final.

[ChaosKing]

The models are "very old" in releases. These are up to date https://github.com/HomeOfVapourSynthEvolution/VapourSynth-Waifu2x-w2xc/tree/master/Waifu2x-w2xc/models

[Myrsloik]

The models are "very old" in releases. These are up to date https://github.com/HomeOfVapourSynthEvolution/VapourSynth-Waifu2x-w2xc/tree/master/Waifu2x-w2xc/models

Someone will have to package them and release. I don't care enough about that detail. Actually someone who stores so many floating point numbers as json text deserves "old" models.

[ChaosKing]

The vspackages.zip file on vapoursynth.com is broken

[Myrsloik]

The vspackages.zip file on vapoursynth.com is broken

Fixed now

[~ VEGETA ~]

can you list the available packages in a txt file so we can choose?

Also, make something like "install common" to get most used packages like mvtools, masktools, lsmash, ffms2, fmt,..etc all in one line.

[Myrsloik]

can you list the available packages in a txt file so we can choose?

Also, make something like "install common" to get most used packages like mvtools, masktools, lsmash, ffms2, fmt,..etc all in one line.

Read the help and use the "available" command to list them all. Common means different things but at this point simply installing havsfunc is pretty much that.

Or if you have another base package suggestion simply post the command to install it all at once.

[~ VEGETA ~]

i try to install a certain package like "mv" but it doesn't do a thing... cli doesn't change after my command "python vsrepo install mv" for so long. what is the problem?

I have python 3.6 and x64-bit and I am using your latest github script.

[Myrsloik]

i try to install a certain package like "mv" but it doesn't do a thing... cli doesn't change after my command "python vsrepo install mv" for so long. what is the problem?

I have python 3.6 and x64-bit and I am using your latest github script.

You probably need to male it vsrepo.PY for ot to work

[~ VEGETA ~]

Actually I put it .py but didn't write it properly here

[~ VEGETA ~]

Now works good but I couldn't find lsmas here. My python is 64 bit thus plugins32 in VS directory probably won't be used? so I need lsmas for x64?

[ChaosKing]

Now works good but I couldn't find lsmas here. My python is 64 bit thus plugins32 in VS directory probably won't be used? so I need lsmas for x64?

Yes, everything (except scripts) needs to be 64bit to work together.
lsmas is not available in vsrepo yet.

[Myrsloik]

Yes, everything (except scripts) needs to be 64bit to work together.
lsmas is not available in vsrepo yet.

Speaking of that. I wish someone would adopt lsmas and the orphaned scripts without proper github repos. I really can't go around adding random dropbox links to stuff.

This is especially easy to do for scripts and would only require minimal effort in case someone submits a pull request.

hint hint wink wink

[ChaosKing]

Are there many orphaned scripts? I could only find 1-2 ...

//Edit
But I guess every script would need its own git repo!? A collection wouldn't work with your git update logic, correct?

[Myrsloik]

Are there many orphaned scripts? I could only find 1-2 ...

Maybe only 1-2. That's still 1-2 too many.

[Myrsloik]

Are there many orphaned scripts? I could only find 1-2 ...

//Edit
But I guess every script would need its own git repo!? A collection wouldn't work with your git update logic, correct?

Yes, they'd need a separate repo. You could just lump all the orphaned scripts into one repo too. Basically it needs to show up on the github releases tab or it won't be picked up automatically. (I also think pastebin-like things are bad places to store things that should be permanent)

[lansing]

I got a "HTTP Error 404: Not Found" trying to install hnwvsfunc

[Wolfberry]

You should run the update script first (if it works). Since I only keep the latest release, it will fail if you try to get the older release.
Sorry for the inconvenience.

[Myrsloik]

I got a "HTTP Error 404: Not Found" trying to install hnwvsfunc

It's because wolfberry deletes old releases just to annoy people. Didn't have time to properly fix vsrupdate to handle it yet.

[Myrsloik]

I've updated the packages and fixed vsupdate so it doesn't barf when releases are removed.

Now only people who completely rename files between releases need to die:)

[lansing]

Does the update file have logic to update itself once vapoursynth got updated to use newer python version like 3.7?

[Myrsloik]

Does the update file have logic to update itself once vapoursynth got updated to use newer python version like 3.7?

The python version doesn't matter. The only quirk is that you need to install all scripts in the python site packages dir again or move them between the major python updates. No logic needed there.

[Yrosma]

I was trying to setup portable edition of vapoursynth and run into a missing dependecy after installing havsfunc using vsrepo.

In my test script I'm using Deblock_QED from havsfunc and got the error it was missing the namespace dctf.

I found it here:
https://github.com/HomeOfVapourSynthEvolution/VapourSynth-DCTFilter/releases

After adding this one it was working again.
Is there a reason why this one was left out and shouldn't I be using this one?
Or was this one just forgotten?

[Myrsloik]

I was trying to setup portable edition of vapoursynth and run into a missing dependecy after installing havsfunc using vsrepo.

In my test script I'm using Deblock_QED from havsfunc and got the error it was missing the namespace dctf.

I found it here:
https://github.com/HomeOfVapourSynthEvolution/VapourSynth-DCTFilter/releases

After adding this one it was working again.
Is there a reason why this one was left out and shouldn't I be using this one?
Or was this one just forgotten?

I think nobody got around to creating a package for it yet. Somebody *hint* *hint* will probably do it soon though since you pointed it out. I actually thought havsfunc had all its dependencies done already.

[Myrsloik]

And it's added.

[Myrsloik]

I've now updated the packages again. It took a little longer than usual since I had to add a published date to every release as well (in order to always properly preserve the release order when releases are deleted).

[Myrsloik]

I've released RC2. It performs slightly more input validation before starting operations and has a new package definitions.

The unsharpmask and xsharpen functions are also added as WarpSharpSupport (wss) for those of you who want to install it conveniently.

[hydra3333]

I must be doing something wrong,
C:\SOFTWARE\Vapoursynth-x64>.\python.exe .\vsrepo.py -p -t win64 update -b vapoursynth64\plugins\dll-to-choose-from -s vapoursynth64\scripts
Local definitions already up to date: Sun, 09 Sep 2018 22:02:52 GMT

C:\SOFTWARE\Vapoursynth-x64>.\python.exe .\vsrepo.py -p -t win64 installed -b vapoursynth64\plugins\dll-to-choose-from -s vapoursynth64\scripts
Name Namespace Installed Latest Identifier
+Bifrost bifrost Unknown v2.2 com.nodame.bifrost
Bilateral bilateral r3 r3 com.invalid.identifier.bilateral
BM3D bm3d r7 r7 com.vapoursynth.bm3d
+Cnr2 cnr2 Unknown v1 com.nodame.cnr2
CTMF ctfm r4 r4 com.holywu.ctmf
+D2V Source d2v Unknown v1.1 com.sources.d2vsource
DCTFilter dctf r2 r2 com.holywu.dctfilter
Deblock deblock r6 r6 com.holywu.deblock
DeLogo delogo v0.4 v0.4 in.7086.delogo
DFTTest dfttest r4 r4 com.holywu.dfttest
EEDI2 eedi2 r7 r7 com.holywu.eedi2
+FFmpegSource2 ffms2 Unknown 2.23 com.vapoursynth.ffms2
+FFTW3 Library fftw3_library Unknown 3.3.5 org.fftw.fftw3
+fmtconv fmtc Unknown r20 fmtconv
*AddGrain grain r5 r6 com.holywu.addgrain
*KNLMeansCL knlm v1.1.0 v1.1.1 com.Khanattila.KNLMeansCL
MVTools mv v20 v20 com.nodame.mvtools
MVTools (float) mvsf r9 r9 com.nodame.mvsf
+NNEDI3 nnedi3 Unknown v11 com.deinterlace.nnedi3
NNEDI3CL nnedi3cl r7 r7 com.holywu.nnedi3cl
NNEDI3 Weights nnedi3_weights r1 r1 com.deinterlace.nnedi3.weights
Retinex retinex r3 r3 com.invalid.identifier.retinex
TCanny tcanny r10 r10 com.holywu.tcanny
TDeintMod tdm r10 r10 com.holywu.tdeintmod
Tonemap tonemap R2 R2 com.ifb.tonemap
+vcfreq vcfreq Unknown 2016-05-12 in.vcmohan.freq
VagueDenoiser vd r2 r2 com.holywu.vaguedenoiser
VSFilter vsf r2 r2 com.holywu.vsfilter
+Waifu2x w2xc Unknown r7 com.holywu.waifu2x-w2xc
W3FDIF w3fdif r1 r1 com.holywu.w3fdif
+AWarpSharp2 warp Unknown v4 com.nodame.awarpsharp2
Yadifmod yadifmod r10 r10 com.holywu.yadifmod
ZNEDI3 znedi3 r1 r1 xxx.abc.znedi3

C:\SOFTWARE\Vapoursynth-x64>.\python.exe .\vsrepo.py -p -t win64 upgrade-all -f -b vapoursynth64\plugins\dll-to-choose-from -s vapoursynth64\scripts
Fetching: https://github.com/dubhater/vapoursynth-bifrost/releases/download/v2.2/vapoursynth-bifrost-v2.2-win64.7z
Traceback (most recent call last):
File ".\vsrepo.py", line 455, in <module>
inst = upgrade_all_packages(args.force)
File ".\vsrepo.py", line 368, in upgrade_all_packages
res = upgrade_files(get_package_from_id(id, True))
File ".\vsrepo.py", line 346, in upgrade_files
install_files(p)
File ".\vsrepo.py", line 310, in install_files
result = subprocess.run([cmd7zip_path, "e", "-so", tfpath, fn_props[0]], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "subprocess.py", line 403, in run
File "subprocess.py", line 709, in __init__
File "subprocess.py", line 997, in _execute_child
FileNotFoundError: [WinError 2] The system cannot find the file specified
When I extract portable 7zip to the folder C:\SOFTWARE\Vapoursynth-x64 I see a 7za.exe rather than a 7z.exe

Suggestions welcome.

[Myrsloik]

Sigh, just rename it to 7z.exe and see if it works. I didn't even consider 7zip having a portable version too...

[hydra3333]

thank you, that works ...

[hydra3333]

tried it today and got this message ...
.\python.exe .\vsrepo.py -p -t win64 -f -b vapoursynth64\plugins\dll-to-choose-from -s vapoursynth64\scripts upgrade-all
Fetching: http://rationalqm.us/hdr/DGHDRtoSDR_1.9.rar
Traceback (most recent call last):
File ".\vsrepo.py", line 455, in <module>
inst = upgrade_all_packages(args.force)
File ".\vsrepo.py", line 368, in upgrade_all_packages
res = upgrade_files(get_package_from_id(id, True))
File ".\vsrepo.py", line 346, in upgrade_files
install_files(p)
File ".\vsrepo.py", line 311, in install_files
result.check_returncode()
File "subprocess.py", line 369, in check_returncode
subprocess.CalledProcessError: Command '['7z.exe', 'e', '-so', 'C:\\Users\\u\\AppData\\Local\\Temp\\vsmez18y1v_', 'x64/DGHDRtoSDR.dll']' returned non-zero exit status 2.

Tried a 7z manually
7z.exe l DGHDRtoSDR_1.9.rar
7-Zip (a) 18.05 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2018-04-30
Scanning the drive for archives:
1 file, 145886 bytes (143 KiB)

Listing archive: DGHDRtoSDR_1.9.rar

ERROR: DGHDRtoSDR_1.9.rar : Can not open the file as archive

The .rar opens OK with winzip.

Advice welcomed.

[ChaosKing]

Maybe you have an older 7z version? I tested this package and had no problems extracting it.

//edit:
hmm ok, 18.05 is the newest one. But why does it work for me...
I can also open and extract rar files with the 7z gui -> 7zFM.exe

//edit2
on the 7zip page rar is listed as supported
Try to reinstall 7zip I guess...

[hydra3333]

OK, I'll re-download try again. And the 32bit version too I suppose.
Out of interest, did you try the commandline version ?
7za.exe l DGHDRtoSDR_1.9.rar

edit: freshly downloaded the x32 and x64 standalone versions from https://www.7-zip.org/download.html (I never install it) and ...
E:\SOFTWARE-Win10\ZIP\7zip\x>7za.exe l DGHDRtoSDR_1.9.rar

7-Zip (a) 18.05 (x86) : Copyright (c) 1999-2018 Igor Pavlov : 2018-04-30

Scanning the drive for archives:
1 file, 145886 bytes (143 KiB)

Listing archive: DGHDRtoSDR_1.9.rar

ERROR: DGHDRtoSDR_1.9.rar : Can not open the file as archive

Errors: 1

E:\SOFTWARE-Win10\ZIP\7zip\x>x64\7za.exe l DGHDRtoSDR_1.9.rar

7-Zip (a) 18.05 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2018-04-30

Scanning the drive for archives:
1 file, 145886 bytes (143 KiB)

Listing archive: DGHDRtoSDR_1.9.rar

ERROR: DGHDRtoSDR_1.9.rar : Can not open the file as archive

Errors: 1

Winzip definitely opens it OK. How strange.

[ChaosKing]

It seems that there is a difference between 7z.exe and 7za.exe. Listing also works with 7z.exe.
Use this https://www.7-zip.org/a/7z1805-x64.exe


https://superuser.com/questions/1105516/comparing-7z-exe-and-7za-exe

[hydra3333]

Ah, thanks for that and the superuser link.

edit: DG is considering .zip or something instead of .rar

[Myrsloik]

I fixed the two biggest issues but as usual there may be a typo or something so help me test it. Just grab vsrepo.py from git because I'm lazy.

The first change is that it no longer uninstalls a previous version of a package before the new version is downloaded AND has the hashes verified. So much harder to get your plugins and scripts into a broken state by accident.

The second change is that packages where the download fails are skipped and reported as failed at the end, that way scripts with huge dependency lists won't end up broken as often. Note that multiple failures to fetch the same file may happen if multiple other packages depend on it.

Btw, I think DG secretly updates things after release to fix minor bugs => hash mismatches

I don't think there's a good way to handle that and it still barfs an exception. Discuss this one and think of a better solution if you can.

[ChaosKing]

In my initial idea for a database / plugin manager I had planned to create a bot which checks gist, github, bitbucket and http links periodically for new releases.
Maybe a http last-modified header check could be implemented, so one can keep up with "secret" updates, deletes or new releases ... I don't think there is an easy solution for this but only if the author would follow certain steps.

I can add a cronjob in my vsdb databse to check for non github links and report it if something changes like a hash mismatch or delete.

//Edit:
Auto generating an updated json could be also implemented. => just check if ok and push to github, less work


OR

This was discussed in the github issue: A myplugin.json could be maintained by the author itself and vsrepo just grabs it.

[Myrsloik]

In my initial idea for a database / plugin manager I had planned to create a bot which checks gist, github, bitbucket and http links periodically for new releases.
Maybe a http last-modified header check could be implemented, so one can keep up with "secret" updates, deletes or new releases ... I don't think there is an easy solution for this but only if the author would follow certain steps.

I can add a cronjob in my vsdb databse to check for non github links and report it if something changes like a hash mismatch or delete.

//Edit:
Auto generating an updated json could be also implemented. => just check if ok and push to github, less work

I guess your idea to regularly scan for secretly changed packages is required. Don't feel like implementing it due to how ugly it is though. I'll think about it...

Automating the current thing is trivial. The problem is if files are added or removed in a package because then it needs manual help so now and then it'd push completely broken updates. Minimal human review once per week is probably a must.

[gonca]

Btw, I think DG secretly updates things after release to fix minor bugs => hash mismatches
Actually, I believe he slip streams minor fixes, and appends the slip stream number to the version number, but the number on the download from his web site does not change.
No secret, all very public and announced to the users by e-mail and in the binaries update thread

[ChaosKing]

I guess your idea to regularly scan for secretly changed packages is required. Don't feel like implementing it due to how ugly it is though. I'll think about it...

Automating the current thing is trivial. The problem is if files are added or removed in a package because then it needs manual help so now and then it'd push completely broken updates. Minimal human review once per week is probably a must.

I don't think there will be many plugins that add or remove files. In 99% it stays the same. I thing this would be the easiest way:

Lets say we have a http://asd.xd/superplugin.zip url and a superplugin.json
- implement a scraper / bot function that checks if a) all urls of superplugin are still valid and b) if one of the "versions" have a different "last-modified" date. (maybe the whole zip could be hashed?)
- if a zip/rar has changed, download it, re-hash all known files that are stored in superplugin.json AND list also all contents of the zip/rar AND print a warning if a file is missing. Maybe it is a good idea to store the previous files list somewhere for comparison so that any new files could also be detected.
=> result: human review is reduced to a minimum.

@gonca yes, but it is only obvious if one checks the forum. VSRepo can't do that :)

[Myrsloik]

I don't think there will be many plugins that add or remove files. In 99% it stays the same. I thing this would be the easiest way:

Lets say we have a http://asd.xd/superplugin.zip url and a superplugin.json
- implement a scraper / bot function that checks if a) all urls of superplugin are still valid and b) if one of the "versions" have a different "last-modified" date. (maybe the whole zip could be hashed?)
- if a zip/rar has changed, download it, re-hash all known files that are stored in superplugin.json AND list also all contents of the zip/rar AND print a warning if a file is missing. Maybe it is a good idea to store the previous files list somewhere for comparison so that any new files could also be detected.
=> result: human review is reduced to a minimum.

@gonca yes, but it is only obvious if one checks the forum. VSRepo can't do that :)

I'd say it's simply beyond the scope. Let's just go shit on people who don't use github. It's so much faster and easier. Shaming works.

[ChaosKing]

.\vsrepo.py -p install adjust
adjust already exists. Output text is "Nothing done". I think it showed something like "Already installed" before...

[Myrsloik]

.\vsrepo.py -p install adjust
adjust already exists. Output text is "Nothing done". I think it showed something like "Already installed" before...

I'm going to improve that text later.

[Myrsloik]

Here's an issue someone with far too much time can help with:
https://github.com/vapoursynth/vsrepo/issues/80

Too lazy to do it myself and it's not that important.

[ChaosKing]

Can't you just check for invalid (empty) releases with vsrupdate and then just add it to the ignore list? I mean isn't vsrupdate checking for all releases anyway, empty ones included?

[Myrsloik]

Can't you just check for invalid (empty) releases with vsrupdate and then just add it to the ignore list? I mean isn't vsrupdate checking for all releases anyway, empty ones included?

No. Empty entries can either mean that the release has no relevant binaries (for example see HolyWu's linux build system updates that do nothing for windows users) OR that the filename guessing algorithm failed to create a new entry. It needs to be manually checked to determine which one it is.

This issue about editing existing entries and doing it manually is probably faster than writing a script. Or about the same speed since it's a one time thing.

[ChaosKing]

There are only 2 plugins with empty release tags left.
https://github.com/IFeelBloated/RGSF/releases <- see my commit
https://github.com/HomeOfVapourSynthEvolution/VapourSynth-Waifu2x-w2xc/releases <- older models have some different filenames/file count. No one will install the older releses anyway... not worth the work.