Getting started with VSGAN for a VapourSynth newbie

[FranceBB]

Hi there,
I'm a long time Avisynth user (2006) and I've never ever used VapourSynth in my life, but you know, my boss wanted me to test the new GAN models and there just isn't such a thing for Avisynth, so I had to tell myself "the day has come" and install VapourSynth.
After so many years I'm not really planning in changing frameserver, but you know, I will at least test all the new VSGAN models, compare them against NNEDI3 in real life v210 lossless footage from tapes and then decide.

Anyway, I'm facing some issues.

OS: Windows 10 22H2 Enterprise x64
CPU: i7 5930K 6c/12th 3.50GHz
RAM: 32GB DDR4 (8x4)
GPU: NVIDIA GTX 980Ti 6GB GDDR5

Steps I've done, following the guide:

- Installed Python 3.11 x64
- Installed VapourSynth x64
- Put in the C:\Program Files\VapourSynth\plugins path the following plugins: akarin.dll, ffms2.dll, ffmsindex.exe, libhistogram.dll, LibP2P.dll, LSMASHSource.dll
- Installed Anaconda x64
- From Anaconda:

pip install VSGAN
pip install torch torchaudio
pip install torchvision
conda install pytorch -c pytorch

and yet:

https://i.imgur.com/T2Q5eWl.png


I'm definitely doing something stupid, I know, I'm sure, but I don't know what.

By the way, before you think I'm crazy, the computer above is what I'll be using for testing purposes. As far as the real encoding is concerned, it would, eventually, be done on a 20c/40th Intel Xeon with an NVIDIA Quadro P4000.


So... what am I doing wrong?

[lansing]

It seems like your VSGAN had not installed correctly, follow the instruction here:
https://vsgan.phoeniix.dev/en/stable/installation.html

And in your script, you need to load the different model, not the filter itself. You can find models from this database:
https://upscale.wiki/wiki/Model_Database


esrgan = ESRGAN(clip, "cuda")

model = r'2x_LD-Anime_Skr_v1.0.pth'
clip = esrgan.load(model).apply().clip

clip.set_output()

[FranceBB]

Well, I did, but it's not really finding it... :(

https://i.imgur.com/0HsrNXl.png
https://i.imgur.com/fZ27XU3.png

Same result...
From the sources I'm reading about on the web, it looks like things have to be installed for the current user.
So I thought "hold on a second, it might be that I'm trying to use two different users?"
I started VapourSynthEditor2 using the same Windows user I used to run Anaconda and install VSGAN, then, but nothing.
I'm lost. :(

[lansing]

Does any other filter work beside vsgan?

[poisondeathray]

Probably an anaconda issue ; perhaps a conflict with site-packages directory.

Notice you have 2 different locations: python310\site-packages and anaconda3\lib\site-packages. vsgan is in the 1st location.

For portable installations, there is a python<version>._pth file that you can edit in notepad to modify directory locations. It might work with anaconda, not sure

[ChaosKing]

Anaconda has some issues with VS (scroll down to last post)
https://github.com/vapoursynth/vapoursynth/issues/225

You could try the "diagnose" button in vsrepogui https://github.com/theChaosCoder/VSRepoGUI
It mainly checks for plugin issues, but also shows the installed python and vs versions.

[FranceBB]

Does any other filter work beside vsgan?

Yep, ffms2 as an indexer works:

https://i.imgur.com/Qhz0hIz.png

That's the output of "Check Script".
On the other hand, clicking on "Script, Preview" in VapourSynthEditor2, does nothing and if I click on the "Play" button on the right hand side, VapourSynthEditor2 crashes:

https://i.imgur.com/NmpB1I9.png


At the same time, though, VirtualDub plays the content from the .vpy script just fine (the one with just the indexing, that is):

https://i.imgur.com/ImSaLPY.png


You could try the "diagnose" button in vsrepogui https://github.com/theChaosCoder/VSRepoGUI
It mainly checks for plugin issues, but also shows the installed python and vs versions.

Well, I downloaded it and downloaded the dependencies, but... it won't load... :(

https://i.imgur.com/JwYsXVz.png

Probably an anaconda issue ; perhaps a conflict with site-packages directory.

I see...

Anaconda has some issues with VS (scroll down to last post)
https://github.com/vapoursynth/vapoursynth/issues/225

Damn... I didn't know Anaconda was so bad...
I'm not particularly attached to it, I just downloaded it the other day, so I'm totally open to changes.
Besides, the only reason why I used it is because it was mentioned in the VSGAN guide somewhere... :(

Anyway, so, it is an Anaconda issue and Anaconda is bad as it doesn't play nicely with VapourSynth, gotcha.

The next question is gonna be:
what should I use instead of Anaconda to install VSGAN properly and in a VapourSynth friendly way?

[ChaosKing]

Well, I downloaded it and downloaded the dependencies, but... it won't load... :(

https://i.imgur.com/JwYsXVz.png



Try running vsrepo update first in cmd (or vsrepo.py update)



The next question is gonna be:
what should I use instead of Anaconda to install VSGAN properly and in a VapourSynth friendly way?

Official python from their website should work.
You could also try the portable version. Simply download python embedded and extract the VS files into the same folder. Same with the VS Editor.

Make sure you use the python.exe from the portable folder. Then "install" pip and vsgan etc..

[Selur]

my boss wanted me to test the new GAN models and there just isn't such a thing for Avisynth,
http://avisynth.nl/index.php/Avs-mlrt should work if you convert the models with chaiNNer to .onnx

Also, if you want I can send you a link to my portable Vapoursynth folder which I use in Hybrid as 'torchAddon', it does include the VSGAN and much more.
I can also send you a link to my current dev version of Hybrid, which allows using VSGAN&Co more easily, main benefit to start is probably that you can look at the scripts Hybrid uses.
"NVIDIA GTX 980Ti 6GB GDDR5" <- my condolences, trying to use VSGAN and other ml based filters will be a real pain.

That said, if you just want to use VSGAN and create a portable version by yourself, try:

Created a new empy Vapoursynth-folder.
Download 'Windows embeddable package (64-bit)' from https://www.python.org/downloads/release/python-3108/.
Extract the Python download into the download into the 'Vapoursynth'-folder.
Downloaded 'VapourSynth64-Portable-R61.7z' from https://github.com/vapoursynth/vapoursynth/releases.
Extracted the Vapoursynth portable download into the 'Vapoursynth'-folder.
Download get-pip.py from https://bootstrap.pypa.io/get-pip.py and save it into the 'Vapoursynth'-folder.
Opened a 'Windows Command Prompt'-window and navigate into the 'Vaporusynth'-folder.
Install pip by calling :
pyton get-pip.py
Opened the python310._pth inside the Vapoursynth-folder in a text editor and add the following to lines above anything else in that file and saved the file

Scripts
Lib\site-packages

Install VSGAN, by calling:

python -m pip install vsgan
python -m pip install torch===1.13.0+cu117 torchvision==0.14.0 -f https://download.pytorch.org/whl/torch_stable.html

in the 'Windows Command Prompt'-window.

Now you got a portable Vapoursynth with VSGAN in it.

You could then use something like:
# Imports
import vapoursynth as vs
import os
import sys
# getting Vapoursynth core
core = vs.core

# source: 'G:\TestClips&Co\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/test.avi", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)

# adjusting color space from YUV420P8 to RGBS for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="full")

# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "F:/Hybrid/64bit/vsgan_models/2x_RealESRGAN_x2plus.pth"
vsgan.load(model)
vsgan.apply() # 1280x704
clip = vsgan.clip

# adjusting output color from: RGBS to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited", dither_type="error_diffusion")
# Output
clip.set_output()
to use VSGAN.

Cu Selur

[Zarxrax]

Damn... I didn't know Anaconda was so bad...
I'm not particularly attached to it, I just downloaded it the other day, so I'm totally open to changes.
Besides, the only reason why I used it is because it was mentioned in the VSGAN guide somewhere... :(

Anyway, so, it is an Anaconda issue and Anaconda is bad as it doesn't play nicely with VapourSynth, gotcha.

The next question is gonna be:
what should I use instead of Anaconda to install VSGAN properly and in a VapourSynth friendly way?

Anaconda is basically just python with a different name. My guess to your problem might be that you first installed python then vapoursynth, so vapoursynth was tied to your normal python installation. Then you installed anaconda and all the other stuff through anaconda, which is a TOTALLY SEPERATE python installation, so all this stuff wasn't able to talk to each other.

So I think you need to start over and either install JUST python or JUST anaconda.

[FranceBB]

So I think you need to start over and either install JUST python or JUST anaconda.

Gotcha.
I got rid of everything and I started over, trying to get it to work again.
I wanna say a big fat thank you to everyone who came here and spent some time to help me.
I don't wanna make you feel like you wasted time, because I actually appreciated the fact that you tried to give me some insights and help me, believe me, I did.
On the other hand, I've been using Avisynth since June 2006 and when I saw the .onnx models being supported by mlrt_ncnn() in Avisynth, I have been shying away from VapourSynth and went back to Avisynth and got it to work within minutes (https://i.imgur.com/JcEYt7R.png) (not every model, but still, some, but it might just be my GPU being too old for the most complex models).


Anyway, thank you for everything, guys, I appreciated everything, so don't feel bad if I'm just gonna go back to Avisynth, but I guess it's just a way of keeping my habits the same... :')

[Dogway]

Sorry to resurrect this but I got a Win11 laptop at work with a RTX 3050 Ti (4Gb VRAM though so I might need tiling support) and want to try all the AI plugins, specially HolyWu's.
Do you recommend me to install vapoursynth or use portable version? basically on ease of use terms. Also does latest R65 work on Python 3.12? Also plan to install Python system-wide.

[lansing]

For single image testing to play around you should use Chainner (https://github.com/chaiNNer-org/chaiNNer/tree/v0.20.2), a node based program for running these upscale models.

[Dogway]

I can do single image on AviSynth but in order to use the fancy ones on CUDA for video I require Vapoursynth (codeformer, basicvsrpp, colorization, inpainting, etc). I will use runway for generative probably.
I simply want to get the workflow done for presentation, and finally upgrade the system for real work.

[Selur]

@Dogway: send you a pm with my current dev version of Hybrid and all the addons for it, which includes a portable Vapoursynth (with pytorch&HolyWus plugins) which should get you started.
But be warned: 4GB VRAM really isn't much.

[Dogway]

@Selur: Thanks a lot, I installed it without issues. I guess I don't need to install python alongside since it will be retrieved from the included python311.zip file?
And to add more flexibility I can add vapoursynth editor (VapourSynthEditor2 looks abandoned). For the time being I won't install R65 nor Python 3.12 just in case it breaks something.
As long as I can show a single output frame I'm good. I only need it now to study the workflow and showcase for work/clients.

[Selur]

Yes, the Vapoursynth-folder directly comes with a portable Python version.
Copying the R65-RC1 into the Vapoursynth folder and overwriting the R64version works fine here. Haven't tried if switching out Python 3.12 works, but I suspect it would cause issues.

Cu Selur

[kedautinh12]

@Dogway, I hope you can ported vapoursynth plugins to avisynth plugins :D

[Dogway]

Ok, the nightmare starts. Where am I supposed to drop the VSEditor content? Using VapourSynth_Editor-r19-mod-6.3, I tried in Hybrid root folder, opens but warns with wrong library and plugin paths which neither worked. Then dropped the editor into Hybrid\64\Vapoursynth, can't even open the editor with a "entry point procedure couldn¡t be found on the library symbolic link" error dialog.

[Selur]

The error should also state 'Please set up the library search paths in settings.' ;)
So,...
a. copy/install VapourSynth Editor anywhere you want.
b. Start it and go to "Edit->Settings->Path->VapourSnyth library search paths", select and add your 'Hybrid/64bit/Vapoursynth', then hit 'Apply'
c. restart Vapoursynth Editor
Note that Hybrid by default uses separate folders for the plugins, thus you need to explicitly load dependencies.

Cu Selur

[Dogway]

This is my first time using an OS outside Win7 (+XP) so I'm bit puzzled about the permissions stuff on "Program Files" and being a PIA to install portables there, as I thought VSEdit had to be within the vapoursynth folder.

The PATHs were added but still getting the issues. Now VSEdit on a write permitted folder let me set PATH for library but won't open for plugins PATH.
Plugins PATH is a folder I made in Vapoursynth without subfolders:

vapoursynth_library_paths=C:/Program Files/Hybrid/64bit/Vapoursynth
vapoursynth_plugins_paths=C:/Program Files/Hybrid/64bit/pluginsVS

This is the plugins list (https://pastebin.com/nJkJ95Y9).

Do I also need to add the vsscripts folder to plugin path?

EDIT: I organized the folders a bit for the models, including them into a single folder called "models", still no dice.

GLSL
models
anime-segmentation
anime_style_art_rgb
cugan
dpir
models-DF2K
models-DF2K_JPEG
models-nose
models-pro
models-se
models-srmd
oidn
onnx_models
photo
ppocr
RealESRGANv2
rife
rife-anime
rife-HD
rife-UHD
rife-v2
rife-v2.3
rife-v2.4
rife-v3.0
rife-v3.1
rife-v4
rife-v4.1_ensembleFalse_fastTrue
rife-v4.1_ensembleTrue_fastFalse
rife-v4.2_ensembleFalse_fastTrue
rife-v4.2_ensembleTrue_fastFalse
rife-v4.3_ensembleFalse_fastTrue
rife-v4.3_ensembleTrue_fastFalse
rife-v4.4_ensembleFalse_fastTrue
rife-v4.4_ensembleTrue_fastFalse
rife-v4.5_ensembleFalse
rife-v4.5_ensembleTrue
rife-v4.6_ensembleFalse
rife-v4.6_ensembleTrue
rife-v4_ensembleFalse_fastTrue
rife-v4_ensembleTrue_fastFalse
rife_v2
sudo_rife4_ensembleFalse_fastTrue
sudo_rife4_ensembleTrue_fastFalse
sudo_rife4_ensembleTrue_fastTrue
waifu2x
TimeCube
vsgan_models
vsmlrt-cuda
vsort
vsov

[Selur]

I normally explicitly load everything manually.
That said, afaik. Vapoursynth\vapoursynth64\plugins is the place you put plugins to get autoloaded by a system-wide install.
Looking at http://www.vapoursynth.com/doc/installation.html#plugin-autoloading this might be wrong nowadays, or the documentation is wrong. ;)
For scripts, edit the python311._pth file. see: http://www.vapoursynth.com/doc/installation.html#plugins-and-scripts

[Dogway]

Thanks, I added ..\vsscripts to python311._pth.

Still having issues with VSEdit, I made an empty folder called plugins2, add only a few source loaders. With avisource.dll it opens, as soon I add LSMASHSource, DGDecode, or ffms2 it won't open.
Some plugins have issues like those ones, tried with addgrainc and no problem.

EDIT: Might be an API mismatch? I think the VSEdit I use (I think it was the most updated) isn't still (https://github.com/YomikoR/VapourSynth-Editor/issues/36) on API4.

[Selur]

Placing the plugins in the Vapoursynth\vapoursynth64\plugins-folder
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# source: 'G:\TestClips&Co\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/test.avi", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)
# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip, _Matrix=5)
# Setting color transfer info (470bg), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
# Setting color primaries info (), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
# adjusting output color from: YUV420P8 to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")
# set output frame rate to 25fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()
works fine here.
AviSource works fine too, since "Vapoursynth\vapoursynth64\coreplugins" contains AvsCompat.dll.

Cu Selur

[Dogway]

Now it worked, but I had to leave the plugins path empty in VSEdit. Guess it was already autoloading. I could call HINET, and also temporalsmooth.
As for the scripts in vsscripts, still not loading. At least in the autocomplete call. Should I copy them to the plugins folder too?

I find the editor very limited as it doesn't have tabs, guess I'd need to use some interleave to flip frames back and forth. Thanks for the help.

EDIT: No prob with scripts either, they seem to be autoloaded now after adding them to python311._pth

import havsfunc as haf
...
hin = haf.Deblock_QED(clip,50,50)
hin.set_output()

I'm set

[poisondeathray]

I find the editor very limited as it doesn't have tabs, guess I'd need to use some interleave to flip frames back and forth. Thanks for the help.


you can set numbered output nodes and swap with the number keys for vsedit mod , or there is a vsedit2 with tabs (also swap with number keys)

a.set_output(0)
b.set_output(1)
c.set_output(2)
.
.
.