Getting started with VSGAN for a VapourSynth newbie

[FranceBB]

Hi there,
I'm a long time Avisynth user (2006) and I've never ever used VapourSynth in my life, but you know, my boss wanted me to test the new GAN models and there just isn't such a thing for Avisynth, so I had to tell myself "the day has come" and install VapourSynth.
After so many years I'm not really planning in changing frameserver, but you know, I will at least test all the new VSGAN models, compare them against NNEDI3 in real life v210 lossless footage from tapes and then decide.

Anyway, I'm facing some issues.

OS: Windows 10 22H2 Enterprise x64
CPU: i7 5930K 6c/12th 3.50GHz
RAM: 32GB DDR4 (8x4)
GPU: NVIDIA GTX 980Ti 6GB GDDR5

Steps I've done, following the guide:

- Installed Python 3.11 x64
- Installed VapourSynth x64
- Put in the C:\Program Files\VapourSynth\plugins path the following plugins: akarin.dll, ffms2.dll, ffmsindex.exe, libhistogram.dll, LibP2P.dll, LSMASHSource.dll
- Installed Anaconda x64
- From Anaconda:

pip install VSGAN
pip install torch torchaudio
pip install torchvision
conda install pytorch -c pytorch

and yet:




I'm definitely doing something stupid, I know, I'm sure, but I don't know what.

By the way, before you think I'm crazy, the computer above is what I'll be using for testing purposes. As far as the real encoding is concerned, it would, eventually, be done on a 20c/40th Intel Xeon with an NVIDIA Quadro P4000.


So... what am I doing wrong?

[lansing]

It seems like your VSGAN had not installed correctly, follow the instruction here:
https://vsgan.phoeniix.dev/en/stable/installation.html

And in your script, you need to load the different model, not the filter itself. You can find models from this database:
https://upscale.wiki/wiki/Model_Database

esrgan = ESRGAN(clip, "cuda")

model = r'2x_LD-Anime_Skr_v1.0.pth'
clip = esrgan.load(model).apply().clip

clip.set_output()

[FranceBB]

Well, I did, but it's not really finding it...




Same result...
From the sources I'm reading about on the web, it looks like things have to be installed for the current user.
So I thought "hold on a second, it might be that I'm trying to use two different users?"
I started VapourSynthEditor2 using the same Windows user I used to run Anaconda and install VSGAN, then, but nothing.
I'm lost.

[lansing]

Does any other filter work beside vsgan?

[poisondeathray]

Probably an anaconda issue ; perhaps a conflict with site-packages directory.

Notice you have 2 different locations: python310\site-packages and anaconda3\lib\site-packages. vsgan is in the 1st location.

For portable installations, there is a python<version>._pth file that you can edit in notepad to modify directory locations. It might work with anaconda, not sure

[ChaosKing]

Anaconda has some issues with VS (scroll down to last post)
https://github.com/vapoursynth/vapoursynth/issues/225

You could try the "diagnose" button in vsrepogui https://github.com/theChaosCoder/VSRepoGUI
It mainly checks for plugin issues, but also shows the installed python and vs versions.

[FranceBB]

Originally Posted by lansing

Does any other filter work beside vsgan?

Yep, ffms2 as an indexer works:



That's the output of "Check Script".
On the other hand, clicking on "Script, Preview" in VapourSynthEditor2, does nothing and if I click on the "Play" button on the right hand side, VapourSynthEditor2 crashes:




At the same time, though, VirtualDub plays the content from the .vpy script just fine (the one with just the indexing, that is):

Originally Posted by ChaosKing

You could try the "diagnose" button in vsrepogui https://github.com/theChaosCoder/VSRepoGUI
It mainly checks for plugin issues, but also shows the installed python and vs versions.

Well, I downloaded it and downloaded the dependencies, but... it won't load...

Originally Posted by poisondeathray

Probably an anaconda issue ; perhaps a conflict with site-packages directory.

I see...

Originally Posted by ChaosKing

Anaconda has some issues with VS (scroll down to last post)
https://github.com/vapoursynth/vapoursynth/issues/225

Damn... I didn't know Anaconda was so bad...
I'm not particularly attached to it, I just downloaded it the other day, so I'm totally open to changes.
Besides, the only reason why I used it is because it was mentioned in the VSGAN guide somewhere...

Anyway, so, it is an Anaconda issue and Anaconda is bad as it doesn't play nicely with VapourSynth, gotcha.

The next question is gonna be:
what should I use instead of Anaconda to install VSGAN properly and in a VapourSynth friendly way?

[ChaosKing]

Originally Posted by FranceBB

Well, I downloaded it and downloaded the dependencies, but... it won't load...

Try running vsrepo update first in cmd (or vsrepo.py update)

Originally Posted by FranceBB

The next question is gonna be:
what should I use instead of Anaconda to install VSGAN properly and in a VapourSynth friendly way?

Official python from their website should work.
You could also try the portable version. Simply download python embedded and extract the VS files into the same folder. Same with the VS Editor.

Make sure you use the python.exe from the portable folder. Then "install" pip and vsgan etc..

[Selur]

my boss wanted me to test the new GAN models and there just isn't such a thing for Avisynth,

http://avisynth.nl/index.php/Avs-mlrt should work if you convert the models with chaiNNer to .onnx

Also, if you want I can send you a link to my portable Vapoursynth folder which I use in Hybrid as 'torchAddon', it does include the VSGAN and much more.
I can also send you a link to my current dev version of Hybrid, which allows using VSGAN&Co more easily, main benefit to start is probably that you can look at the scripts Hybrid uses.
"NVIDIA GTX 980Ti 6GB GDDR5" <- my condolences, trying to use VSGAN and other ml based filters will be a real pain.

That said, if you just want to use VSGAN and create a portable version by yourself, try:

• Created a new empy Vapoursynth-folder.
• Download 'Windows embeddable package (64-bit)' from https://www.python.org/downloads/release/python-3108/.
• Extract the Python download into the download into the 'Vapoursynth'-folder.
• Downloaded 'VapourSynth64-Portable-R61.7z' from https://github.com/vapoursynth/vapoursynth/releases.
• Extracted the Vapoursynth portable download into the 'Vapoursynth'-folder.
• Download get-pip.py from https://bootstrap.pypa.io/get-pip.py and save it into the 'Vapoursynth'-folder.
• Opened a 'Windows Command Prompt'-window and navigate into the 'Vaporusynth'-folder.
• Install pip by calling :
  

  pyton get-pip.py

• Opened the python310._pth inside the Vapoursynth-folder in a text editor and add the following to lines above anything else in that file and saved the file
  

  Scripts
  Lib\site-packages

• Install VSGAN, by calling:
  • python -m pip install vsgan
  • python -m pip install torch===1.13.0+cu117 torchvision==0.14.0 -f https://download.pytorch.org/whl/torch_stable.html
  in the 'Windows Command Prompt'-window.

Now you got a portable Vapoursynth with VSGAN in it.

You could then use something like:

# Imports
import vapoursynth as vs
import os
import sys
# getting Vapoursynth core
core = vs.core

# source: 'G:\TestClips&Co\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/test.avi", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)

# adjusting color space from YUV420P8 to RGBS for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="full")

# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "F:/Hybrid/64bit/vsgan_models/2x_RealESRGAN_x2plus.pth"
vsgan.load(model)
vsgan.apply() # 1280x704
clip = vsgan.clip

# adjusting output color from: RGBS to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited", dither_type="error_diffusion")
# Output
clip.set_output()

to use VSGAN.

Cu Selur

[Zarxrax]

Originally Posted by FranceBB

Damn... I didn't know Anaconda was so bad...
I'm not particularly attached to it, I just downloaded it the other day, so I'm totally open to changes.
Besides, the only reason why I used it is because it was mentioned in the VSGAN guide somewhere...

Anyway, so, it is an Anaconda issue and Anaconda is bad as it doesn't play nicely with VapourSynth, gotcha.

The next question is gonna be:
what should I use instead of Anaconda to install VSGAN properly and in a VapourSynth friendly way?

Anaconda is basically just python with a different name. My guess to your problem might be that you first installed python then vapoursynth, so vapoursynth was tied to your normal python installation. Then you installed anaconda and all the other stuff through anaconda, which is a TOTALLY SEPERATE python installation, so all this stuff wasn't able to talk to each other.

So I think you need to start over and either install JUST python or JUST anaconda.

[FranceBB]

Originally Posted by Zarxrax

So I think you need to start over and either install JUST python or JUST anaconda.

Gotcha.
I got rid of everything and I started over, trying to get it to work again.
I wanna say a big fat thank you to everyone who came here and spent some time to help me.
I don't wanna make you feel like you wasted time, because I actually appreciated the fact that you tried to give me some insights and help me, believe me, I did.
On the other hand, I've been using Avisynth since June 2006 and when I saw the .onnx models being supported by mlrt_ncnn() in Avisynth, I have been shying away from VapourSynth and went back to Avisynth and got it to work within minutes (not every model, but still, some, but it might just be my GPU being too old for the most complex models).


Anyway, thank you for everything, guys, I appreciated everything, so don't feel bad if I'm just gonna go back to Avisynth, but I guess it's just a way of keeping my habits the same... :')

[Dogway]

Sorry to resurrect this but I got a Win11 laptop at work with a RTX 3050 Ti (4Gb VRAM though so I might need tiling support) and want to try all the AI plugins, specially HolyWu's.
Do you recommend me to install vapoursynth or use portable version? basically on ease of use terms. Also does latest R65 work on Python 3.12? Also plan to install Python system-wide.

[lansing]

For single image testing to play around you should use Chainner, a node based program for running these upscale models.

[Dogway]

I can do single image on AviSynth but in order to use the fancy ones on CUDA for video I require Vapoursynth (codeformer, basicvsrpp, colorization, inpainting, etc). I will use runway for generative probably.
I simply want to get the workflow done for presentation, and finally upgrade the system for real work.

[Selur]

@Dogway: send you a pm with my current dev version of Hybrid and all the addons for it, which includes a portable Vapoursynth (with pytorch&HolyWus plugins) which should get you started.
But be warned: 4GB VRAM really isn't much.

[Dogway]

@Selur: Thanks a lot, I installed it without issues. I guess I don't need to install python alongside since it will be retrieved from the included python311.zip file?
And to add more flexibility I can add vapoursynth editor (VapourSynthEditor2 looks abandoned). For the time being I won't install R65 nor Python 3.12 just in case it breaks something.
As long as I can show a single output frame I'm good. I only need it now to study the workflow and showcase for work/clients.

[Selur]

Yes, the Vapoursynth-folder directly comes with a portable Python version.
Copying the R65-RC1 into the Vapoursynth folder and overwriting the R64version works fine here. Haven't tried if switching out Python 3.12 works, but I suspect it would cause issues.

Cu Selur

[kedautinh12]

@Dogway, I hope you can ported vapoursynth plugins to avisynth plugins

[Dogway]

Ok, the nightmare starts. Where am I supposed to drop the VSEditor content? Using VapourSynth_Editor-r19-mod-6.3, I tried in Hybrid root folder, opens but warns with wrong library and plugin paths which neither worked. Then dropped the editor into Hybrid\64\Vapoursynth, can't even open the editor with a "entry point procedure couldn¡t be found on the library symbolic link" error dialog.

[Selur]

The error should also state 'Please set up the library search paths in settings.'
So,...
a. copy/install VapourSynth Editor anywhere you want.
b. Start it and go to "Edit->Settings->Path->VapourSnyth library search paths", select and add your 'Hybrid/64bit/Vapoursynth', then hit 'Apply'
c. restart Vapoursynth Editor
Note that Hybrid by default uses separate folders for the plugins, thus you need to explicitly load dependencies.

Cu Selur