VSGAN - VapourSynth GAN Implementation, based on ESRGAN's implementation

[Selur]

New problem,...

I updated VSGAN:

I:\Hybrid\64bit\Vapoursynth>python -m pip install --upgrade vsgan
Requirement already satisfied: vsgan in i:\hybrid\64bit\vapoursynth\lib\site-packages (1.2.1)
Collecting vsgan
  Downloading vsgan-1.3.0-py3-none-any.whl (10 kB)
Collecting torch<2.0.0,>=1.9.1
  Using cached torch-1.9.1-cp39-cp39-win_amd64.whl (222.0 MB)
Collecting numpy==1.19.5
  Downloading numpy-1.19.5-cp39-cp39-win_amd64.whl (13.3 MB)
     |████████████████████████████████| 13.3 MB 2.2 MB/s
Requirement already satisfied: typing-extensions in i:\hybrid\64bit\vapoursynth\lib\site-packages (from torch<2.0.0,>=1.9.1->vsgan) (3.10.0.2)
Installing collected packages: torch, numpy, vsgan
  Attempting uninstall: torch
    Found existing installation: torch 1.9.0+cu111
    Uninstalling torch-1.9.0+cu111:
      Successfully uninstalled torch-1.9.0+cu111
  WARNING: The scripts convert-caffe2-to-onnx.exe and convert-onnx-to-caffe2.exe are installed in 'I:\Hybrid\64bit\Vapoursynth\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  Attempting uninstall: numpy
    Found existing installation: numpy 1.21.2
    Uninstalling numpy-1.21.2:
      Successfully uninstalled numpy-1.21.2
  WARNING: The script f2py.exe is installed in 'I:\Hybrid\64bit\Vapoursynth\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  Attempting uninstall: vsgan
    Found existing installation: vsgan 1.2.1
    Uninstalling vsgan-1.2.1:
      Successfully uninstalled vsgan-1.2.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.10.0+cu111 requires torch==1.9.0, but you have torch 1.9.1 which is incompatible.
torchaudio 0.9.0 requires torch==1.9.0, but you have torch 1.9.1 which is incompatible.
Successfully installed numpy-1.19.5 torch-1.9.1 vsgan-1.3.0

I:\Hybrid\64bit\Vapoursynth>python -m pip install --upgrade vsgan
Requirement already satisfied: vsgan in i:\hybrid\64bit\vapoursynth\lib\site-packages (1.3.0)
Requirement already satisfied: torch<2.0.0,>=1.9.1 in i:\hybrid\64bit\vapoursynth\lib\site-packages (from vsgan) (1.9.1)
Requirement already satisfied: numpy==1.19.5 in i:\hybrid\64bit\vapoursynth\lib\site-packages (from vsgan) (1.19.5)
Requirement already satisfied: typing-extensions in i:\hybrid\64bit\vapoursynth\lib\site-packages (from torch<2.0.0,>=1.9.1->vsgan) (3.10.0.2)

and now I get:

Failed to evaluate the script:
Python exception: VSGAN: Either NVIDIA CUDA or the device (cuda) isn't available.

Traceback (most recent call last):
  File "src\cython\vapoursynth.pyx", line 2324, in vapoursynth.vpy_evaluateScript
  File "src\cython\vapoursynth.pyx", line 2325, in vapoursynth.vpy_evaluateScript
  File "C:\Users\Selur\Desktop\test2.vpy", line 29, in <module>
    vsgan = VSGAN("cuda")
  File "I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsgan\__init__.py", line 34, in __init__
    raise EnvironmentError("VSGAN: Either NVIDIA CUDA or the device (%s) isn't available." % device)
OSError: VSGAN: Either NVIDIA CUDA or the device (cuda) isn't available.

when calling:

# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# Import scripts
import mvsfunc
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
from vsgan import VSGAN
# adjusting color space from YUV420P8 to RGB24 for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
# resizing using VSGAN
vsgan = VSGAN("cuda")
model = "I:/Hybrid/64bit/vsgan_models/4x_BSRGAN.pth"
vsgan.load_model(model)
clip = vsgan.run(clip=clip) # 2560x1408
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1056, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output(

Any idea what I can do how to fix this?

Cu Selur

[Selur]

Okay, fixed it by calling:

python.exe -m pip install torch===1.9.1+cu111 torchvision===0.10.1 -f https://download.pytorch.org/whl/torch_stable.html

Cu Selur

[PRAGMA]

Originally Posted by Selur

Okay, fixed it by calling:

python.exe -m pip install torch===1.9.1+cu111 torchvision===0.10.1 -f https://download.pytorch.org/whl/torch_stable.html

Cu Selur

Yes it's a common mistake to let the vsgan deps install torch (which does not include cuda bundled in). However, it would be a mistake to not count pytorch as a dependency.

PyTorch honestly isnt normally meant to have cuda bundled, its just a possible installation method. By default/normally, it expects it installed on the system globally of course, which is why this deps installation *can* work, but wont for a portable install or those who dont want to manually install cuda.

Regardless, I'm glad you have it solved. To reiterate, a new troubleshooting page exists which has this issue listed, if you encounter further issues perhaps refer to it: https://rlaphoenix.github.io/VSGAN/troubleshooting/

[PRAGMA]

VSGAN 1.4.0 has padded/overlapping chunking support. However, it can now only chunk to 4 quadrants only. Perhaps in the future I will implement the ability to chunk each quadrant as well.

[PRAGMA]

Chaining models is broken in v1.4.0 due to a FrameEval mistake. Please install from source code or wait for the next version which will be soon hopefully.

Edit: Fixed in 1.4.1

[PRAGMA]

RealESRGAN support has been added. It's code is NOT great, but it works. Its basically using new-arch of normal ESRGAN (which is restrictive and annoying) but is necessary for the new models RealESRGAN uses.

It will be in the next release after 1.4.1.

EDIT: Full Real-ESRGAN support, including 2x and 1x scale models is added to code, and it no longer uses new-arch, so it works a lot better now.
It will be in v1.5.0.

[poisondeathray]

Thanks for the updates, especially overlap chunking

[poisondeathray]

Thanks also for A-ESRGAN

https://github.com/aesrgan/A-ESRGAN

v1.5.0 Latest

    Adds support for ESRGAN+ models, Real-ESRGAN models (including 2x and 1x if pixel-shuffle was used),
    and A-ESRGAN models

[PRAGMA]

Ok so more updates.

Support for half-accuracy was added to all models. This means you can reduce accuracy, which reduces VRAM in ultimately half, which also increases speed by up to double in some cases. It will decrease the accuracy of course, but yeah. For those with very limited VRAM, half accuracy + seamless chunk mode may help quite a bit. This half mode will be applied when you use load_model with `half=True` and will be added to the next version (probably 1.5.1).

Support for ESRGAN+, Real-ESRGAN, and A-ESRGAN are now supported in v1.5.0. With Real-ESRGAN v2 (basically ultra-lite Real-ESRGAN) coming in the next version.

What may appear in next version, if not then the version after is support for EGVSR, which is an inter-frame architecture taking in data from n neighboring frames. It's still in testing and still being worked on but its going well. It is at a state where basic models work, but detection of arch settings has not been worked on yet.

Once auto-detection of arch settings can be done, support will be added in whatever the next version from then is.

[Selur]

Nice! Looking forward to it.

[ReinerSchweinlin]

nice

From my experience with other FP16 Models, Quality still is fine in most cases - and the speedup (on FP16 2:1 vapable cards like recent AMD, GEN20 RTX..) as well as the halved VRAM needed is a very good thing - rarely turning back to FP32 models these days.

[PRAGMA]

Ok v1.6.0 has been released, with a fairly big change to how VSGAN is ultimately used. This update includes the previously talked about features like half-precision, Real-ESRGANv2, EGVSR, and more.
There's also an upgraded tiling system that will automatically enable or disable depending on if you need the VRAM or not. It will also recursively tile to reduce VRAM even further (if necessary of course).

Please note though, that EGVSR does not auto-detect settings from the model file yet. So you should manually override any settings that needs it when you load the model for correct results.

For more details see the changelog: https://vsgan.phoeniix.dev/en/stable/changelog.html

[poisondeathray]

Thanks for adding EGVSR.

Originally Posted by PRAGMA

Please note though, that EGVSR does not auto-detect settings from the model file yet. So you should manually override any settings that needs it when you load the model for correct results.

Can you explain how to do that ?

I'm getting bizarre results with the "EGVSR_iter420000.pth" model - it looks like some fine grid pattern overlaid

[PRAGMA]

Originally Posted by poisondeathray

Thanks for adding EGVSR.




Can you explain how to do that ?

I'm getting bizarre results with the "EGVSR_iter420000.pth" model - it looks like some fine grid pattern overlaid

Refer to https://github.com/rlaphoenix/VSGAN/issues/18
maybe this issue was made by you? not sure.

[poisondeathray]

Originally Posted by PRAGMA

Refer to https://github.com/rlaphoenix/VSGAN/issues/18
maybe this issue was made by you? not sure.

Not me, but it's a similar overlay grid pattern

(That guy should IVTC first)

Adding nb=10, degradation="BD" fixes it, thanks

load(r'PATH\EGVSR_iter420000.pth', nb=10, degradation="BD")

[Selur]

Small question, so when using a ESRGAN,ESRGAN+,Real-ESRGAN or A-ESRGAN model I use

from vsgan import ESRGAN

and when using a EGVSR model I use:

from vsgan import EGVSR

correct?

Also: Does anyone know a source for EGVSR models?
(only one I found so far is from https://github.com/Thmen/EGVSR/tree/...trained_models)

[poisondeathray]

Originally Posted by Selur

Small question, so when using a ESRGAN,ESRGAN+,Real-ESRGAN or A-ESRGAN model I use

from vsgan import ESRGAN

and when using a EGVSR model I use:

from vsgan import EGVSR

correct?

Works for me

Also: Does anyone know a source for EGVSR models?

AFAIK there is only 1 model (from the official repo)

[Selur]

Ah, okay.
Things seem to work fine with ESRGAN models (like BSRGAN), but using:

# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# Import scripts
import mvsfunc
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting color space from YUV420P8 to RGB24 for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
# resizing using VSGAN
from vsgan import EGVSR
vsgan = EGVSR(clip=clip,device="cuda")
model = "I:/Hybrid/64bit/vsgan_models/4x_iter420000_EGVSR.pth"
vsgan.load(model, nb=10, degradation="BD", out_nc=3, nf=64)
vsgan.apply() # 2560x1408
clip = vsgan.clip
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1056, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

I run out of VRAM

CUDA out of memory. Tried to allocate 56.00 MiB (GPU 0; 8.00 GiB total capacity; 6.62 GiB already allocated; 0 bytes free; 6.76 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Sadly 'overlap' seems to only be implemented for ESRGAN, since when using "vsgan.apply(overlap=16)" in the above example, I get:

Python exception: apply() got an unexpected keyword argument 'overlap'

Cu Selur

[poisondeathray]

It looks like tiling is only implemented in ESRGAN + derivatives, but not EGVSR yet

[Selur]

Okay, also got another issue with the current version.
Using:

# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# Import scripts
import mvsfunc
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting color space from YUV420P8 to RGB24 for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "I:/Hybrid/64bit/vsgan_models/4x_BSRGAN.pth"
vsgan.load(model)
vsgan.apply() # 2560x1408
clip = vsgan.clip
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=640, h=352, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

When I open this, it takes a bit to calculate the first frame during which VRAM usage goes up to the max (8GB) and doesn't go down again and hinders me from looking at another frame. :/
when using "vsgan.apply(overlap=16)" the same happens.
-> at least here the new version does only work for one frame.
I also tried different other models, all seem to have the same effect on my system.
I sometimes I can step through a few frames (two work, third always fails), but then vram gets stuck and thats it.