VSGAN - VapourSynth GAN Implementation, based on ESRGAN's implementation

[PRAGMA]

https://github.com/rlaphoenix/VSGAN
docs: https://vsgan.phoeniix.dev

VSGAN as of 1.6.0 is a port of multiple kinds of machine learning GAN Networks and Architectures.
As of 1.6.0, ESRGAN (and its derivatives), and EGVSR is supported.

For much more information, getting started, more details on supported models, networks, architectures, check the docs.

[poisondeathray]

Thanks, it works well

Any lists for nice pre-trained models ? I found a bunch on Google , but does anyone have any "favorites" that work well in specific situations?
(including Pragma's own AD model)

https://upscale.wiki/wiki/Model_Database

[PRAGMA]

Thanks, it works well

Any lists for nice pre-trained models ? I found a bunch on Google , but does anyone have any "favorites" that work well in specific situations?
(including Pragma's own AD model)

https://upscale.wiki/wiki/Model_Database

Yep, take a look at:
https://github.com/WolframRhodium/Super-Resolution-Zoo

[poisondeathray]

Yep, take a look at:
https://github.com/WolframRhodium/Super-Resolution-Zoo

Yes, I remember this, from the other thread - and I could never get that working

But these are converted models (params , json ; not pth), are they drop in compatible ?

(And some of the links to the original projects and papers have pth models)

[ChaosKing]

Thanks, it works well

Any lists for nice pre-trained models ? I found a bunch on Google , but does anyone have any "favorites" that work well in specific situations?
(including Pragma's own AD model)

https://upscale.wiki/wiki/Model_Database

I tested some filters and the most usefull (and safe) one which also does not generate artifacts was 1x_DeSharpen.

The "spongebob model" looked also not bad for some older animes but had some blurry spots for some reason.

[ChaosKing]

I get out of memory errors for 1080p content using a GTX 1070. Does it really need what much ram? used model 1x_DeSharpen

CUDA out of memory. Tried to allocate 1.48 GiB (GPU 0; 8.00 GiB total capacity; 2.56 GiB already allocated; 72.05 MiB free; 3.65 GiB cached)

Restarting vsedit didn't help.

[poisondeathray]

I think so...I can only use 1x_Desharpen on smaller dimensions . You can look at GPU caps viewer or similar utilities and it looks like it's using all

I wonder if there is a way to share system memory with CUDA memory ? eg. Although it's slower, some 3D/CG renderers can offload graphics card memory to system memory when doing calculations (like a shared pool) enabling you to complete if the scene is too large

[brucethemoose]

I think so...I can only use 1x_Desharpen on smaller dimensions . You can look at GPU caps viewer or similar utilities and it looks like it's using all

I wonder if there is a way to share system memory with CUDA memory ? eg. Although it's slower, some 3D/CG renderers can offload graphics card memory to system memory when doing calculations (like a shared pool) enabling you to complete if the scene is too large

I believe graphics drivers automatically do this in 3D programs, but that might not be the case with CUDA.

PyTorch does have a "empty cache" function. Maybe you could call it every few frames with FrameEval (http://www.vapoursynth.com/doc/functions/frameeval.html).

https://pytorch.org/docs/stable/cuda.html#memory-management

[poisondeathray]

I believe graphics drivers automatically do this in 3D programs, but that might not be the case with CUDA.


Definitely CUDA for some 3D renderers like redshift . It's known as "out of core" rendering. I don't think it's an "automatic" feature for CUDA, some programmer figured it out. It's a big selling point for some types of renderers



PyTorch does have a "empty cache" function. Maybe you could call it every few frames with FrameEval (http://www.vapoursynth.com/doc/functions/frameeval.html).

https://pytorch.org/docs/stable/cuda.html#memory-management

Thanks, but I can't even do 1 1920x1080 frame, it probably won't help calling it periodically. It's only that 1x_DeSharpen, the other models work ok (I didn't test them all , just a few) . So maybe something is up with that model

[PRAGMA]

I get out of memory errors for 1080p content using a GTX 1070. Does it really need what much ram? used model 1x_DeSharpen

CUDA out of memory. Tried to allocate 1.48 GiB (GPU 0; 8.00 GiB total capacity; 2.56 GiB already allocated; 72.05 MiB free; 3.65 GiB cached)

Restarting vsedit didn't help.

I tested some filters and the most usefull (and safe) one which also does not generate artifacts was 1x_DeSharpen.

The "spongebob model" looked also not bad for some older animes but had some blurry spots for some reason.

This is an ESRGAN port, so it will generate results the exact same as ESRGAN, and use the same amount of memory as ESRGAN.
But yes, generally speaking x1080 resolution content require quite a bit of memory, i'm trying to work out a way to split up the input frame into multiple split images, to then run them all separately and then combine them back to 1 frame and return, that way using the same amount of memory, but less memory simultaneously, would be slower, but would be the only viable option for most GPU's.
I'm having a bit of trouble working it out right now but hopefully I or someone else, figures out a way to do so.

[WolframRhodium]

i'm trying to work out a way to split up the input frame into multiple split images, to then run them all separately and then combine them back to 1 frame and return

Example from EDSR: https://github.com/thstkdgus35/EDSR-PyTorch/blob/9d3bb0ec620ea2ac1b5e5e7a32b0133fbba66fd2/src/model/__init__.py#L105

[PRAGMA]

Example from EDSR: https://github.com/thstkdgus35/EDSR-PyTorch/blob/9d3bb0ec620ea2ac1b5e5e7a32b0133fbba66fd2/src/model/__init__.py#L105

I updated VSGAN with chunk support, it will chunk into 4 sub images, and render them separately instead of all at once, then re-compile it back into 1 image and returns to frame server.

I didn't end up using anything like EDSR though, instead I simply cropped the frame, and then ran them.

[poisondeathray]

Thanks for the update

Request: If you have time would it be possible to port EDVR ? Looks great and deals with video specific issues, temporal and motion blur

https://xinntao.github.io/projects/EDVR
https://github.com/xinntao/EDVR

[aegisofrime]

Hey PRAGMA,

Just wanted to let you know that the __init__.py file installed by pip and by the zip file provided by you in the manual installation steps have different file sizes. Is this intended?

[poisondeathray]

Request#2 , another pytorch implementation, PPON

https://github.com/Zheng222/PPON

Someone forked it and wrote some demo code , would it be possible to implement in vapoursynth ?
https://github.com/vlad3996/PPON/blob/master/inference_utils.py
https://github.com/vlad3996/PPON/blob/master/demo.ipynb

Thanks

[PRAGMA]

Hey PRAGMA,

Just wanted to let you know that the __init__.py file installed by pip and by the zip file provided by you in the manual installation steps have different file sizes. Is this intended?

I believe when uploaded/setup for PyPI is gets optimized/minified or something before being sent off to PyPI's repo, but I could be wrong, regardless, they are both up to date both the repo and PyPI pip package.

[aegisofrime]

I believe when uploaded/setup for PyPI is gets optimized/minified or something before being sent off to PyPI's repo, but I could be wrong, regardless, they are both up to date both the repo and PyPI pip package.

Well, maybe it's something silly I did somewhere, but I got an unable to import VSGAN error using pip, but was able to use it with the manual installation zip you provided.

As an aside, I wonder what speeds are you folks getting? With a GTX 1070, upscaling 480p content using the "PSNR_x4_DB.pth" model, I'm getting like... 1fps :/

[PRAGMA]

Well, maybe it's something silly I did somewhere, but I got an unable to import VSGAN error using pip, but was able to use it with the manual installation zip you provided.

As an aside, I wonder what speeds are you folks getting? With a GTX 1070, upscaling 480p content using the "PSNR_x4_DB.pth" model, I'm getting like... 1fps :/

Yeah on a GTX 1080ti it takes about 1.8s per frame for a 720x480 input image.
RTX 2080ti takes about 1.4s per frame for the same input.

[Cary Knoop]

Yeah on a GTX 1080ti it takes about 1.8s per frame for a 720x480 input image.
RTX 2080ti takes about 1.4s per frame for the same input.
That's strange, why would the 2080ti be slower than the 1080ti?

[poisondeathray]

That's strange, why would the 2080ti be slower than the 1080ti?

lower sec / frame means it 's faster

[Cary Knoop]

lower sec / frame means it 's faster
Oops, my mistake, I thought I read frames per second.

[Cary Knoop]

It would probably be useful to add installation instructions for portable Vapoursynth installs as well.

[AlphaAtlas]

It would probably be useful to add installation instructions for portable Vapoursynth installs as well.

It works the same in the VS Fatpack for me. Just open a console window wherever python.exe is, and then install all the dependencies with pip the same way.

Unfortunately CUDA isn't portable, so that part needs a system install.

[Cary Knoop]

It works the same in the VS Fatpack for me. Just open a console window wherever python.exe is, and then install all the dependencies with pip the same way.

It's looking for the Vapoursynth installation path.

[PRAGMA]

It's looking for the Vapoursynth installation path.

VSGAN doesn't care where or how you have VapourSynth installed. All it cares is you have installed it via pip and that you have all dependencies especially PyTorch installed.

[Cary Knoop]

VSGAN doesn't care where or how you have VapourSynth installed. All it cares is you have installed it via pip and that you have all dependencies especially PyTorch installed.
That was my whole point if you have an install (not through pip) it fails.

[brucethemoose]

That was my whole point if you have an install (not through pip) it fails.

AFAIK it's not really practical to install without pip, if that's what you mean.

To be clear, what I do with the vapoursynth fatpack is call "C:/pathtofatpack/VapourSynth64/python.exe -m pip install vsgan". You have to specify the portable python path, otherwise you'll call pip in a python setup that doesn't have access to VS.

On Linux, you can also set the PYTHONPATH environment variable in the terminal if you have multiple Python installs.

[Cary Knoop]

To be clear, what I do with the vapoursynth fatpack is call "C:/pathtofatpack/VapourSynth64/python.exe -m pip install vsgan". You have to specify the portable python path, otherwise you'll call pip in a python setup that doesn't have access to VS.

That would work.

I merely wanted to point out that not everybody will know this.

[Selur]

Here's what I did:

download lates VapourSynth64Portable
https://github.com/theChaosCoder/vapoursynth-portable-FATPACK/releases
upgrade pip
python.exe -m pip install --upgrade pip
installed torch (inside the Vapoursynth64 folder)
python.exe -m pip install torch===1.4.0 torchvision===0.5.0 -f https://download.pytorch.org/whl/torch_stable.html
installed vsgan:
python -m pip install vsgan


Then I used a script that worked before:
# Imports
import vapoursynth as vs
core = vs.get_core()
# Loading Plugins
from vsgan import VSGAN
vsgan_device = VSGAN("cuda")
# Loading C:/Users/Selur/Desktop/5000frames.mp4 using LWLibavSource
clip = core.lsmas.LWLibavSource(source="F:/TestClips&Co/files/5000frames.mp4", format="YUV420P10", cache=0, prefer_hw=0)
# making sure input color matrix is set as 470bg
clip = core.resize.Point(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)

vsgan_device.load_model(
model="I:/Hybrid/64bit/vsfilters/ResizeFilter/VSGRAN/models/4x_xbrz_90k.pth",
scale=2
)
clip = vsgan_device.run(clip=clip, chunk=True)

clip.set_output()
(see: https://github.com/rlaPHOENiX/VSGAN/issues/1)
which now gives me:

Python exception: Error(s) in loading state_dict for RRDBNet:
Missing key(s) in state_dict: "conv_first.weight", "conv_first.bias", "RRDB_trunk.0.RDB1.conv1.weight", "RRDB_trunk.0.RDB1.conv1.bias", "RRDB_trunk.0.RDB1.conv2.weight", "RRDB_trunk.0.RDB1.conv2.bias", "RRDB_trunk.0.RDB1.conv3.weight", "RRDB_trunk.0.RDB1.conv3.bias", "RRDB_trunk.0.RDB1.conv4.weight", "RRDB_trunk.0.RDB1.conv4.bias", "RRDB_trunk.0.RDB1.conv5.weight", "RRDB_trunk.0.RDB1.conv5.bias", "RRDB_trunk.0.RDB2.conv1.weight", "RRDB_trunk.0.RDB2.conv1.bias", "RRDB_trunk.0.RDB2.conv2.weight", "RRDB_trunk.0.RDB2.conv2.bias", "RRDB_trunk.0.RDB2.conv3.weight", "RRDB_trunk.0.RDB2.conv3.bias", "RRDB_trunk.0.RDB2.conv4.weight", "RRDB_trunk.0.RDB2.conv4.bias", "RRDB_trunk.0.RDB2.conv5.weight", "RRDB_trunk.0.RDB2.conv5.bias", "RRDB_trunk.0.RDB3.conv1.weight", "RRDB_trunk.0.RDB3.conv1.bias", "RRDB_trunk.0.RDB3.conv2.weight", "RRDB_trunk.0.RDB3.conv2.bias", "RRDB_trunk.0.RDB3.conv3.weight", "RRDB_trunk.0.RDB3.conv3.bias", "RRDB_trunk.0.RDB3.conv4.weight", "RRDB_trunk.0.RDB3.conv4.bias", "RRDB_trunk.0.RDB3.conv5.weight", "RRDB_trunk.0.RDB3.conv5.bias", "RRDB_trunk.1.RDB1.conv1.weight", "RRDB_trunk.1.RDB1.conv1.bias", "RRDB_trunk.1.RDB1.conv2.weight", "RRDB_trunk.1.RDB1.conv2.bias", "RRDB_trunk.1.RDB1.conv3.weight", "RRDB_trunk.1.RDB1.conv3.bias", "RRDB_trunk.1.RDB1.conv4.weight", "RRDB_trunk.1.RDB1.conv4.bias", "RRDB_trunk.1.RDB1.conv5.weight", "RRDB_trunk.1.RDB1.conv5.bias", "RRDB_trunk.1.RDB2.conv1.weight", "RRDB_trunk.1.RDB2.conv1.bias", "RRDB_trunk.1.RDB2.conv2.weight", "RRDB_trunk.1.RDB2.conv2.bias", "RRDB_trunk.1.RDB2.conv3.weight", "RRDB_trunk.1.RDB2.conv3.bias", "RRDB_trunk.1.RDB2.conv4.weight", "RRDB_trunk.1.RDB2.conv4.bias", "RRDB_trunk.1.RDB2.conv5.weight", "RRDB_trunk.1.RDB2.conv5.bias", "RRDB_trunk.1.RDB3.conv1.weight", "RRDB_trunk.1.RDB3.conv1.bias", "RRDB_trunk.1.RDB3.conv2.weight", "RRDB_trunk.1.RDB3.conv2.bias", "RRDB_trunk.1.RDB3.conv3.weight", "RRDB_trunk.1.RDB3.conv3.bias", "RRDB_trunk.1.RDB3.conv4.weight", "RRDB_trunk.1.RDB3.conv4.bias", "RRDB_trunk.1.RDB3.conv5.weight", "RRDB_trunk.1.RDB3.conv5.bias", "RRDB_trunk.2.RDB1.conv1.weight", "RRDB_trunk.2.RDB1.conv1.bias", "RRDB_trunk.2.RDB1.conv2.weight", "RRDB_trunk.2.RDB1.conv2.bias", "RRDB_trunk.2.RDB1.conv3.weight", "RRDB_trunk.2.RDB1.conv3.bias", "RRDB_trunk.2.RDB1.conv4.weight", "RRDB_trunk.2.RDB1.conv4.bias", "RRDB_trunk.2.RDB1.conv5.weight", "RRDB_trunk.2.RDB1.conv5.bias", "RRDB_trunk.2.RDB2.conv1.weight", "RRDB_trunk.2.RDB2.conv1.bias", "RRDB_trunk.2.RDB2.conv2.weight", "RRDB_trunk.2.RDB2.conv2.bias", "RRDB_trunk.2.RDB2.conv3.weight", "RRDB_trunk.2.RDB2.conv3.bias", "RRDB_trunk.2.RDB2.conv4.weight", "RRDB_trunk.2.RDB2.conv4.bias", "RRDB_trunk.2.RDB2.conv5.weight", "RRDB_trunk.2.RDB2.conv5.bias", "RRDB_trunk.2.RDB3.conv1.weight", "RRDB_trunk.2.RDB3.conv1.bias", "RRDB_trunk.2.RDB3.conv2.weight", "RRDB_trunk.2.RDB3.conv2.bias", "RRDB_trunk.2.RDB3.conv3.weight", "RRDB_trunk.2.RDB3.conv3.bias", "RRDB_trunk.2.RDB3.conv4.weight", "RRDB_trunk.2.RDB3.conv4.bias", "RRDB_trunk.2.RDB3.conv5.weight", "RRDB_trunk.2.RDB3.conv5.bias", "RRDB_trunk.3.RDB1.conv1.weight", "RRDB_trunk.3.RDB1.conv1.bias", "RRDB_trunk.3.RDB1.conv2.weight", "RRDB_trunk.3.RDB1.conv2.bias", "RRDB_trunk.3.RDB1.conv3.weight", "RRDB_trunk.3.RDB1.conv3.bias", "RRDB_trunk.3.RDB1.conv4.weight", "RRDB_trunk.3.RDB1.conv4.bias", "RRDB_trunk.3.RDB1.conv5.weight", "RRDB_trunk.3.RDB1.conv5.bias", "RRDB_trunk.3.RDB2.conv1.weight", "RRDB_trunk.3.RDB2.conv1.bias", "RRDB_trunk.3.RDB2.conv2.weight", "RRDB_trunk.3.RDB2.conv2.bias", "RRDB_trunk.3.RDB2.conv3.weight", "RRDB_trunk.3.RDB2.conv3.bias", "RRDB_trunk.3.RDB2.conv4.weight", "RRDB_trunk.3.RDB2.conv4.bias", "RRDB_trunk.3.RDB2.conv5.weight", "RRDB_trunk.3.RDB2.conv5.bias", "RRDB_trunk.3.RDB3.conv1.weight", "RRDB_trunk.3.RDB3.conv1.bias", "RRDB_trunk.3.RDB3.conv2.weight", "RRDB_trunk.3.RDB3.conv2.bias", "RRDB_trunk.3.RDB3.conv3.weight", "RRDB_trunk.3.RDB3.conv3.bias", "RRDB_trunk.3.RDB3.conv4.weight", "RRDB_trunk.3.RDB3.conv4.bias", "RRDB_trunk.3.RDB3.conv5.weight", "RRDB_trunk.3.RDB3.conv5.bias", "RRDB_trunk.4.RDB1.conv1.weight", "RRDB_trunk.4.RDB1.conv1.bias", "RRDB_trunk.4.RDB1.conv2.weight", "RRDB_trunk.4.RDB1.conv2.bias", "RRDB_trunk.4.RDB1.conv3.weight", "RRDB_trunk.4.RDB1.conv3.bias", "RRDB_trunk.4.RDB1.conv4.weight", "RRDB_trunk.4.RDB1.conv4.bias", "RRDB_trunk.4.RDB1.conv5.weight", "RRDB_trunk.4.RDB1.conv5.bias", "RRDB_trunk.4.RDB2.conv1.weight", "RRDB_trunk.4.RDB2.conv1.bias", "RRDB_trunk.4.RDB2.conv2.weight", "RRDB_trunk.4.RDB2.conv2.bias", "RRDB_trunk.4.RDB2.conv3.weight", "RRDB_trunk.4.RDB2.conv3.bias", "RRDB_trunk.4.RDB2.conv4.weight", "RRDB_trunk.4.RDB2.conv4.bias", "RRDB_trunk.4.RDB2.conv5.weight", "RRDB_trunk.4.RDB2.conv5.bias", "RRDB_trunk.4.RDB3.conv1.weight", "RRDB_trunk.4.RDB3.conv1.bias", "RRDB_trunk.4.RDB3.conv2.weight", "RRDB_trunk.4.RDB3.conv2.bias", "RRDB_trunk.4.RDB3.conv3.weight", "RRDB_trunk.4.RDB3.conv3.bias", "RRDB_trunk.4.RDB3.conv4.weight", "RRDB_trunk.4.RDB3.conv4.bias", "RRDB_trunk.4.RDB3.conv5.weight", "RRDB_trunk.4.RDB3.conv5.bias", "RRDB_trunk.5.RDB1.conv1.weight", "RRDB_trunk.5.RDB1.conv1.bias", "RRDB_trunk.5.RDB1.conv2.weight", "RRDB_trunk.5.RDB1.conv2.bias", "RRDB_trunk.5.RDB1.conv3.weight", "RRDB_trunk.5.RDB1.conv3.bias", "RRDB_trunk.5.RDB1.conv4.weight", "RRDB_trunk.5.RDB1.conv4.bias", "RRDB_trunk.5.RDB1.conv5.weight", "RRDB_trunk.5.RDB1.conv5.bias", "RRDB_trunk.5.RDB2.conv1.weight", "RRDB_trunk.5.RDB2.conv1.bias", "RRDB_trunk.5.RDB2.conv2.weight", "RRDB_trunk.5.RDB2.conv2.bias", "RRDB_trunk.5.RDB2.conv3.weight", "RRDB_trunk.5.RDB2.conv3.bias", "RRDB_trunk.5.RDB2.conv4.weight", "RRDB_trunk.5.RDB2.conv4.bias", "RRDB_trunk.5.RDB2.conv5.weight", "RRDB_trunk.5.RDB2.conv5.bias", "RRDB_trunk.5.RDB3.conv1.weight",
....
Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 1956, in vapoursynth.vpy_evaluateScript
File "src\cython\vapoursynth.pyx", line 1957, in vapoursynth.vpy_evaluateScript
File "C:/Users/Selur/Desktop/vsgantest.vpy", line 16, in
vsgan_device.load_model(
File "C:\Users\Selur\Desktop\VapourSynth64Portable\VapourSynth64\Lib\site-packages\vsgan\__init__.py", line 34, in load_model
self.rrdb_net_model.load_state_dict(torch.load(self.model_file), strict=True)
File "C:\Users\Selur\Desktop\VapourSynth64Portable\VapourSynth64\Lib\site-packages\torch\nn\modules\module.py", line 829, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for RRDBNet:
Missing key(s) in state_dict: "conv_first.weight", "conv_first.bias", "RRDB_trunk.0.RDB1.conv1.weight", "RRDB_trunk.0.RDB1.conv1.bias", "RRDB_trunk.0.RDB1.conv2.weight", "RRDB_trunk.0.RDB1.conv2.bias", "RRDB_trunk.0.RDB1.conv3.weight", "RRDB_trunk.0.RDB1.conv3.bias", "RRDB_trunk.0.RDB1.conv4.weight", "RRDB_trunk.0.RDB1.conv4.bias", "RRDB_trunk.0.RDB1.conv5.weight", "RRDB_trunk.0.RDB1.conv5.bias", "RRDB_trunk.0.RDB2.conv1.weight", "RRDB_trunk.0.RDB2.conv1.bias", "RRDB_trunk.0.RDB2.conv2.weight", "RRDB_trunk.0.RDB2.conv2.bias", "RRDB_trunk.0.RDB2.conv3.weight", "RRDB_trunk.0.RDB2.conv3.bias", "RRDB_trunk.0.RDB2.conv4.weight", "RRDB_trunk.0.RDB2.conv4.bias", "RRDB_trunk.0.RDB2.conv5.weight", "RRDB_trunk.0.RDB2.conv5.bias", "RRDB_trunk.0.RDB3.conv1.weight", "RRDB_trunk.0.RDB3.conv1.bias", "RRDB_trunk.0.RDB3.conv2.weight", "RRDB_trunk.0.RDB3.conv2.bias", "RRDB_trunk.0.RDB3.conv3.weight", "RRDB_trunk.0.RDB3.conv3.bias", "RRDB_trunk.0.RDB3.conv4.weight", "RRDB_trunk.0.RDB3.conv4.bias", "RRDB_trunk.0.RDB3.conv5.weight", "RRDB_trunk.0.RDB3.conv5.bias", "RRDB_trunk.1.RDB1.conv1.weight", "RRDB_trunk.1.RDB1.conv1.bias", "RRDB_trunk.1.RDB1.conv2.weight", "RRDB_trunk.1.RDB1.conv2.bias", "RRDB_trunk.1.RDB1.conv3.weight", "RRDB_trunk.1.RDB1.conv3.bias", "RRDB_trunk.1.RDB1.conv4.weight", "RRDB_trunk.1.RDB1.conv4.bias", "RRDB_trunk.1.RDB1.conv5.weight", "RRDB_trunk.1.RDB1.conv5.bias", "RRDB_trunk.1.RDB2.conv1.weight", "RRDB_trunk.1.RDB2.conv1.bias", "RRDB_trunk.1.RDB2.conv2.weight", "RRDB_trunk.1.RDB2.conv2.bias", "RRDB_trunk.1.RDB2.conv3.weight", "RRDB_trunk.1.RDB2.conv3.bias", "RRDB_trunk.1.RDB2.conv4.weight", "RRDB_trunk.1.RDB2.conv4.bias", "RRDB_trunk.1.RDB2.conv5.weight", "RRDB_trunk.1.RDB2.conv5.bias", "RRDB_trunk.1.RDB3.conv1.weight", "RRDB_trunk.1.RDB3.conv1.bias", "RRDB_trunk.1.RDB3.conv2.weight", "RRDB_trunk.1.RDB3.conv2.bias", "RRDB_trunk.1.RDB3.conv3.weight", "RRDB_trunk.1.RDB3.conv3.bias", "RRDB_trunk.1.RDB3.conv4.weight", "RRDB_trunk.1.RDB3.conv4.bias", "RRDB_trunk.1.RDB3.conv5.weight", "RRDB_trunk.1.RDB3.conv5.bias", "RRDB_trunk.2.RDB1.conv1.weight", "RRDB_trunk.2.RDB1.conv1.bias", "RRDB_trunk.2.RDB1.conv2.weight", "RRDB_trunk.2.RDB1.conv2.bias", "RRDB_trunk.2.RDB1.conv3.weight", "RRDB_trunk.2.RDB1.conv3.bias", "RRDB_trunk.2.RDB1.conv4.weight", "RRDB_trunk.2.RDB1.conv4.bias", "RRDB_trunk.2.RDB1.conv5.weight", "RRDB_trunk.2.RDB1.conv5.bias", "RRDB_trunk.2.RDB2.conv1.weight", "RRDB_trunk.2.RDB2.conv1.bias", "RRDB_trunk.2.RDB2.conv2.weight", "RRDB_trunk.2.RDB2.conv2.bias", "RRDB_trunk.2.RDB2.conv3.weight", "RRDB_trunk.2.RDB2.conv3.bias", "RRDB_trunk.2.RDB2.conv4.weight", "RRDB_trunk.2.RDB2.conv4.bia
...


Okay, I thought since its 4 months since I last looked at this may be something changed that I'm not aware of so I looked at https://drive.google.com/drive/u/0/folders/17VYV_SoZZesU6mbxz2dMAIccSSlqLecY and tried the 'RRDB_ESRGAN_x4.pth'-model
using:
# Imports
import vapoursynth as vs
core = vs.get_core()
# Loading Plugins
from vsgan import VSGAN
vsgan_device = VSGAN("cuda")
# Loading C:/Users/Selur/Desktop/5000frames.mp4 using LWLibavSource
clip = core.lsmas.LWLibavSource(source="F:/TestClips&Co/files/5000frames.mp4", format="YUV420P10", cache=0, prefer_hw=0)
# making sure input color matrix is set as 470bg
clip = core.resize.Point(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)

vsgan_device.load_model(
model="I:/Hybrid/64bit/vsfilters/ResizeFilter/VSGRAN/models/RRDB_ESRGAN_x4.pth",
scale=2
)
clip = vsgan_device.run(clip=clip, chunk=True)

clip.set_output()
but that only gives me:

Error on frame 0 request:
error in LoadLibraryA
using RRDB_ESRGAN_x4_old_arch instead of RRDB_ESRGAN_x4 gives me the same errors as with 4x_xbrz_90k.

-> Does anyone know how to fix this?

Cu Selur

[ChaosKing]

You could also try https://github.com/AlphaAtlas/VapourSynth-Super-Resolution-Helper
I noticed that there are some modules that don't play nicely with the python embedded version (https://github.com/Irrational-Encoding-Wizardry/yuuno is one of such module). I think you can take a regular python intstallation and use it more or less as a "portable" version (install in a vm and then just copy the install folder to you hdd)

Maybe then it will work.

[Selur]

The strange thing is that the above did work a while back.
My main problem is I don't know why it isn't working now. :)

[poisondeathray]

You're using a 4x trained model , but specified scale=2 . Did you try scale=4 ?

[Selur]

Nope, since it worked before with 2, but I just did and still got the same error.
Error on frame 0 request:
error in LoadLibraryA

[ChaosKing]

Have you tried newer/older torch torchvision version?

[Selur]

Nope, didn't have the time to play with it that much. Only looked at it since it looked interesting and I thought about adding support for it in Hybrid. ;)
Also since that torch version worked before, I haven't really spend much thought about trying other versions.
Thought others might have played with it and directly knew what was going wrong. :)
(Don't know when I will look at this more actively since I'm quite busy with normal live atm.)

Cu Selur

[_Al_]

Installing pytorch using cpu only, which went well:
pip install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
then for final install:
pip install vsgan
I get error not having Microsoft Visual C++ v 14.0. But I have in PC 2008, 2013 and Microsoft Visual C++ 2017 which is 14.15.26706
Anyone what could go wrong?
C:\Users\xxx\AppData\Local\Programs\Python\Python37\Scripts>pip3 install vsgan
Collecting vsgan
Downloading https://files.pythonhosted.org/packages/1d/eb/369bc17433a3e8ab8bf7ab0a74ab7052f28b35ecb2de07683163d385eb2f/vsgan-1.0.8-py3-none-any.whl
Collecting vapoursynth (from vsgan)
Downloading https://files.pythonhosted.org/packages/41/a6/75c8e6c37e26641f73cd967f4c365655b206f279ddd52461f4a1b9bd1621/VapourSynth-51.zip (426kB)
100% |████████████████████████████████| 430kB 5.4MB/s
Requirement already satisfied: torch in c:\users\xxx\appdata\local\programs\python\python37\lib\site-packages (from vsgan) (1.6.0+cpu)
Requirement already satisfied: numpy in c:\users\xxx\appdata\local\programs\python\python37\lib\site-packages (from vsgan) (1.16.2)
Requirement already satisfied: future in c:\users\xxx\appdata\local\programs\python\python37\lib\site-packages (from torch->vsgan) (0.18.2)
Installing collected packages: vapoursynth, vsgan
Running setup.py install for vapoursynth ... error
Complete output from command c:\users\xxx\appdata\local\programs\python\python37\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\
\xxx\\AppData\\Local\\Temp\\pip-install-s6avgaje\\vapoursynth\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n'
);f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\xxx\AppData\Local\Temp\pip-record-ntdlen2q\install-record.txt --single-ver
sion-externally-managed --compile:
Found VapourSynth.dll at: C:\Program Files (x86)\VapourSynth\core64\vapoursynth.dll
running install
running build
running build_ext
skipping 'src\cython\vapoursynth.c' Cython extension (up-to-date)
building 'vapoursynth' extension
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/

----------------------------------------
Command "c:\users\xxx\appdata\local\programs\python\python37\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\xxxl\\AppData\\Local\\Tem
p\\pip-install-s6avgaje\\vapoursynth\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile
(code, __file__, 'exec'))" install --record C:\Users\xx\AppData\Local\Temp\pip-record-ntdlen2q\install-record.txt --single-version-externally-managed
--compile" failed with error code 1 in C:\Users\xxx\AppData\Local\Temp\pip-install-s6avgaje\vapoursynth\

[Cary Knoop]

Try to uninstall VC 2017 and then install VC 2015 Build Tools and then install VC 2017 again.

[Selur]

btw. has anyone compiled and tried https://github.com/Sg4Dylan/vapoursynth-fsrcnn-ncnn-vulkan ?

[ReinerSchweinlin]

No, I haven´t,

maybe this shader implementation can be used?
https://github.com/igv/FSRCNN-TensorFlow/releases

[Selur]

Since I was testing some stuff here's how on can create portable VSGAN (https://github.com/rlaPHOENiX/VSGAN) version:


Download latest Vapoursynth portabe Fatpack (https://github.com/theChaosCoder/vapoursynth-portable-FATPACK/releases) and extract it to a folder
Delete the links an folders inside the VapourSynth64 aside from VapourSynth64 and the Scripts folder (optional)
Open a Windows command prompt and change into the VapourSynth64/VapourSynth64-folder.
Call `python.exe -m pip install --upgrade pip" to update pip` to update pip
Call `python.exe -m pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio===0.8.1 -f https://download.pytorch.org/whl/torch_stable.html` to install pytorch
For cuda10 use `python.exe -m pip install torch==1.8.1+cu102 torchvision==0.9.1+cu102 torchaudio===0.8.1 -f https://download.pytorch.org/whl/torch_stable.html` to install pytorch
Call `python.exe -m pip install pyd2v pymediainfo vsgan" to install vsgan`
Close Windows Command Prompt.

now you got a 6GB folder containing a portable Vapoursynth (~560MB) with VSGAN (~5.4GB).

In case you want to try out VSGAN:

Download some models
I went to https://upscale.wiki/wiki/Model_Database and then to Anime Oldies Alternative (https://drive.google.com/drive/folders/1ldwajXL50uC7PCS63B4Wato6Dnk-svNL) and downloaded the pth files and extract them somewhere.
Open VapourSynth64Portable\VapourSynth64\vsedit.exe and write a script like:

# Imports
import vapoursynth as vs
core = vs.get_core()
# Loading Plugins
from vsgan import VSGAN

# Loading F:\TestClips&Co\files/5000frames.mp4 using LWLibavSource
clip = core.lsmas.LWLibavSource(source="F:/TestClips&Co/files/5000frames.mp4", format="YUV420P10", cache=0, prefer_hw=0)
# making sure input color matrix is set as 470bg
clip = core.resize.Point(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)

# creating a VSGAN instance
vsgan = VSGAN("cuda")
# setting model
model = "C:/Users/Selur/Desktop/Vapoursynth64portable/vsgan_models/PSNR Pretrain Models/4xPSNR.pth"
vsgan.load_model(model)
# convert clip color space from YUV420P8 to RGB24 for vsgan
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
clip = vsgan.run(clip=clip)

clip.set_output()
(paths need to be adjusted to your setup)
start the preview.

Cu Selur

[ReinerSchweinlin]

:) I hope I didn´t talk you into too much work :) Really appreciate this addition, thanx !!

My amateur suspection is, that the ...GAN folders should be somehow reduceable in size (since some other similar implementations come in smaller packages), but my skills are far from doing that :)

[Selur]

Yeah, my guess too is that one probably can delete some of the large cuda files inside VapourSynth64\Lib\site-packages\torch\lib since probably not all are needed, but haven't tested and probably won't spend time on it.
If someone has the time and motivation to test which dlls are really needed let me know. :)

Cu Selur

[Selur]

Just wondering: Can the .params&.json files from https://github.com/WolframRhodium/Super-Resolution-Zoo
be used in VSGAN somehow?

Cu Selur

[Zarxrax]

I have installed VSGAN yesterday on windows, and the installation seems to have gone smoothly, but when I try the example script from the documentation, I receive this error:
raise EnvironmentError(f"VSGAN: Either NVIDIA CUDA or the device ({device}) isn't available.")
OSError: VSGAN: Either NVIDIA CUDA or the device (cuda) isn't available.

Any ideas what this error means? My video card is GTX 750 ti, which I know is quite old, but I am able to use other implementations of ESRGAN on it. I'm not sure if this is telling me my video card is not compatible, or something else.

[Selur]

Wild guess would be a driver or rights issue.
Here (https://forum.selur.net/thread-1858.html) is how I set it up.

[ChaosKing]

You can check cuda support with gpuz.

[Zarxrax]

Wild guess would be a driver or rights issue.
Here (https://forum.selur.net/thread-1858.html) is how I set it up.

It was the torch version. I uninstalled torch and then installed an older version using the command you provided, and it works now.

[takla]

Would anyone be so kind to post some benchmark numbers for 1080p to 2160p with one model for anime and one model for real life? Please name your gpu and time in seconds per frame. Thanks.

[Selur]

Can't do 1080p to 2160p with my Geforce GTX 1070ti (8GB VRAM) even when using a the 2x_PSNR model, when using one of the models with a 4x resize factor my guess it that you at least need a 30xx card with 12GB or more VRAM.

[poisondeathray]

Can't do 1080p to 2160p with my Geforce GTX 1070ti (8GB VRAM) even when using a the 2x_PSNR model, when using one of the models with a 4x resize factor my guess it that you at least need a 30xx card with 12GB or more VRAM.

Yes it would help if PRAGMA introduced the tiling script portion that some of the ESRGAN scripts and some GUI's use .

When using decent size margins, you don't see any seams or artifacts (but they are "microscopically" detected with amplified differences)

[Selur]

Yup, some tiling would to keep the memory consumption low would be nice.

[Selur]

Hmm, wrote a small script that splits an image into tileCount*tileCount tiles and applies vsgan on each of them.
import vapoursynth as vs
from vapoursynth import core
from vsgan import VSGAN
import torch

# clip: the video source
# model: path to the model to callable
# tileCount number of times source will be splitt horizontally and vertically
def tileVSGAN(clip, tileCount = 8, model = None) -> vs.VideoNode:
if not isinstance(clip, vs.VideoNode):
raise vs.Error('tileVSGAN: clip must be an instance of vs.VideoNode')
if clip.format.id != vs.RGB24:
raise vs.Error('tileVSGAN: clip must be RGB24')
if model == None:
raise vs.Error('tileVSGAN: model needs to be specified')
if tileCount%2 != 0:
raise vs.Error('tileVSGAN: tileCount needs to dividiable by 2.')
if clip.width%(tileCount*2) != 0:
raise vs.Error('tileVSGAN: clip.width must be dividable by 2*tileCount without rest')
if clip.width%(tileCount*2) != 0:
raise vs.Error('tileVSGAN: clip.height must be dividable by 2*tileCount without rest')

tileWidth = clip.width/tileCount
tileHeight = clip.height/tileCount
tiles = [[None for x in range(tileCount)] for y in range(tileCount)]
for index1 in range (tileCount): # vertical
for index2 in range (tileCount): # horizontal
tile = core.std.CropRel(clip=clip, left=tileWidth*index2, right = tileWidth*(tileCount-index2-1), top=tileHeight*index1, bottom=tileHeight*(tileCount-index1-1))
#tiles[index1][index2] = core.text.Text(clip=tile,text=str(index1)+'/'+str(index2),scale=1)


# apply VSGAN on each tile

for index1 in range (tileCount): # vertical
for index2 in range (tileCount): # horizontal
vsgan = VSGAN("cuda")
vsgan.load_model(model)
tiles[index1][index2] = vsgan.run(clip=tiles[index1][index2])
del vsgan # delete vsgan object
torch.cuda.empty_cache() # trigger pytorch to free gpu memory cache

horizontal = [None for x in range(tileCount)]
for index in range (tileCount):
horizontal[index] = core.std.StackHorizontal(tiles[index])

clip = core.std.StackVertical(horizontal)

return clip

the intention was that running vsgan on lower resolution images should require less vram.
Problem is that is doesn't seem to work, even with tileCount 64 on 4k content all my 8GB VRAM is full. :/

Cu Selur

[_Al_]

unfortunately I do not have VSGAN to test it, but what about this, trying to process one clip at a time, put it more into functions so things get garbage collected:
def clipVSGAN(clip, model):
vsgan = VSGAN("cuda")
vsgan.load_model(model)
new_clip = vsgan.run(clip=clip)
del vsgan # delete vsgan object
torch.cuda.empty_cache() # trigger pytorch to free gpu memory cache
return new_clip

def tileVSGAN(clip, tileCount = 8, model = None) -> vs.VideoNode:
if not isinstance(clip, vs.VideoNode):
raise vs.Error('tileVSGAN: clip must be an instance of vs.VideoNode')
if clip.format.id != vs.RGB24:
raise vs.Error('tileVSGAN: clip must be RGB24')
if model == None:
raise vs.Error('tileVSGAN: model needs to be specified')
if tileCount%2 != 0:
raise vs.Error('tileVSGAN: tileCount needs to dividiable by 2.')
if clip.width%(tileCount*2) != 0:
raise vs.Error('tileVSGAN: clip.width must be dividable by 2*tileCount without rest')
if clip.width%(tileCount*2) != 0:
raise vs.Error('tileVSGAN: clip.height must be dividable by 2*tileCount without rest')

def fetch_new_tile(clip, tileWidth, tileHeight, tileCount):
for index1 in range (tileCount): # vertical
for index2 in range (tileCount): # horizontal
tile = core.std.CropRel(clip = clip,
left = tileWidth*index2,
right = tileWidth*(tileCount-index2-1),
top = tileHeight*index1,
bottom = tileHeight*(tileCount-index1-1)
)
yield clipVSGAN(tile, model)
del tile

tileWidth = clip.width/tileCount
tileHeight = clip.height/tileCount
tiles = [[None for x in range(tileCount)] for y in range(tileCount)]
new_tile_generator = fetch_new_tile(clip, tileWidth, tileHeight, tileCount)

for index1 in range (tileCount):
for index2 in range (tileCount):
tiles[index1][index2] = next(new_tile_generator)

horizontal = [None for x in range(tileCount)]
for index in range (tileCount):
horizontal[index] = core.std.StackHorizontal(tiles[index])

clip = core.std.StackVertical(horizontal)
return clip

[poisondeathray]

There are several tiling scripts with variable input size and padding/margins out there

eg
https://github.com/Oriode/ESRGAN-Tiling-Script/blob/master/upscale.py

They just need to be "massaged" into vapoursynth form .

Several of HolyWu's scripts have tiling/padding too. HolyWu could probably add it easily

[Selur]

@_Al_: doesn't seem to behave differntly
RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 8.00 GiB total capacity; 6.75 GiB already allocated; 0 bytes free; 6.90 GiB reserved in total by PyTorch)

So if someone comes up with a working way to use this please share. :)

Cu Selur

[_Al_]

https://github.com/rlaphoenix/VSGAN/blob/master/vsgan/__init__.py
PRAGMA had implemented that already, his parameter chunk in run()
def run(self, clip: vs.VideoNode, chunk: bool = False) -> vs.VideoNode:
chunk=True would split clip to quadrants

def chunk(self, clip: vs.VideoNode) -> Iterable[vs.VideoNode]:
"""
Split clip down the center into two clips (a left and right clip)
Then split those 2 clips in the center into two clips (a top and bottom clip).
Resulting in a total of 4 clips (aka chunk).
"""
return itertools.chain.from_iterable([
self.split(x, axis=1) for x in self.split(clip, axis=0)
])

that function would need to be changed to return list of tiles

then he is reassembling clips doing this:
# if chunked, rejoin the chunked clips otherwise return the result
clip = core.std.StackHorizontal([
core.std.StackVertical([results[0], results[1]]),
core.std.StackVertical([results[2], results[3]])
]) if chunk else results[0]
that would need to be changed to reassemble it same way as it was chopped off

but there is a concern, because that poissondeathray link workflow uses margins, which they are cut off before assembling tiles, probably if not done, borders could be visible with some artifacts

[poisondeathray]

but there is a concern, because that poissondeathray link workflow uses margins, which they are cut off before assembling tiles, probably if not done, borders could be visible with some artifacts

Yes, you need padding, otherwise you get artifacts. Quality loss is negligible if padding is large enough (but detectable with metrics or amplified differences - you see the split lines, but invisible to human eye)

[PRAGMA]

https://github.com/rlaphoenix/VSGAN/blob/master/vsgan/__init__.py
PRAGMA had implemented that already, his parameter chunk in run()
def run(self, clip: vs.VideoNode, chunk: bool = False) -> vs.VideoNode:
chunk=True would split clip to quadrants

def chunk(self, clip: vs.VideoNode) -> Iterable[vs.VideoNode]:
"""
Split clip down the center into two clips (a left and right clip)
Then split those 2 clips in the center into two clips (a top and bottom clip).
Resulting in a total of 4 clips (aka chunk).
"""
return itertools.chain.from_iterable([
self.split(x, axis=1) for x in self.split(clip, axis=0)
])

that function would need to be changed to return list of tiles

then he is reassembling clips doing this:
# if chunked, rejoin the chunked clips otherwise return the result
clip = core.std.StackHorizontal([
core.std.StackVertical([results[0], results[1]]),
core.std.StackVertical([results[2], results[3]])
]) if chunk else results[0]
that would need to be changed to reassemble it same way as it was chopped off

but there is a concern, because that poissondeathray link workflow uses margins, which they are cut off before assembling tiles, probably if not done, borders could be visible with some artifacts

Hi, yeah, it has support for this but generally speaking I advise people not to use it because the current way it works isnt as unnoticeable as other solutions can return.

I could probably work on adding some black padding fairly easily to each one, and then crop if that works well enough. I see selur made an issue post asking for it, so if I ever work on it I will push updates there. https://github.com/rlaphoenix/VSGAN/issues/9

[PRAGMA]

I updated the Installation docs page to add some small information about how to use python/pip on a portable installation. Some users here struggled with that.
I also created a new troubleshooting page with some solutions or advice to problems posted in pages 2 and 3.

[zorr]

I believe padding with black is not going to help, rather it needs the surrounding image data. Kind of like MVTools overlap.

[Selur]

New problem,...

I updated VSGAN:

I:\Hybrid\64bit\Vapoursynth>python -m pip install --upgrade vsgan
Requirement already satisfied: vsgan in i:\hybrid\64bit\vapoursynth\lib\site-packages (1.2.1)
Collecting vsgan
Downloading vsgan-1.3.0-py3-none-any.whl (10 kB)
Collecting torch<2.0.0,>=1.9.1
Using cached torch-1.9.1-cp39-cp39-win_amd64.whl (222.0 MB)
Collecting numpy==1.19.5
Downloading numpy-1.19.5-cp39-cp39-win_amd64.whl (13.3 MB)
|████████████████████████████████| 13.3 MB 2.2 MB/s
Requirement already satisfied: typing-extensions in i:\hybrid\64bit\vapoursynth\lib\site-packages (from torch<2.0.0,>=1.9.1->vsgan) (3.10.0.2)
Installing collected packages: torch, numpy, vsgan
Attempting uninstall: torch
Found existing installation: torch 1.9.0+cu111
Uninstalling torch-1.9.0+cu111:
Successfully uninstalled torch-1.9.0+cu111
WARNING: The scripts convert-caffe2-to-onnx.exe and convert-onnx-to-caffe2.exe are installed in 'I:\Hybrid\64bit\Vapoursynth\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Attempting uninstall: numpy
Found existing installation: numpy 1.21.2
Uninstalling numpy-1.21.2:
Successfully uninstalled numpy-1.21.2
WARNING: The script f2py.exe is installed in 'I:\Hybrid\64bit\Vapoursynth\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Attempting uninstall: vsgan
Found existing installation: vsgan 1.2.1
Uninstalling vsgan-1.2.1:
Successfully uninstalled vsgan-1.2.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.10.0+cu111 requires torch==1.9.0, but you have torch 1.9.1 which is incompatible.
torchaudio 0.9.0 requires torch==1.9.0, but you have torch 1.9.1 which is incompatible.
Successfully installed numpy-1.19.5 torch-1.9.1 vsgan-1.3.0

I:\Hybrid\64bit\Vapoursynth>python -m pip install --upgrade vsgan
Requirement already satisfied: vsgan in i:\hybrid\64bit\vapoursynth\lib\site-packages (1.3.0)
Requirement already satisfied: torch<2.0.0,>=1.9.1 in i:\hybrid\64bit\vapoursynth\lib\site-packages (from vsgan) (1.9.1)
Requirement already satisfied: numpy==1.19.5 in i:\hybrid\64bit\vapoursynth\lib\site-packages (from vsgan) (1.19.5)
Requirement already satisfied: typing-extensions in i:\hybrid\64bit\vapoursynth\lib\site-packages (from torch<2.0.0,>=1.9.1->vsgan) (3.10.0.2)

and now I get:
Failed to evaluate the script:
Python exception: VSGAN: Either NVIDIA CUDA or the device (cuda) isn't available.

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 2324, in vapoursynth.vpy_evaluateScript
File "src\cython\vapoursynth.pyx", line 2325, in vapoursynth.vpy_evaluateScript
File "C:\Users\Selur\Desktop\test2.vpy", line 29, in <module>
vsgan = VSGAN("cuda")
File "I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsgan\__init__.py", line 34, in __init__
raise EnvironmentError("VSGAN: Either NVIDIA CUDA or the device (%s) isn't available." % device)
OSError: VSGAN: Either NVIDIA CUDA or the device (cuda) isn't available.

when calling:
# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# Import scripts
import mvsfunc
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
from vsgan import VSGAN
# adjusting color space from YUV420P8 to RGB24 for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
# resizing using VSGAN
vsgan = VSGAN("cuda")
model = "I:/Hybrid/64bit/vsgan_models/4x_BSRGAN.pth"
vsgan.load_model(model)
clip = vsgan.run(clip=clip) # 2560x1408
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1056, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output(

Any idea what I can do how to fix this?

Cu Selur

[Selur]

Okay, fixed it by calling:
python.exe -m pip install torch===1.9.1+cu111 torchvision===0.10.1 -f https://download.pytorch.org/whl/torch_stable.html

Cu Selur

[PRAGMA]

Okay, fixed it by calling:
python.exe -m pip install torch===1.9.1+cu111 torchvision===0.10.1 -f https://download.pytorch.org/whl/torch_stable.html

Cu Selur

Yes it's a common mistake to let the vsgan deps install torch (which does not include cuda bundled in). However, it would be a mistake to not count pytorch as a dependency.

PyTorch honestly isnt normally meant to have cuda bundled, its just a possible installation method. By default/normally, it expects it installed on the system globally of course, which is why this deps installation *can* work, but wont for a portable install or those who dont want to manually install cuda.

Regardless, I'm glad you have it solved. To reiterate, a new troubleshooting page exists which has this issue listed, if you encounter further issues perhaps refer to it: https://rlaphoenix.github.io/VSGAN/troubleshooting/

[PRAGMA]

VSGAN 1.4.0 has padded/overlapping chunking support. However, it can now only chunk to 4 quadrants only. Perhaps in the future I will implement the ability to chunk each quadrant as well.

[PRAGMA]

Chaining models is broken in v1.4.0 due to a FrameEval mistake. Please install from source code or wait for the next version which will be soon hopefully.

Edit: Fixed in 1.4.1

[PRAGMA]

RealESRGAN support has been added. It's code is NOT great, but it works. Its basically using new-arch of normal ESRGAN (which is restrictive and annoying) but is necessary for the new models RealESRGAN uses.

It will be in the next release after 1.4.1.

EDIT: Full Real-ESRGAN support, including 2x and 1x scale models is added to code, and it no longer uses new-arch, so it works a lot better now.
It will be in v1.5.0.

[poisondeathray]

Thanks for the updates, especially overlap chunking

[poisondeathray]

Thanks also for A-ESRGAN

https://github.com/aesrgan/A-ESRGAN


v1.5.0 Latest

Adds support for ESRGAN+ models, Real-ESRGAN models (including 2x and 1x if pixel-shuffle was used),
and A-ESRGAN models

[PRAGMA]

Ok so more updates.

Support for half-accuracy was added to all models. This means you can reduce accuracy, which reduces VRAM in ultimately half, which also increases speed by up to double in some cases. It will decrease the accuracy of course, but yeah. For those with very limited VRAM, half accuracy + seamless chunk mode may help quite a bit. This half mode will be applied when you use load_model with `half=True` and will be added to the next version (probably 1.5.1).

Support for ESRGAN+, Real-ESRGAN, and A-ESRGAN are now supported in v1.5.0. With Real-ESRGAN v2 (basically ultra-lite Real-ESRGAN) coming in the next version.

What may appear in next version, if not then the version after is support for EGVSR, which is an inter-frame architecture taking in data from n neighboring frames. It's still in testing and still being worked on but its going well. It is at a state where basic models work, but detection of arch settings has not been worked on yet.

Once auto-detection of arch settings can be done, support will be added in whatever the next version from then is.

[Selur]

Nice! Looking forward to it. :)

[ReinerSchweinlin]

nice :)

From my experience with other FP16 Models, Quality still is fine in most cases - and the speedup (on FP16 2:1 vapable cards like recent AMD, GEN20 RTX..) as well as the halved VRAM needed is a very good thing - rarely turning back to FP32 models these days.

[PRAGMA]

Ok v1.6.0 has been released, with a fairly big change to how VSGAN is ultimately used. This update includes the previously talked about features like half-precision, Real-ESRGANv2, EGVSR, and more.
There's also an upgraded tiling system that will automatically enable or disable depending on if you need the VRAM or not. It will also recursively tile to reduce VRAM even further (if necessary of course).

Please note though, that EGVSR does not auto-detect settings from the model file yet. So you should manually override any settings that needs it when you load the model for correct results.

For more details see the changelog: https://vsgan.phoeniix.dev/en/stable/changelog.html

[poisondeathray]

Thanks for adding EGVSR.




Please note though, that EGVSR does not auto-detect settings from the model file yet. So you should manually override any settings that needs it when you load the model for correct results.


Can you explain how to do that ?

I'm getting bizarre results with the "EGVSR_iter420000.pth" model - it looks like some fine grid pattern overlaid

[PRAGMA]

Thanks for adding EGVSR.




Can you explain how to do that ?

I'm getting bizarre results with the "EGVSR_iter420000.pth" model - it looks like some fine grid pattern overlaid

Refer to https://github.com/rlaphoenix/VSGAN/issues/18
maybe this issue was made by you? not sure.

[poisondeathray]

Refer to https://github.com/rlaphoenix/VSGAN/issues/18
maybe this issue was made by you? not sure.

Not me, but it's a similar overlay grid pattern

(That guy should IVTC first)

Adding nb=10, degradation="BD" fixes it, thanks


load(r'PATH\EGVSR_iter420000.pth', nb=10, degradation="BD")

[Selur]

Small question, so when using a ESRGAN,ESRGAN+,Real-ESRGAN or A-ESRGAN model I use
from vsgan import ESRGAN
and when using a EGVSR model I use:
from vsgan import EGVSR
correct?

Also: Does anyone know a source for EGVSR models?
(only one I found so far is from https://github.com/Thmen/EGVSR/tree/master/pretrained_models)

[poisondeathray]

Small question, so when using a ESRGAN,ESRGAN+,Real-ESRGAN or A-ESRGAN model I use
from vsgan import ESRGAN
and when using a EGVSR model I use:
from vsgan import EGVSR
correct?



Works for me


Also: Does anyone know a source for EGVSR models?

AFAIK there is only 1 model (from the official repo)

[Selur]

Ah, okay.
Things seem to work fine with ESRGAN models (like BSRGAN), but using:
# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# Import scripts
import mvsfunc
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting color space from YUV420P8 to RGB24 for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
# resizing using VSGAN
from vsgan import EGVSR
vsgan = EGVSR(clip=clip,device="cuda")
model = "I:/Hybrid/64bit/vsgan_models/4x_iter420000_EGVSR.pth"
vsgan.load(model, nb=10, degradation="BD", out_nc=3, nf=64)
vsgan.apply() # 2560x1408
clip = vsgan.clip
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1056, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()
I run out of VRAM :(
CUDA out of memory. Tried to allocate 56.00 MiB (GPU 0; 8.00 GiB total capacity; 6.62 GiB already allocated; 0 bytes free; 6.76 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Sadly 'overlap' seems to only be implemented for ESRGAN, since when using "vsgan.apply(overlap=16)" in the above example, I get:
Python exception: apply() got an unexpected keyword argument 'overlap'


Cu Selur

[poisondeathray]

It looks like tiling is only implemented in ESRGAN + derivatives, but not EGVSR yet

[Selur]

Okay, also got another issue with the current version.
Using:
# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# Import scripts
import mvsfunc
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting color space from YUV420P8 to RGB24 for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "I:/Hybrid/64bit/vsgan_models/4x_BSRGAN.pth"
vsgan.load(model)
vsgan.apply() # 2560x1408
clip = vsgan.clip
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=640, h=352, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()
When I open this, it takes a bit to calculate the first frame during which VRAM usage goes up to the max (8GB) and doesn't go down again and hinders me from looking at another frame. :/
when using "vsgan.apply(overlap=16)" the same happens.
-> at least here the new version does only work for one frame.
I also tried different other models, all seem to have the same effect on my system. :(
I sometimes I can step through a few frames (two work, third always fails), but then vram gets stuck and thats it.

[poisondeathray]

Selur are you running 1.6.3 ? Maybe some memory release issue - but it seems to work ok for me using ESRGAN (or derivatives). Multiple seeks ok on a resolution that definitely uses tiles

[Selur]

yes, I'm using VSGAN 1.6.3 (using NVIDIA Game-reay driver version 511.23), after a system restart I can do a few more seeks inside the source, but after a bit vram stays at max until preview crashs. :/

[PRAGMA]

Ah, okay.
Things seem to work fine with ESRGAN models (like BSRGAN), but using:
# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# Import scripts
import mvsfunc
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting color space from YUV420P8 to RGB24 for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
# resizing using VSGAN
from vsgan import EGVSR
vsgan = EGVSR(clip=clip,device="cuda")
model = "I:/Hybrid/64bit/vsgan_models/4x_iter420000_EGVSR.pth"
vsgan.load(model, nb=10, degradation="BD", out_nc=3, nf=64)
vsgan.apply() # 2560x1408
clip = vsgan.clip
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1056, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()
I run out of VRAM :(
CUDA out of memory. Tried to allocate 56.00 MiB (GPU 0; 8.00 GiB total capacity; 6.62 GiB already allocated; 0 bytes free; 6.76 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Sadly 'overlap' seems to only be implemented for ESRGAN, since when using "vsgan.apply(overlap=16)" in the above example, I get:
Python exception: apply() got an unexpected keyword argument 'overlap'


Cu Selur

Hi, There are some problems I've discovered with EGVSR atm. It will take further research and testing than initially expected.

Currently, half accuracy for EGVSR does not work correctly, even with fixes on master atm.

It's also somewhat VRAM intensive due to it essentially storing and running multiple frames (6 in total by default) at a time.

One mistake people are doing is letting VS use the default multi-threading with EGVSR when it should be disabled with `core.num_threads = 1`. Once you do this, it will only run the model on the current frame + n(interval) next frames at a time, instead of e.g. 72 frames with a num_threads of 12.

The fixes I'm speaking of right now are in the GitHub repo, but not in a version yet. You could install it straight from the GitHub master if you want to give it a quick test.

I'm still working on trying to get half-accuracy properly working for EGVSR, and still trying to work on methods to reduce VRAM but sadly it's just not going all that well. It might simply just take a lot of VRAM considering the number of frames the network processes at once.

And as for overlap, yes, it's not implemented in EGVSR at the moment, but perhaps that's something we could try one day to lower VRAM requirements.

[Selur]

sadly using 'core.num_threads = 1' doesn't help here.
Using:
# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Limit thread count to 1
core.num_threads = 1
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# Import scripts
import mvsfunc
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting color space from YUV420P8 to RGB24 for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
# resizing using VSGAN
from vsgan import EGVSR
vsgan = EGVSR(clip=clip,device="cuda")
model = "I:/Hybrid/64bit/vsgan_models/4x_iter420000_EGVSR.pth"
# using model parameters from 4x_iter420000_EGVSR.defaults
vsgan.load(model, nb=10, degradation="BD", out_nc=3, nf=64)
vsgan.apply() # 2560x1408
clip = vsgan.clip
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1056, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()
VRAM usage directly jumps to max usage reports:
CUDA out of memory. Tried to allocate 56.00 MiB (GPU 0; 8.00 GiB total capacity; 6.62 GiB already allocated; 0 bytes free; 6.76 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
and is stuck at max vram usage until I close the viewer.
Similar situation when using "core.num_threads = 1" with:
vsgan = ESRGAN(clip=clip,device="cuda")
model = "I:/Hybrid/64bit/vsgan_models/4x_BSRGAN.pth"
vsgan.load(model)
vsgan.apply(overlap=16) # 2560x1408
First frame works, vram usage is at max and stays there.
Old version 1.5 version worked fine with the same source and model.

trying to go back to 1.5.0 with
I:\Hybrid\64bit\Vapoursynth>python -m pip install --user --force VSGAN==1.5.0
I get:

Collecting VSGAN==1.5.0
Using cached vsgan-1.5.0-py3-none-any.whl (11 kB)
Collecting numpy<2.0.0,>=1.19.5
Using cached numpy-1.22.1-cp39-cp39-win_amd64.whl (14.7 MB)
Installing collected packages: numpy, VSGAN
ERROR: Exception:
Traceback (most recent call last):
File "I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\pip\_internal\cli\base_command.py", line 164, in exc_logging_wrapper
status = run_func(*args)
File "I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\pip\_internal\cli\req_command.py", line 205, in wrapper
return func(self, options, args)
File "I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\pip\_internal\commands\install.py", line 404, in run
installed = install_given_reqs(
File "I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\pip\_internal\req\__init__.py", line 73, in install_given_reqs
requirement.install(
File "I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\pip\_internal\req\req_install.py", line 765, in install
scheme = get_scheme(
File "I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\pip\_internal\locations\__init__.py", line 208, in get_scheme
old = _distutils.get_scheme(
File "I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\pip\_internal\locations\_distutils.py", line 130, in get_scheme
scheme = distutils_scheme(dist_name, user, home, root, isolated, prefix)
File "I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\pip\_internal\locations\_distutils.py", line 69, in distutils_scheme
i.finalize_options()
File "distutils\command\install.py", line 274, in finalize_options
File "distutils\command\install.py", line 437, in finalize_other
distutils.errors.DistutilsPlatformError: User base directory is not specified

probably will have to uinstall and reinstall VSGAN to swithc back to 1.5.0, but I'll stick to current 1.6.3 and see whether someone else also has the same problem. :)
Okay,scratch that, I made a backup of my Vapoursynth folder before updating to 1.6.3, using 1.5.0 memory usage with version 1.5.0 and:
# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# Import scripts
import mvsfunc
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
from vsgan import VSGAN
# adjusting color space from YUV420P8 to RGB24 for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
# resizing using VSGAN
vsgan = VSGAN(clip=clip,device="cuda")
model = "I:/Hybrid/64bit/vsgan_models/4x_BSRGAN.pth"
vsgan.load_model(model)
vsgan.run(overlap=16) # 2560x1408
clip = vsgan.clip
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1056, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()
stays at 3.1GB and while the preview is open and I can again switch between frames. (I kept the 1.6.3 folder so I can switch back easily and test stuff if needed.)

Cu Selur

[PRAGMA]

Okay, also got another issue with the current version.
Using:
# Imports
import os
import sys
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'I:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# Import scripts
import mvsfunc
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# adjusting color space from YUV420P8 to RGB24 for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")
# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "I:/Hybrid/64bit/vsgan_models/4x_BSRGAN.pth"
vsgan.load(model)
vsgan.apply() # 2560x1408
clip = vsgan.clip
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=640, h=352, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()
When I open this, it takes a bit to calculate the first frame during which VRAM usage goes up to the max (8GB) and doesn't go down again and hinders me from looking at another frame. :/
when using "vsgan.apply(overlap=16)" the same happens.
-> at least here the new version does only work for one frame.
I also tried different other models, all seem to have the same effect on my system. :(
I sometimes I can step through a few frames (two work, third always fails), but then vram gets stuck and thats it.

Hi, so, the VRAM requirements are generally high but you shouldn't have this much issues using it on a <480p 4x model scenario. When you say you get stuck or cant render a frame, what exactly is the block? The VRAM seems to stay in use until you try render a new frame, the VRAM seems to clear so fast and then get re-used incredibly quickly.

Have you tried using vs-pipe instead of through vs-edit just to see what performance you actually get from as minimal going on as possible? (`vspipe script.vpy . -p`).

Perhaps try close all VS-edit instances, check the GPU usage has gone down (it will still have a bit loaded by pytorch from some reason), and then reopen and try? Sometimes randomly seeking and clicking play a few times just uses too much VRAM before it clears, hence crashes. But if you seek around without clicking play more than a few times it shouldnt give you troubles. Clicking F5/Preview after a fair while also gives troubles, though these are vs-edit issues to do with queue caches or something.

[PRAGMA]

Update: v1.6.4 fixes a memory leak that happens because I have no fucking idea, but alas, if anyone had any VRAM issues please try on v1.6.4.
Some other changes are there as well, mainly changes to how half=True param works. Spoiler: It doesnt exist, but dont worry see changelog.

[Selur]

v1.6.4 works again for me :) Thanks!

[mastrboy]

v1.6.4 works again for me :) Thanks!

I get insta crashes on 1.6.4, 1.6.0-1.6.3 works for some frames before it crash...
The only "stable" version for me seems to be 1.5.0.

Testing with model:RealESRGAN_x2plus.pth

Is there any debug logs I can activate to figure out the issue?

Hardware:
Nvidia RTX3090 (tried both 4xx and 5xx series drivers)
AMD 3900x
32GB Ram

[Selur]

What does your script look like?
What resolution is your input?
You might simply be running out of VRAM.

[mastrboy]

What does your script look like?
What resolution is your input?
You might simply be running out of VRAM.

Source resolution I'm testing on is 720x540, script:

chroma = video.resize.Spline36(video.width*2,video.height*2)
video = video.fmtc.resample (css="444")
video = video.fmtc.matrix (mat="709", col_fam=vs.RGB)
vsgan = VSGAN(video, device="cuda")
vsgan.load_model(r"RealESRGAN_x2plus.pth")
vsgan.run()
video = vsgan.clip
video = video.fmtc.matrix (mat="709", col_fam=vs.YUV, bits=16)
video = video.fmtc.resample (css="420")
video = video.fmtc.bitdepth (bits=8)
video= core.std.Merge(clipa=video, clipb=chroma, weight=[0, 1])
video.set_output()


It runs fine with 1.5.0, during an encode HWinfo64 reports ~3,5-4GB VRAM allocated

[Selur]

Yeah, that will not work with current VSGAN as the syntax changed.
Instead of:
vsgan = VSGAN(video, device="cuda")
vsgan.load_model(r"RealESRGAN_x2plus.pth")
vsgan.run()
video = vsgan.clip

you would call:

vsgan = ESRGAN(clip=video ,device="cuda")
vsgan.load(r"RealESRGAN_x2plus.pth") # load() instead of load_model()
vsgan.apply() # <- not run()
video = vsgan.clip

see: https://vsgan.phoeniix.dev/en/stable/getting-started.html and the changelog of v1.6.0

Cu Selur

[mastrboy]

Yeah, that will not work with current VSGAN as the syntax changed.
see: https://vsgan.phoeniix.dev/en/stable/getting-started.html and the changelog of v1.6.0

Cu Selur

Thanks for the help, I'll give it another try with the new syntax.

[mastrboy]

Is it normal for Linux to be almost 2x faster than Windows when using the exact same vapoursynth script with VSGAN?

Windows: 3.98fps
Linux: 7.67fps

Or did I do something wrong with the enviroment setup for pytorch/vapoursynth/vsgan?

[ReinerSchweinlin]

A speed difference of this magnitude is a sign of different configurations... Maybe, if you are running NVIDIA Cards, setting the card to compute mode in windows will help?

[knumag]

Hi.
Im using a 1060 6GB and only getting around 0.2 fps with "2x_VHS-upscale-and-denoise_Film_477000_G.pth" from https://upscale.wiki/wiki/Model_Database
Also getting this warning in cmd "vsgan\utilities.py:36: UserWarning: The given buffer is not writable, and PyTorch does not support non-writable tensors. This means you can write to the underlying (supposedly non-writable) buffer using the tensor. You may want to copy the buffer to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\utils\tensor_new.cpp:998.)
torch.frombuffer("
Is that speed correct with 1060 6GB?
When looking at task manager the gpu jumps to 20% every 2-3 seconds, but idle between.
See that my dedicated memory is almost full. Is that the reason for it being that slow?
Do i need another card with more vram?
I have a threadripper 1950x, but no way to use that instead of GPU?

With "RealESRGAN_x2plus.pth" I'm getting 0.5 fps

[mastrboy]

A speed difference of this magnitude is a sign of different configurations... Maybe, if you are running NVIDIA Cards, setting the card to compute mode in windows will help?

It's an Nvidia RTX 3090, I can't find that option in the control panel, is it only available on Quadro cards?

I can see a compute mode with the CLI though, it's currently configured to Default:
nvidia-smi -q | Select-String -Pattern "compute"
Compute Mode : Default

[Selur]

What does the script look like? What is the resolution of the source?

[knumag]

Pal SD from vhs after qtgmc.
But should be something wrong somewhere, I'm just getting black video as ouput... :D
import vapoursynth as vs
core = vs.core
core.num_threads = 8
core.max_cache_size = 6000
video = core.lsmas.LWLibavSource(source=r"1.mp4")
from vsgan import ESRGAN
video = core.fmtc.resample (clip=video, css="444")
video = core.fmtc.matrix (clip=video, mat="709", col_fam=vs.RGB)
vsgan = ESRGAN(clip=video ,device="cuda")
vsgan.load(r"RealESRGAN_x2plus.pth")
vsgan.apply()
video = vsgan.clip
video = core.fmtc.matrix (clip=video, mat="709", col_fam=vs.YUV, bits=16)
video = core.fmtc.resample (clip=video, css="420")
video = core.fmtc.bitdepth (clip=video, bits=8)
video = core.resize.Spline36(video, 1440, 1080)
video.set_output()

Changed to this and it's working

import vapoursynth as vs
core = vs.core
core.num_threads = 8
core.max_cache_size = 6000
video = core.lsmas.LWLibavSource(source=r"1.mp4")
from vsgan import ESRGAN
video = core.resize.Bicubic(clip=video, format=vs.RGB24, matrix_in_s="709", range_s="limited")
vsgan = ESRGAN(clip=video ,device="cuda")
vsgan.load(r"RealESRGAN_x2plus.pth")
vsgan.apply()
video = vsgan.clip
video = core.fmtc.matrix (clip=video, mat="709", col_fam=vs.YUV, bits=16)
video = core.fmtc.resample (clip=video, css="420")
video = core.fmtc.bitdepth (clip=video, bits=8)
video = core.resize.Spline36(video, 1440, 1080)
video.set_output()

[Selur]

Script seems fine to me.
(as a side note: using https://github.com/HolyWu/vs-realesrgan/tree/master/vsrealesrgan is nearly 2 times faster than vsgan with RealESRGAN here.)
Do you use the same driver version on Linux and Windows?

[knumag]

Script seems fine to me.
(as a side note: using https://github.com/HolyWu/vs-realesrgan/tree/master/vsrealesrgan is nearly 2 times faster than vsgan with RealESRGAN here.)
Do you use the same driver version on Linux and Windows?

Having problems installing vsrealesrgan.
Followed your guide here: https://forum.selur.net/thread-1858.html
But when installing vsdpir and vsrealesrgan, Im getting errors.

Using cached VapourSynth-58.zip (558 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [15 lines of output]
Traceback (most recent call last):
File "C:\Users\knumag\AppData\Local\Temp\pip-install-2415kpn4\vapoursynth_712c69d39f4a4718a3f6b523a85b39eb\setup.py", line 64, in <module>
dll_path = query(winreg.HKEY_LOCAL_MACHINE, REGISTRY_PATH, REGISTRY_KEY)
File "C:\Users\knumag\AppData\Local\Temp\pip-install-2415kpn4\vapoursynth_712c69d39f4a4718a3f6b523a85b39eb\setup.py", line 38, in query
reg_key = winreg.OpenKey(hkey, path, 0, winreg.KEY_READ)
FileNotFoundError: [WinError 2] The system cannot find the file specified

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "C:\Users\knumag\AppData\Local\Temp\pip-install-2415kpn4\vapoursynth_712c69d39f4a4718a3f6b523a85b39eb\setup.py", line 67, in <module>
raise OSError("Couldn't detect vapoursynth installation path")
OSError: Couldn't detect vapoursynth installation path
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Any suggestions?

[Selur]

a. That guide, won't work with current vsrealesrgan.
b. Assuming you use Vapoursynth R57 and not the new Vapoursynth R58 the steps I posted over at https://forum.doom9.org/showthread.php?t=184000 should work.
Using VapourSynth R58 would require to use Python 3.8 (not 3.9 or 3.10) since R58 only supports 3.8 and 3.10, but onnxruntime does not support 3.10 atm.

Cu Selur

[knumag]

I got it working now, but why does VSGAN seem better when looking at the result? Same model is being used, but it still seem sharper.

[knumag]

And also, using the latest vsrealesrgan with onnx, R58 and pyton 3.8, I got 90-100% GPU usage with same fps as i got with the older vsrealesrgan 1.2.0 at 10% GPU which I did get to work with R57 and python 3.9.
Very confusing. CPU might av been a bit higher without ONNX, but that different is weird, when getting same FPS, no?

[knumag]

a. That guide, won't work with current vsrealesrgan.
b. Assuming you use Vapoursynth R57 and not the new Vapoursynth R58 the steps I posted over at https://forum.doom9.org/showthread.php?t=184000 should work.
Using VapourSynth R58 would require to use Python 3.8 (not 3.9 or 3.10) since R58 only supports 3.8 and 3.10, but onnxruntime does not support 3.10 atm.

Cu Selur

I tried that guide also, didnt work for me. Had to install earlier versions of a lot of them to get it to work for some reason.
Been at this now for 10-15 hours.. :D

[lansing]

What are the required GPU ram to run this? I'm upscaling 2x on a 1080p image and it's eating up all my 6G of ram.

[Selur]

@lansing: going for 4k from 1080p with an 2x_-model with 16bit indicates a VRAM usage of 6.3GB on my card (Geforce rtx 4080), so you might be out of luck. You could try setting an overlap value, maybe that triggers the tiling support of VSGAN and help with the ram shortage.

[kedautinh12]

Lol, 4080, a rich man :D

[Selur]

Not any more. :( (new card + new power supply + new ups)
After 5 years my old 1070 simply was having more and more issues, since NVIDIA messed up the driver support more and more. (Accessing my 5k display which needs dual display port import wasn't possible with any drivers newer than the drivers from May.)

[lansing]

@lansing: going for 4k from 1080p with an 2x_-model with 16bit indicates a VRAM usage of 6.3GB on my card (Geforce rtx 4080), so you might be out of luck. You could try setting an overlap value, maybe that triggers the tiling support of VSGAN and help with the ram shortage.

Is the plugin Nvidia card only? AMD card offers so much more ram for lesser price.

[kedautinh12]

But don't support CUDA if VSGAN will support CUDA :D

[Julek]

Is the plugin Nvidia card only? AMD card offers so much more ram for lesser price.

You can use vsncnn with vs-mlrt (https://github.com/AmusementClub/vs-mlrt#vsncnn-ncnn-based-gpu-vulkan-runtime) to run on AMD, here is the list of models (https://github.com/AmusementClub/vs-mlrt/wiki#models) currently supported by vs-mlrt, for other specific models you need to convert to onnx by yourself.

[lansing]

You can use vsncnn with vs-mlrt (https://github.com/AmusementClub/vs-mlrt#vsncnn-ncnn-based-gpu-vulkan-runtime) to run on AMD, here is the list of models (https://github.com/AmusementClub/vs-mlrt/wiki#models) currently supported by vs-mlrt, for other specific models you need to convert to onnx by yourself.

I'm waiting for the RX 7900 XTX launch next week to upgrade to AMD. But wow this field is so dominated by Nvidia. It's like nobody is developing anything for AMD...

The models I'm interested in are the 2x_LD-Anime_Skr_v1.0 and some anime sharpen AI for old anime. It's turning my anime dvd into blu-ray.

original
https://i.imgur.com/yhPEkaI.png

2x_LD-Anime_Skr_v1.0
https://i.imgur.com/wpSbhLB.png

color matched to cel
https://i.imgur.com/ahY3plj.jpg


I tried to run another 2x sharpen upscale to the result but ran out of memory.

[ChaosKing]

Some test I made, One Piece DVD:
https://cdn.discordapp.com/attachments/426802196194263051/1003626930169454592/op1.png <-- (stronger filtering)

https://cdn.discordapp.com/attachments/426802196194263051/1003622612938801204/unknown.png
https://cdn.discordapp.com/attachments/426802196194263051/1003622370965209128/unknown.png
https://cdn.discordapp.com/attachments/426802196194263051/1003623183980699738/unknown.png

One problem that I often see with these models is that there is often a color shift. It is not very fast, but awesome quality (if the source video "matches")

[kedautinh12]

Do you had channel of discord for encoder?? Can let me in? :D

[Krizzz989]

ChaosKing those look great, are you open to sharing your script for that? I'm interesting in upscaling the same anime but I'm completely lost on where to start.

[lansing]

Some test I made, One Piece DVD:
https://cdn.discordapp.com/attachments/426802196194263051/1003626930169454592/op1.png <-- (stronger filtering)

https://cdn.discordapp.com/attachments/426802196194263051/1003622612938801204/unknown.png
https://cdn.discordapp.com/attachments/426802196194263051/1003622370965209128/unknown.png
https://cdn.discordapp.com/attachments/426802196194263051/1003623183980699738/unknown.png

One problem that I often see with these models is that there is often a color shift. It is not very fast, but awesome quality (if the source video "matches")

I just found the model's author on discord, I'll see if I can get some support from him.



UPDATE:

I got reply from the author. I don't know how to interpret it, so I direct quoted him:
Because LDs have crap colors. I had to color match the entire data set, so in the end, there's still a bit of shift

So the shift is expected since our source is DVD instead of LD.

[lansing]

ChaosKing those look great, are you open to sharing your script for that? I'm interesting in upscaling the same anime but I'm completely lost on where to start.

OP has a installation guide:
https://vsgan.phoeniix.dev/en/stable/installation.html

Total file size would be a few GB.

And then go here to look for a model, there are model for different situations. What we use here is 2x_LD-Anime_Skr_v1.0:
https://upscale.wiki/wiki/Model_Database#Anime


import vapoursynth as vs
from vsgan import ESRGAN

core = vs.core

# your dvd source
clip = core.ffms.Source()

# convert to RGB
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_s="limited")

# instantiate esrgan and load the model
esrgan = ESRGAN(clip, "cuda")

model = r'my_path\2x_LD-Anime_Skr_v1.0.pth'

esrgan.load(model)
esrgan.apply()
clip = esrgan.clip

# get output
clip_set_output()

[ChaosKing]

I don't remember which models I used, I simply tried out different models (and combined some).
You can start with one of these: 2x_AnimeClassics_UltraLite_510K, 2x_LD-Anime_Skr_v1.0, 2x_SHARP_ANIME_V1, 2x_DigitalFlim_SubCompact_nf24-nc8_289k_net_g

I also used the example script from the docs. Nothing special, just patience, oh and save your script from time to time, because your editor will crash!

LD = Laserdisc? My source is a jap. R2 DVD.

[lansing]

After going through many models, for upscaling old anime, I think the models with the best quality are the one that were trained using the actual blu-ray, such as 2x_LD-Anime_Skr_v1.0. There's a guy in the sailormoon forum (https://www.sailormoonforum.com/index.php?threads/sailor-moon-color-correction.33743/page-6#post-975611) that trained his model using blu-ray from the sailor moon movies is also getting amazing result.

[ChaosKing]

I really need to learn how to train models myself. Imagine if you had like 200 CELs to train on.
I read that you need at least 100 different frames to get decent results.

[lansing]

I need to learn how to train models myself. Imagine if you had like 200 CELs to train on.
I read that you need at least 100 different frames to get decent results.

I'm also thinking about training one myself (don't even know how yet). I need a 1x model for sharpening right after 2x_LD-Anime but couldn't find any that were trained from blu-ray.

[Selur]

color matched to cel
https://i.imgur.com/ahY3plj.jpg

@lansing: How did you do the color adjustment?

[lansing]

@lansing: How did you do the color adjustment?

I use 3D LUT Creator

[Selur]

Ah, okay. (I was hoping for some Vapoursynth plugin I wasn't aware of. ;))

[lansing]

Ah, okay. (I was hoping for some Vapoursynth plugin I wasn't aware of. ;))

Vapoursynth is not the right tool for color adjustment

[kedautinh12]

Vapoursynth is not the right tool for color adjustment

Vapoursynth had vscube can load 3D LUTs
https://github.com/sekrit-twc/timecube

[lansing]

Vapoursynth had vscube can load 3D LUTs
https://github.com/sekrit-twc/timecube

What I mean is that we can load the finished product into vs sure but we can't use it for the adjustment process.

[mastrboy]

I really need to learn how to train models myself. Imagine if you had like 200 CELs to train on.
I read that you need at least 100 different frames to get decent results.

Any chance you could write down the process if you figure this out?
My attempts at training my own models has not produced good results at all and there's not a lot of good documentation out there either...

[Selur]

I agree some good documentation would be nice. :)

[lansing]

I want to create images with softer edge from blu-ray sources for training, is there any filter that only blur the line? I tried downsize the original to 50% and resize it back in photoshop, but it add some ringing along the lines.

[Selur]

Multiple solutions:
a. don't use a VSGAN model, but something like:
# denoising using KNLMeansCL
# adjusting color space from RGB24 to YUV444P8 for vsKNLMeans
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P8, matrix_s="470bg", range_s="limited")
clip = core.knlm.KNLMeansCL(clip=clip, d=0, h=10.00, channels="Y")
clip = havsfunc.DeHalo_alpha(clip)
# adjusting color space from YUV444P8 to YUV420P10 for vsHysteria
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")
clip = hysteria.Hysteria(clip=clip)
for the cleaning and simple don't sharpen the lines. (I got rgb24 as source color sampling since I used your image as source)

b. use a filter that does the smoothing and only apply it using an edge mask. :) (see: https://guide.encode.moe/encoding/masking-limiting-etc.html for the general idea)
c. use a dehalo filter :)

Cu Selur

[pandy]

I want to create images with softer edge from blu-ray sources for training, is there any filter that only blur the line? I tried downsize the original to 50% and resize it back in photoshop, but it add some ringing along the lines.

Use classic kernel filter:

0 .5 0
0 1 0
0 .5 0

For resize use bilinear then no ringing should be introduced.

[lansing]

I was testing out this standalone image processing program call chaiNNer for running ESRGAN. Upscaling using the same 640x480 image and a 2x model, during processing it uses 1.8 GB of GPU ram, and it drops to 600 MB when finished. While vapoursynth uses 1.7 GB and never releases it.

[lansing]

I've found the denoising in 2x_LD-Anime_Skr_v1.0 model exceptionally impressive. It is able to smooth out the chroma noise in near black color, making it possible to change color/brightness on those areas without having artifacts. I couldn't even do this in Neat Video.

https://imgsli.com/MTM5ODIz

[Selur]

@lansing: Couldn't you use DPIR or CCD for that, which both should be faster?

[lansing]

@lansing: Couldn't you use DPIR or CCD for that, which both should be faster?

I just tested them, CCD is not doing a thing on these super dark area. DPIR did do some smoothing but not good enough to avoid artifact.

DPIR +50% brightness in dark
https://imgur.com/rM3jYI8

[Selur]

What threshold did you use for CCD? (try 5 or even higher)
You could also use something like, if you mainly want to filter just the dark area.
## Starting applying 'limit' masked filtering for vsCCD
clipMask = clip
clipMask = core.std.BinarizeMask(clipMask, 30)
clipMask = core.std.InvertMask(clipMask)
clipFiltered = clip
# adjusting color space from RGB24 to RGBS for vsCCD
clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.RGBS, range_s="limited")
# chroma denoising using CCD
clipFiltered = core.ccd.CCD(clip=clipFiltered, threshold=50.00)
clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.RGB24, range_s="limited", dither_type="error_diffusion")
clip = core.std.MaskedMerge(clip, clipFiltered, clipMask)
## Finished applying 'limit' masked filtering for vsCCD

With DPIR:
from vsdpir import DPIR
## Starting applying 'limit' masked filtering for vsDPIRDeblock
clipMask = clip
clipMask = core.std.BinarizeMask(clipMask, 30)
clipMask = core.std.InvertMask(clipMask)
clipFiltered = clip
# adjusting color space from RGB24 to RGBS for vsDPIRDeblock
clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.RGBS, range_s="limited")
# deblocking using DPIRDeblock
clipFiltered = DPIR(clip=clipFiltered, strength=150.000, task="deblock", provider=1, device_id=0, dual=True)
clipFiltered = core.resize.Bicubic(clip=clipFiltered, format=vs.RGB24, range_s="limited", dither_type="error_diffusion")
clip = core.std.MaskedMerge(clip, clipFiltered, clipMask)
## Finished applying 'limit' masked filtering for vsDPIRDeblock
(Masked BasicVSR++ should also do the trick, but that might be slower)
Taking the image as source:
## Starting applying 'limit' masked filtering for vsLevels
clipMask = clip
clipMask = core.std.BinarizeMask(clipMask, 30)
clipMask = core.std.InvertMask(clipMask)
clipFiltered = clip
# Color Adjustment using Levels on RGB24 (8 bit)
clipFiltered = core.std.Levels(clip=clipFiltered, min_in=16, max_in=235, min_out=16, max_out=235)
clip = core.std.MaskedMerge(clip, clipFiltered, clipMask)
## Finished applying 'limit' masked filtering for vsLevels
seems also to work,..

[lansing]

I used the CCD from Virtualdub, set it to max 10 and nothing changed on the dark areas, I don't think it's really meant for this.

You methods are simply clipping out the dark area to wipe out the chroma noise with limited range, it has nothing to do with the tested filters.

[Selur]

I used the CCD from Virtualdub, set it to max 10 and nothing changed on the dark areas, I don't think it's really meant for this.
Okay, I thought you were using Vapoursynth.

You methods are simply clipping out the dark area to wipe out the chroma noise with limited range, it has nothing to do with the tested filters.
Yes, that was the last approach, since I was thinking that it would make no sense that the video is 4:4:4 and pc scale. :)

[Selur]

btw. for those filtering animes&cartoons: https://github.com/HolyWu/vs-animesr/ might be interesting :)
(TensorRT + RGBH doesn't seems to be the only thing not working atm.)

[lansing]

btw. for those filtering animes&cartoons: https://github.com/HolyWu/vs-animesr/ might be interesting :)
(TensorRT + RGBH doesn't seems to be the only thing not working atm.)

Where do you unzip the file?

[Selur]

I installed it through pip install -U vsanimesr downloading CUDA-11.7_cuDNN-8.6.0_TensorRT-8.5.2.2_win64.7z extracting the dll into the runtime folder (I add in my scripts) and installing ' tensorrt-8.5.2.2-cp310-none-win_amd64.whl' through python -m pip install tensorrt-8.5.2.2-cp310-none-win_amd64.whl.

[lansing]

Got it working. Its pretrain models didn't look good on my old anime. What source is it supposed to be good at atm?

[Selur]

No clue, what's supposed to be good. But, looking at the examples over at https://github.com/TencentARC/AnimeSR I would say cartoon&animes where mainly compression artifacts are the problem.
So depending on your source, normal denoising&co should be applied and this is mainly to save resize, line darkening and sharpening.

AnimeSR_v1-PaperModel.pth: v1 model, also the paper model. You can use this model for paper results reproducing.
AnimeSR_v2.pth: v2 model. Compare with v1, this version has better naturalness, fewer artifacts, and better texture/background restoration. If you want better results, use this model.
source: https://github.com/TencentARC/AnimeSR#zap-quick-inference
I would say it's mainly suited for simpler animes&cartoon.


Cu Selur

[ChaosKing]

The demo videos seem a bit overfiltered to me. Not bad, but also not that good. :)
But it's good to have options. Could be a good prefilter or you can merge/average it with other filters.

[Selur]

Yup, I too think it's mainly about having options. :)

[lansing]

I tried on their monkey test clip, it pretty much wipes out all small details leaving only the solid lines

[Selur]

Here are some examples of that clip processed with AnimeSR_v2 model:
https://i.ibb.co/z2NPgpj/grafik.png (https://ibb.co/C9WVygN)
https://i.ibb.co/5TvXG0j/grafik.png (https://ibb.co/kK0V57D)
https://i.ibb.co/rZs9ms5/grafik.png (https://ibb.co/Zhm9Vmc)
https://i.ibb.co/5jFcbwc/grafik.png (https://ibb.co/NNSYcbY)
https://i.ibb.co/5n2P2Zd/grafik.png (https://ibb.co/pRnSnqg)
https://i.ibb.co/pZ9d4b6/grafik.png (https://ibb.co/ct7Q2kK)
https://i.ibb.co/FmGvZHX/grafik.png (https://ibb.co/jb7K0Jz)
last image with the intended resolution (x4, not scaled down to source)
https://i.ibb.co/cgNsr4j/grafik.png (https://ibb.co/JBzSsD9)
so everyone can make up their mind.

Cu Selur

[ChaosKing]

Definitely better than the demo video!

[ReinerSchweinlin]

not too bad. Reminds me of results with a clever combination of traditional filters, areas moother, line darkeners, etc.... How is the speed ?

[Selur]

It's relatively fast,...
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import site
import os
import ctypes
# Adding torch dependencies to PATH
path = site.getsitepackages()[0]+'/torch_dependencies/'
ctypes.windll.kernel32.SetDllDirectoryW(path)
path = path.replace('\\', '/')
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]
# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/SourceFilter/DGDecNV/DGDecodeNV.dll")
# source: 'C:\Users\Selur\Desktop\TheMonkeyKing1965.mp4'
# current color space: YUV420P8, bit depth: 8, resolution: 640x480, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading C:\Users\Selur\Desktop\TheMonkeyKing1965.mp4 using DGSource
clip = core.dgdecodenv.DGSource("G:/Temp/mp4_18de9559c23fe1dbb3c03335a2df8593_853323747.dgi")# 25 fps, scanorder: progressive
# Setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=5)
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0)
from vsanimesr import animesr as AnimeSR
# adjusting color space from YUV420P8 to RGBH for VsAnimeSR
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="470bg", range_s="limited")
# resizing using AnimeSR
clip = AnimeSR(clip=clip, model=1, device_index=0, trt=True, trt_cache_path=r"G:\Temp") # 2560x1920
# resizing 2560x1920 to 1920x1440
# adjusting resizing
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, range_s="limited")
clip = core.fmtc.resample(clip=clip, w=1920, h=1440, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGBS to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_s="limited", dither_type="error_diffusion")
# set output frame rate to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()
takes:
Output 2510 frames in 60.94 seconds (41.19 fps)
on a Geforce 4080 (not counting the time it takes to build the temporary tensor model) using vspipe.

Cu Selur

[kedautinh12]

Yes with 4080, very fast :D

[PatchWorKs]

For those who are interested in, we recently found this interesting open software:
enhancr is an elegant and easy to use GUI for Video Frame Interpolation and Video Upscaling which takes advantage of artificial intelligence - built using node.js and Electron. It was created to enhance the user experience for anyone interested in enhancing video footage using artificial intelligence. The GUI was designed to provide a stunning experience powered by state-of-the-art technologies without feeling clunky and outdated like other alternatives.

https://camo.githubusercontent.com/b96d020a8183077b8b7b8c9d0b0cdaf4f103060a7bc379a7a98a185d3916dcd2/68747470733a2f2f6d6166696f736e696b2e6e656564736d656e74616c2e68656c702f31774c474837437845412e706e673f6b65793d756b374d3754616f6c4143697071

It features blazing-fast TensorRT inference by NVIDIA, which can speed up AI processes significantly. Pre-packaged, without the need to install Docker or WSL (Windows Subsystem for Linux) - and NCNN inference by Tencent which is lightweight and runs on NVIDIA, AMD and even Apple Silicon - in contrast to the mammoth of an inference PyTorch is, which only runs on NVIDA GPUs.

Features

Encodes video on the fly and reads frames from source video, without the need of extracting frames or loading into memory
Queue for batch processing
Live Preview of output media
Allows chaining of interpolation, upscaling & restoration
Offers the possibility to trim videos before processing
Can load custom ESRGAN models in onnx format
Has Scene Detection built-in, to skip interpolation on scene change frames & mitigate artifacts
Color Themes for user customization
Discord Rich Presence, to show all your friends progress, current speed & what you're currently enhancing
... and much more


Interpolation

RIFE (NCNN) - megvii-research (https://github.com/megvii-research)/ECCV2022-RIFE (https://github.com/megvii-research/ECCV2022-RIFE) - powered by styler00dollar (https://github.com/styler00dollar)/VapourSynth-RIFE-ncnn-Vulkan (https://github.com/styler00dollar/VapourSynth-RIFE-ncnn-Vulkan)
RIFE (TensorRT) - megvii-research (https://github.com/megvii-research)/ECCV2022-RIFE (https://github.com/megvii-research/ECCV2022-RIFE) - powered by AmusementClub (https://github.com/AmusementClub)/vs-mlrt (https://github.com/AmusementClub/vs-mlrt) & styler00dollar (https://github.com/styler00dollar)/VSGAN-tensorrt-docker (https://github.com/styler00dollar/VSGAN-tensorrt-docker)
CAIN (NCNN) - myungsub (https://github.com/myungsub)/CAIN (https://github.com/myungsub/CAIN) - powered by mafiosnik (https://github.com/mafiosnik777)/vsynth-cain-ncnn-vulkan (unreleased)
CAIN (TensorRT) - myungsub (https://github.com/myungsub)/CAIN (https://github.com/myungsub/CAIN) - powered by HubertSotnowski (https://github.com/HubertSotnowski)/cain-TensorRT (https://github.com/HubertSotnowski/cain-TensorRT)

Thanks to HubertSotnowski (https://github.com/HubertSotnowski) and styler00dollar (https://github.com/styler00dollar) for helping out with implementing CAIN.

Upscaling

waifu2x (NCNN) - nagadomi (https://github.com/nagadomi)/waifu2x (https://github.com/nagadomi/waifu2x) - powered by nlzy (https://github.com/nlzy)/vapoursynth-waifu2x-ncnn-vulkan (https://github.com/nlzy/vapoursynth-waifu2x-ncnn-vulkan)
RealESRGAN (NCNN) - xinntao (https://github.com/xinntao)/Real-ESRGAN (https://github.com/xinntao/Real-ESRGAN) - powered by AmusementClub (https://github.com/AmusementClub)/vs-mlrt (https://github.com/AmusementClub/vs-mlrt)
RealESRGAN (TensorRT) - xinntao (https://github.com/xinntao)/Real-ESRGAN (https://github.com/xinntao/Real-ESRGAN) - powered by AmusementClub (https://github.com/AmusementClub)/vs-mlrt (https://github.com/AmusementClub/vs-mlrt)

Thanks to HubertSotnowski (https://github.com/HubertSotnowski) for helping out with implementing AnimeSR

Restoration

DPIR (TensorRT) - cszn (https://github.com/cszn)/DPIR (https://github.com/cszn/DPIR) - powered by AmusementClub (https://github.com/AmusementClub)/vs-mlrt (https://github.com/AmusementClub/vs-mlrt)


Enjoy: https://github.com/mafiosnik777/enhancr

[kedautinh12]

But, i can't make this app work. Binary here
https://github.com/AmusementClub/enhancr

[ReinerSchweinlin]

Output 2510 frames in 60.94 seconds (41.19 fps)
on a Geforce 4080 (not counting the time it takes to build the temporary tensor model) using vspipe.

Cu Selur
Nice, thanx :)

[lansing]

I set up the Real-ESRGAN trainer to test it out training a model. I only have 11 image pairs and the estimate time is 2 days 6 hours wow, this is discouraging.

[Selur]

out of curiosity: is this mainly eating gpu or cpu resources?

[ChaosKing]

I set up the Real-ESRGAN trainer to test it out training a model. I only have 11 image pairs and the estimate time is 2 days 6 hours wow, this is discouraging.

On some git repo I read that someone trained a model for half a year xD

[lansing]

out of curiosity: is this mainly eating gpu or cpu resources?
GPU I think. I have a Ryzen 7950x and it barely ran.

[ChaosKing]

New toy https://github.com/HolyWu/vs-scunet
Online demo https://replicate.com/cszn/scunet

So far I like it. It creates very consistent and clean results (a bit too clean).

[kedautinh12]

Too clean and after use it, i will add some noise to match my favourite :D

[ChaosKing]

You could merge some % of the original clip back to add back some texture.
I just like the overall consistency of the denoising result.

examples of how clean it gets:
https://imgur.com/a/UOjtk8Y
https://imgur.com/a/HB02pCW
https://imgur.com/a/6mQuyIn

[lansing]

New toy https://github.com/HolyWu/vs-scunet
Online demo https://replicate.com/cszn/scunet

So far I like it. It creates very consistent and clean results (a bit too clean).

Tested on my anime, the denoising is pretty good. There are some aggressive sharpening when it determined something is an blurred edges. The noise sensitivity is also too high, in my anime here it mistaken the pattern on the wall to be noise.

https://imgsli.com/MTQ0MTkz

[zorr]

New toy https://github.com/HolyWu/vs-scunet
Online demo https://replicate.com/cszn/scunet

So far I like it. It creates very consistent and clean results (a bit too clean).

This looks like an excellent prefilter to MVTools, especially for very noisy videos.

I tried to install this to my portable VapourSynth but it looks like vsscunet wants at least Python 3.10 and the latest FatPack is still at 3.9. Are you planning a new release?

[poisondeathray]

scunet is good on gaussian type noise (the pretrained models were trained on synthetic gaussian noise)

non anime example
https://imgsli.com/MTQ0MTk3/0/0

It would be nice if the psnr model had adjustable strength built in.

The gan model retains a bit more detail compared to psnr model; the gan model exhibits less denoising but shifts the colors a bit more

[ChaosKing]

This looks like an excellent prefilter to MVTools, especially for very noisy videos.

I was thinking the same :D

I tried to install this to my portable VapourSynth but it looks like vsscunet wants at least Python 3.10 and the latest FatPack is still at 3.9. Are you planning a new release?

I was actually planning to release a GPU Edition but things got in the way and some laziness. I hope this month :)



The gan model retains a bit more detail compared to psnr model; the gan model exhibits less denoising but shifts the colors a bit more
yeah, color shifts are the most annoying thing with these trained filters.

[lollo2]

I used one of my S-VHS capture (affected by ghosting and halo) with typical noise and the result is very good: deep denoise, but not too much loss of details.
However, I still prefere a classic AviSynth TemporalDegrain2 + LSFmod flow.

Comparison between original and AviSynth: https://imgsli.com/MTI5NjI2

Comparison between original and Scunet: https://imgsli.com/MTQ0MTgw

Comparison between AviSynth and Scunet: https://imgsli.com/MTQ0MTgx

I will try to play with the Scunet options to match AviSynth more "natural" look.

The processing time is very high, >1 minute for a single frame, I hope it will not be like that on a local machine :eek:

[Selur]

Nice.
I agree, looks nice, but it would be nice to have some more control over the denoising strength.
(dot crawlers, light blocking get removed nicely, but general denoising and smoothing seems too strong, probably best applied with some masking)

[zorr]

I agree, looks nice, but it would be nice to have some more control over the denoising strength.

The strength can be somewhat controlled with a couple of tricks. The first one is to amplify the dark areas with gamma (in RGB space) before the denoising and then undoing the gamma change afterwards. It makes the noise look more like actual features so the denoiser leaves more of it intact. The effect is quite subtle though. And it seems to amplify the color drift scunet is doing.


gamma = 1.5
clip = core.std.Levels(clip, gamma=gamma, planes=[0,1,2])
clip = scunet(clip, model = 4)
clip = core.std.Levels(clip, gamma=1.0/gamma, planes=[0,1,2])


Another and more effective way is to upscale the clip before the denoising and rescaling it back to original size afterwards. Here for example with scaling factor 2:


orig_width = clip.width
orig_height = clip.height
clip = core.resize.Lanczos(clip, width=clip.width*2, height=clip.height*2, src_left = -0.25, src_top = -0.25)
clip = scunet(clip, model = 4)
clip = core.resize.Point(clip, width=orig_width, height=orig_height)


Obviously this takes 4 times as much memory and processing power. Scaling by 2 leaves quite a lot more noise so perhaps a smaller scale factor will be enough. And you can of course combine this with the gamma trick.

[lansing]

I'm trying to create a lower quality image from a higher quality image for training a model for anime. I need to recreate a dvd quality image from a cel scan,

from this:
https://imgur.com/FWW2p6B

to this:
https://imgur.com/VdrndTA

Do we have vapoursynth filters that can do that?

[mastrboy]

https://gitlab.com/bg123/vapoursynth-vhs-fx/-/tree/master (I have not tested it)

[lansing]

https://gitlab.com/bg123/vapoursynth-vhs-fx/-/tree/master (I have not tested it)

The syntax was outdated? I'm getting "BlankClip: invalid length" error on this line:

attribute_clip = core.std.BlankClip(clip, length=math.floor(len(clip) * target_fps_num / target_fps_den * clip.fps_den / clip.fps_num), fpsnum=target_fps_num, fpsden=target_fps_den)

[ChaosKing]

Almost perfect decrawl / Derainbow filter 1x_Dotzilla_Compact. https://upscale.wiki/wiki/Model_Database#Anime_and_Cartoons
It just works + speed is also ok.

https://imgsli.com/MTM4ODkz
https://i.imgur.com/hp6tBMC.jpeg

[Selur]

@ChaosKing: nice.

[ReinerSchweinlin]

Almost perfect decrawl / Derainbow filter 1x_Dotzilla_Compact. https://upscale.wiki/wiki/Model_Database#Anime_and_Cartoons
It just works + speed is also ok.

https://imgsli.com/MTM4ODkz
https://i.imgur.com/hp6tBMC.jpeg
looking good :)

[PatchWorKs]

New toy https://github.com/HolyWu/vs-scunet
Online demo https://replicate.com/cszn/scunet

So far I like it. It creates very consistent and clean results (a bit too clean).

Nice, here's a "degrain" of a 1080p source (model - real image denoise):
https://i31.servimg.com/u/f31/16/32/04/85/vlcsna11.jpg
https://i31.servimg.com/u/f31/16/32/04/85/output10.jpg

Definition and shadows looks ok, blurry is minimal.

Anyway I can notice a little banding (check the sky in the center of the image).

EDIT: https://imgsli.com/MTUxMTY5 (overlay comparison)

[ChaosKing]

yeah its very good. But look at the guys shirt. It's like a "shadow" is removed or the shirt got wasched clean. Interessting effect :D

I wonder if a "temporal scunet" would be even better.

If think if you combine the scunet filtered frame with a smdegrain filtered frame the result would be almost perfect (maybe add some debanding too)

[PatchWorKs]

...it would be also interesting to have a "combined" (denoise+SDRtoHDR) tool:

https://github.com/MCG-NKU/FMNet

It could be also *very* useful to perform a block partitioning-optimization at the same time:

https://github.com/fengxinmin/iDepthmap-x265

[PatchWorKs]

Just discovered aydin (https://royerlab.github.io/aydin/), "a user-friendly, feature-rich, and fast image denoising tool that provides a number of self-supervised, auto-tuned, and unsupervised image denoising algorithms":

https://royerlab.github.io/aydin/v0.1.15/_images/aydin_welcome_screen.png
https://royerlab.github.io/aydin/v0.1.15/_images/aydin_denoise_tab.png

Website: https://royerlab.github.io/aydin/

Git: https://github.com/royerlab/aydin

Quite surprised that we didn't found any related search result here ad Doom9...

[Selur]

Not that surprising, since there are not many threads that are image centric.
Usually it's about video. And image based filters can easily lead to temporal inconsistencies.

[PatchWorKs]

Not that surprising, since there are not many threads that are image centric.
Usually it's about video. And image based filters can easily lead to temporal inconsistencies.
Very interesting reply by aydin's author (https://github.com/AhmetCanSolak) @ my GitHub "issue" (https://github.com/royerlab/aydin/issues/292):
I want to clarify that Aydin works on spatio-temporal data(TZYX) when all four dimensions are available and can do TYX as well. Aydin doesn't read video file formats directly though, if you can save it as a file format that is supported then you can denoise movies too. With biological live imaging datasets, we use Aydin for denoising movies on daily basis.

[PatchWorKs]

@Selur and anyone else interested in participating are called to collaborate by "make bug reports/feature requests here on GitHub" according to the aydin's author:
https://github.com/royerlab/aydin/issues/292#issuecomment-1428538108

[Selur]

btw. has anyone tested if there are speed differences depending on the RGB color space VSGAN is fed?
Just noticed that for me (only testst a few models), RGBS seems to be ~10% faster than when I feed any other RGB color space.
-> Is it just me or can others confirm this?

Cu Selur

[poisondeathray]

btw. has anyone tested if there are speed differences depending on the RGB color space VSGAN is fed?
Just noticed that for me (only testst a few models), RGBS seems to be ~10% faster than when I feed any other RGB color space.
-> Is it just me or can others confirm this?

Cu Selur

Similar for me ; RGBS was about 7-10% faster than RGB24 (source was RGB24)

[Selur]

Thanks for testing :)

[Selur]

Argh, I got a strange problem when using vsdpir and vsgan inside the same script:

using both with RGH:

from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBH for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="470bg", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=5.000, task="denoise", device_index=0, trt=True, trt_cache_path="J:/tmp")
# changing range from limited to full range
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# Setting color range to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "C:/Users/Selur/Desktop/testing/1x_Dotzilla_Compact_80k_net_g.pth"
vsgan.load(model)
vsgan.apply() # 640x352
clip = vsgan.clip

works.

using both with RGBS:

from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=5.000, task="denoise", device_index=0, trt=True, trt_cache_path="J:/tmp")
# changing range from limited to full range
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# Setting color range to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "C:/Users/Selur/Desktop/testing/1x_Dotzilla_Compact_80k_net_g.pth"
vsgan.load(model)
vsgan.apply() # 640x352
clip = vsgan.clip

works.

Using VSDIR with RGBS and VSGAN with RGBH:
from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=5.000, task="denoise", device_index=0, trt=True, trt_cache_path="J:/tmp")
# changing range from limited to full range
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# Setting color range to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
# adjusting color space from RGBS to RGBH for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, range_s="full", dither_type="error_diffusion")
# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "C:/Users/Selur/Desktop/testing/1x_Dotzilla_Compact_80k_net_g.pth"
vsgan.load(model)
vsgan.apply() # 640x352
clip = vsgan.clip
works.

Using VSDIR with RGBH and VSGAN with RGBS:

from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBH for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="470bg", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=5.000, task="denoise", device_index=0, trt=True, trt_cache_path="J:/tmp")
# changing range from limited to full range
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# Setting color range to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
# adjusting color space from RGBH to RGBS for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, range_s="full")
# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "C:/Users/Selur/Desktop/testing/1x_Dotzilla_Compact_80k_net_g.pth"
vsgan.load(model)
vsgan.apply() # 640x352
clip = vsgan.clip

fails with:
Input type (float) and bias type (struct c10::Half) should be the same


Using VSDIR with RGBS and VSGAN with RGB48:

from vsdpir import dpir as DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=5.000, task="denoise", device_index=0, trt=True, trt_cache_path="J:/tmp")
# changing range from limited to full range
clip = core.resize.Bicubic(clip, range_in_s="limited", range_s="full")
# Setting color range to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
# adjusting color space from RGBS to RGB48 for vsVSGAN
clip = core.resize.Bicubic(clip=clip, format=vs.RGB48, range_s="full", dither_type="error_diffusion")
# resizing using VSGAN
from vsgan import ESRGAN
vsgan = ESRGAN(clip=clip,device="cuda")
model = "C:/Users/Selur/Desktop/testing/1x_Dotzilla_Compact_80k_net_g.pth"
vsgan.load(model)
vsgan.apply() # 640x352
clip = vsgan.clip

produces graphic glitches here https://ibb.co/jTCbghX

Cu Selur

Ps.: I also reported this to https://github.com/rlaphoenix/VSGAN/issues/31, but thought it might be interesting to others in case they run into the same issue.

[Selur]

Okay, issue seems to be that VSDPIR overwrites the default tensor type. (so not a bug in VSGAN, but an unexpected behavior of VSDPIR)
Using:
import torch
torch.set_default_tensor_type(torch.FloatTensor)
after VSDPIR, should fix this it.

Behavior is fixed with https://github.com/HolyWu/vs-dpir/releases/tag/v3.0.1

Cu Selur

[lansing]

New 2x upscaling model for anime AniScale2
https://github.com/Sirosky/Upscale-Hub/releases/tag/AniScale2

It only works for digital created animes. It came with a base upscale model and a refiner model for sharpening. The model has depth of field awareness so the intended blur effect will be kept intact in the result.

I tested it a little on BLEACH and it looks pretty convincing.

[Selur]

Interesting, thanks for the info. :)