vs-basicvsr

[Selur]

Has anyone tried https://github.com/tongyuantongyu/vs-NNVISR/ ?

[Selur]

Tried to test it by extracing everything into "F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR" ending with:
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cublas64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cublasLt64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudart64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_adv_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_adv_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_cnn_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_cnn_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_ops_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_ops_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvblas64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_builder_resource.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_dispatch.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_lean.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_plugin.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_vc_plugin.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvonnxparser.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvrtc-builtins64_121.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvrtc64_120_0.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\vs-nnvisr.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\fe_n2_2x2_l4_yuv1-1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\fe_n4_4x4_l1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\ff_n2_2x2_l4_yuv1-1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\ff_n4_4x4_l1.onnx

Problem is when i use:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core

# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll")

# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\files\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/files/test.avi", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)

# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip, _Matrix=5)
# Setting color transfer info (470bg), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
# Setting color primaries info (), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive


clip = core.nnvisr.Super(clip, scale_factor=2, use_fp16=True, model="fe_n2_2x2_l4_yuv1-1.onnx", model_path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR")



# set output frame rate to 25fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

I get:
Failed to evaluate the script:
Python exception: Failed to load F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll. GetLastError() returned 1114.

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 3115, in vapoursynth._vpy_evaluate
File "src\cython\vapoursynth.pyx", line 3116, in vapoursynth._vpy_evaluate
File "C:\Users\Selur\Desktop\test_1.vpy", line 8, in
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll")
File "src\cython\vapoursynth.pyx", line 2847, in vapoursynth.Function.__call__
vapoursynth.Error: Failed to load F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll. GetLastError() returned 1114.
So the loading of the dll already fails. :(

-> does anyone have an idea what I'm missing/overlooking?

Cu Selur

[Selur]

Got it working.
dll folder needs to explicitly loaded:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import site
import os
import ctypes
# Adding torch dependencies to PATH
path = site.getsitepackages()[0]+'/../vsfilters/ResizeFilter/NNVISR'
ctypes.windll.kernel32.SetDllDirectoryW(path)
path = path.replace('\\', '/')
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]
and Primaries, Transfer and Matrix all have to be BT709 (for RGB only Primaries and Transfer have to be bt709).

Cu Selur
Ps.: the explicit loading is fixed in 'v1.0.0-pre3'

[ReinerSchweinlin]

Cool you got it working :)

If I understand correctly, this enables multi-frame aware networks to be run, not only single-frame ones. Correct?

[Selur]

Yes, vs-NNVISR support models which use mutliple frames for their calculation, main problem is finding such models (or training them yourself).
We are not aiming at being an alternative of vs-mlrt, but focus mainly on models that do multiple frames to multiple frames enhancement, which vs-mlrt currently not able to do, or at least not easily. source: issue #1 (https://github.com/tongyuantongyu/vs-NNVISR/issues/1#issuecomment-1585733868)

Side note: VSGAN also supports EGVSR models, which are 'temporal aware'.

Current available models only run with specific values.
"'scale_factor': 2, 'input_count': 2,..., 'interpolation': True" means the model always does scale by 2, takes two frames as input and interpolates the frames by 2.
Also, the YUV420 1/1/1 means that it only works if color fromat is YUV420(P8), _Transfer, _Primaries and _Matrix are bt709.
see: https://github.com/tongyuantongyu/vs-NNVISR/blob/main/docs/models.md

So to me, it's nice to see something different is being developed, but atm. I won't use it since it seems too much of a hassle. ;)

Cu Selur

Ps.: Seems like the 'Enhance Everything' folks are revamping https://upscale.wiki/wiki/Model_Database into https://openmodeldb.info/

[mastrboy]

Ps.: Seems like the 'Enhance Everything' folks are revamping https://upscale.wiki/wiki/Model_Database into https://openmodeldb.info/

Looks good, the wiki format doesn't really work that well for content like that...

vsbasicvsrpp does temporal I think, with the length parameter: https://github.com/HolyWu/vs-basicvsrpp/blob/master/vsbasicvsrpp/__init__.py#L50

[Selur]

Yes, BasicVSR++ does, but most models you load in VSGAN/vsMLRT do not.

[Selur]

Trying to set up a new portable Vapoursynth R64 with BasicVSR++&Co I'm stuck.
What I did so far to create a portable Vapoursynth with pytorch&co:

created a new empty Vapoursynth-folder
downloaded 'Windows embeddable package (64-bit)' from https://www.python.org/downloads/release/python-3111/
extracted the Python download into the download into the 'Vapoursynth'-folder
downloaded 'VapourSynth64-Portable-R64' from https://github.com/vapoursynth/vapoursynth/releases
extracted the Vapoursynth portable download into the 'Vapoursynth'-folder
downloaded get-pip.py from https://bootstrap.pypa.io/get-pip.py and save it into the 'Vapoursynth'-folder
opened a 'Windows Command Prompt'-window and navigate into the 'Vapoursynth'-folder
installed pip by calling :
python get-pip.py
opened the python311._pth in a text addition and added the following to lines above anything else in that file and saved the file
Scripts
Lib\site-packages
installed pytorch using:
python -m pip install torch===2.1.0+cu121 torchvision===0.16.0 -f https://download.pytorch.org/whl/torch_stable.html

installed VSGAN (https://github.com/rlaPHOENiX/VSGAN)
python -m pip install vsgan
installed DPIR (https://github.com/HolyWu/vs-dpir)

downloaded https://github.com/HolyWu/vs-rife/releases/download/v4.0.0/CUDA-12.2.2_cuDNN-8.9.5_TensorRT-8.6.1.6_win64.7z extracted tensorrt-8.6.1-cp311-none-win_amd64.whl file into the Vapoursynth-folder.
extracted the other dlls into a 'torch_dependencies/bin'-folder inside the Vapoursynth folder
called set PATH=<PATH TO torch_dependencies>\bin;%PATH%
installed vsdir and its dependencies
python -m pip install tensorrt-8.6.1-cp311-none-win_amd64.whl
python -m pip install -U vsdpir
python -m pip install packaging
python -m vsdpir

installed Real-ESGRAN (https://github.com/HolyWu/vs-realesrgan)
python -m pip install -U vsrealesrgan
python -m vsrealesrgan
installed vs-rife (https://github.com/HolyWu/vs-rife)
python -m pip install --upgrade vsrife
python -m vsrife
installed GMFSS (https://github.com/HolyWu/vs-gmfss_fortuna)
python -m pip install -U vsgmfss-fortuna
installed SwinIR (https://github.com/HolyWu/vs-swinir)
python -m pip install -U vsswinir
python -m vsswinir
installe HINet (https://github.com/HolyWu/vs-hinet)
python -m pip install --upgrade vshinet
python -m vshinet
installed AnimeSR (https://github.com/HolyWu/vs-animesr)
python -m pip install -U vsanimesr
installed FeMaSR (https://github.com/HolyWu/vs-femasr)
python -m pip install -U vsfemasr
python -m vsfemasr
installed SCUNet (https://github.com/HolyWu/vs-scunet)
python -m pip install -U vsscunet
python -m vsscunet
installed CodeFormer (https://github.com/HolyWu/vs-codeformer)
python -m pip install -U vscodeformer
python -m vscodeformer
installed GRLIR (https://github.com/HolyWu/vs-grlir)
python -m pip install -U vsgrlir
python -m vsgrlir

Not I wanted to call:
python -m pip install -U openmim
mim install "mmcv>=2.0.0"
python -m pip install -U vsbasicvsrpp
python -m vsbasicvsrpp

to install BasicVSR++, problem is when calling:
python -m pip install -U openmim
it fails with:
Collecting aliyun-python-sdk-core>=2.13.12 (from oss2~=2.17.0->openxlab->opendatalab->openmim)
Using cached aliyun-python-sdk-core-2.14.0.tar.gz (443 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [7 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "C:\Users\Selur\AppData\Local\Temp\pip-install-t2r9eu4j\aliyun-python-sdk-core_b912f61c599e490b859580d9c77cbbd2\setup.py", line 41, in <module>
VERSION = __import__(PACKAGE).__version__
^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'aliyunsdkcore'
[end of output]
see: https://github.com/HolyWu/vs-basicvsrpp/issues/37

I also tried installing aliyun-python-sdk-core
with:
python -m pip install aliyunsdkcore
and
python -m pip install aliyun-python-sdk-core-v3
but both calls end the same way, with the same error I got above.

=> does anyone have an idea how to fix this?

Cu Selur

[Selur]

Seems to be related to the openmim version, when using:
python -m pip install -U openmim==0.3.7
(0.3.7 was the version I had installed in a previous portable setup)
openmim does install without an issue, but then
set PATH=f:\Hybrid\64bit\Vapoursynth\Scripts;%PATH%
mim install "mmcv>=2.0.0"
fails with:

Looking in links: https://download.openmmlab.com/mmcv/dist/cu121/torch2.1.0/index.html
Collecting mmcv>=2.0.0
Using cached mmcv-2.0.1.tar.gz (469 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
C:\Users\Selur\AppData\Local\Temp\pip-install-xs23k5n_\mmcv_a93dd8751cad4bf1bafcae67f7e00b43\setup.py:5: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
from pkg_resources import DistributionNotFound, get_distribution, parse_version
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "C:\Users\Selur\AppData\Local\Temp\pip-install-xs23k5n_\mmcv_a93dd8751cad4bf1bafcae67f7e00b43\setup.py", line 466, in <module>
ext_modules=get_extensions(),
^^^^^^^^^^^^^^^^
File "C:\Users\Selur\AppData\Local\Temp\pip-install-xs23k5n_\mmcv_a93dd8751cad4bf1bafcae67f7e00b43\setup.py", line 423, in get_extensions
ext_ops = extension(
^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\cpp_extension.py", line 1076, in CUDAExtension
library_dirs += library_paths(cuda=True)
^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\cpp_extension.py", line 1210, in library_paths
paths.append(_join_cuda_home(lib_dir))
^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\cpp_extension.py", line 2416, in _join_cuda_home
raise OSError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "f:\Hybrid\64bit\Vapoursynth\Scripts\mim.exe\__main__.py", line 7, in <module>
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\mim\commands\install.py", line 73, in cli
exit(exit_code)
^^^^
NameError: name 'exit' is not defined

¯\(°_o)/¯
(same when using openmim==0.3.8, openmim==0.3.9 fails at the install of openmim)

Cu Selur

[Selur]

Pre-built package for PyTorch 2.1+cu121 is probably not available. File an issue to MMCV if you really hurry to have it. source: https://github.com/HolyWu/vs-basicvsrpp/issues/37#issuecomment-1749183229
=> so there's hope, but will have to wait

[ReinerSchweinlin]

Sorry, I can´t help, thats beyond my capabilities :)

But I just wanted to say "thanks" for all you do !!

[Selur]

btw. did anyone test NNVISR ? (https://github.com/tongyuantongyu/vs-NNVISR)

[lansing]

btw. did anyone test NNVISR ? (https://github.com/tongyuantongyu/vs-NNVISR)

I couldn't get it working on vsedit2 or vd2 or vsedit_mod. I manual install the plugin and its dependencies. Somehow the plugin won't auto load when I put it in my roaming folder, so I'll have to put everything in Vapoursynth's installation plugin folder. But everything just crashed after some loading time.


import vapoursynth as vs

core = vs.core

clip = core.bs.VideoSource(r'E:\video samples\hand wave.mkv')

model_config = {'scale_factor': 2, 'input_count': 2, 'feature_count': 64, 'extraction_layers': 4, 'interpolation': True, 'extra_frame': True, 'double_frame': True, 'model': 'cycmunet/vimeo90k-deblur'}

clip = core.nnvisr.Super(clip,
model_path=r"C:\Program Files\VapourSynth\plugins", **model_config)

clip.set_output()

[Selur]

When I first tested this, I used this file structure
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cublas64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cublasLt64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudart64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_adv_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_adv_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_cnn_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_cnn_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_ops_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_ops_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvblas64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_builder_resource.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_dispatch.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_lean.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_plugin.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_vc_plugin.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvonnxparser.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvrtc-builtins64_121.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvrtc64_120_0.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\test_1.vpy
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\timing.cache
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\vs-nnvisr.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\timing.cache
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet\vimeo90k-deblur
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet\vimeo90k-deblur\yuv_1_1_1
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet\vimeo90k-deblur\yuv_1_1_1\fe_640x352_2x2_b1_l4_yuv1-1.engine
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet\vimeo90k-deblur\yuv_1_1_1\ff_n2a+_640x352_2x2_b1_l4_yuv1-1.engine
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet\vimeo90k-deblur
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet\vimeo90k-deblur\yuv_1_1_1
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet\vimeo90k-deblur\yuv_1_1_1\fe_640x352_2x2_b1_l4_yuv1-1.engine
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet\vimeo90k-deblur\yuv_1_1_1\ff_n2a+_640x352_2x2_b1_l4_yuv1-1.engine
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet\vimeo90k-deblur
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet\vimeo90k-deblur\yuv_1_1_1
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet\vimeo90k-deblur\yuv_1_1_1\fe_n2_2x2_l4_yuv1-1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet\vimeo90k-deblur\yuv_1_1_1\ff_n2_2x2_l4_yuv1-1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo\vimeo90k
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo\vimeo90k\rgb_1_1
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo\vimeo90k\rgb_1_1\fe_n4_4x4_l1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo\vimeo90k\rgb_1_1\ff_n4_4x4_l1.onnx
and the following script:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import site
import os
import ctypes
# Adding torch dependencies to PATH
path = site.getsitepackages()[0]+'/../vsfilters/ResizeFilter/NNVISR'
ctypes.windll.kernel32.SetDllDirectoryW(path)
path = path.replace('\\', '/')
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]


# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll")

# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\files\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/files/test.avi", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)

# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip, _Matrix=5)
# Setting color transfer info (470bg), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
# Setting color primaries info (), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive

clip = core.resize.Bicubic(clip=clip, matrix_in_s="470bg", matrix_s="709", range_in=0, range=0) # converting to bt709 for NNVISR
clip = core.std.SetFrameProps(clip, _Transfer=1)
clip = core.std.SetFrameProps(clip, _Primaries=1)

model_config = {'scale_factor': 2, 'input_count': 2, 'feature_count': 64, 'extraction_layers': 4, 'interpolation': True, 'extra_frame': True, 'double_frame': True, 'model': 'cycmunet/vimeo90k-deblur'}
clip = core.nnvisr.Super(clip, model_path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR", **model_config)

# Output
clip.set_output()
in a portable Vapoursynth setup. But I had my struggles too (https://github.com/tongyuantongyu/vs-NNVISR/issues/2).
Maybe this helps you getting it working.

Cu Selur

Ps.: iirc. it took ages to get engine files created on my setup when I tested it.

[julius666]

Does anyone know what happened to vs-femasr? The github repository is not available anymore.

[kedautinh12]

HolyWu was deleted it (maybe)

[Selur]

Yup, like a few other repos HolyWu deleted it. (or set it private)

[julius666]

Yup, like a few other repos HolyWu deleted it. (or set it private)

I thought so but do we have any information regarding why he did that? It was one of the best (if not THE best) upscalers in vapoursynth.

[Selur]

No (at least not that I know of), you would have to ask him (maybe by posting an issue in one of this other projects https://github.com/HolyWu/).
Probably because there hasn't been an update to it, not much feedback and keeping it working with latest pytorch&co was too much trouble. :)

[Selur]

btw. for those who need colorization of b&w content https://github.com/HolyWu/vs-ddcolor might be worth a try:
https://i.ibb.co/rwGVJr8/grafik.png (https://ibb.co/d50RyYC)
did some small tests which do seem good so far.

Cu Selur

[kedautinh12]

Wow, respect HolyWu :D

[Selur]

https://github.com/piddnad/DDColor should get the main respect, HolyWu 'just' wrote a Vapoursynth Wrapper for it.

[kedautinh12]

Ok, piddnad too

[Selur]

Anyone has an idea how to stabilize the colorization output of ddcolor? (see: https://forum.selur.net/thread-3532.html)

[Selur]

https://github.com/dan64/vs-deoldify might be interesting for those interested in colorization
https://i.ibb.co/p3Xm0P1/grafik.png (https://ibb.co/gDgnj7r)

[ReinerSchweinlin]

Thank you.

I have a somewhat amateur question:

If this is based on pytorch - what relevance does the used GPU have in this case? Some hybrid filters are clearly based on specific stuff like vulkan or CUDA (CDNN) or some other filters use OPENCL... How does it work here?

I have been playing around with some StableDiffusion GUIs and here It seems (as far as I understand it), all that needs to be modified is how Pytorch passes the calls from the GUI to the underlying inferences. So if I want to use DirectML, I need the py-directML stuff, if i have Nvidia, I stick to the tensorRT or CUDA, hence must install pytor...cuda... etc... So basically, as long as some backend in pytorch for my specific hardware exists, the filter itself will run...

Could you help straighten out my somewhat fuzzy thinking of how it is related ?

Thanx :)

[Selur]

afaik vs-deoldify uses cudnn as backend (see: https://github.com/dan64/vs-deoldify/blob/main/vsdeoldify/deoldify/_device.py)

[ReinerSchweinlin]

afaik vs-deoldify uses cudnn as backend (see: https://github.com/dan64/vs-deoldify/blob/main/vsdeoldify/deoldify/_device.py)
Thanx for pointing me in the right direction :)

[Selur]

Side note for those using tensorrt: atm. there isn't a Python 3.12 version of the bindings (tensorrt-8.6.1-cp311-none-win_amd64.whl is the latest currently), so switching to Vapoursynth R66 will not work.

Cu Selur

[PatchWorKs]

Just out of curiosity: does anyone tested vs-mlrt (https://github.com/AmusementClub/vs-mlrt) performances on Intel CPU+iGPU ?

[Selur]

Only tested that it works with my A380, but never did any speed test since my RTX 4080 is clearly faster.

Cu Selur

[PatchWorKs]

Only tested that it works with my A380, but never did any speed test since my RTX 4080 is clearly faster.

Cu Selur
Ok, I'd like to understand how feasible (video) inferences are WITHOUT a discrete video card...

EDIT
I forgot to mention that I've collected some "AI" (= deep learning/neural nets) video tools in this GitHub page (https://github.com/forart/HyMPS/blob/main/V_AI.md).
Suggestions - and direct contribution too - are welcome.

[Selur]

Did anyone manage to get BasicVSR++ working with R66, now R67? (both requiring Python 3.12)

[Selur]

(since a user asked me via pm)
For those interested, here's how I atm. would create a portable Vapoursynth 67 with pytorch&co:

creat a new empty Vapoursynth-folder
download 'Windows embeddable package (64-bit)' from https://www.python.org/downloads/release/python-3122/
extract the Python download into the download into the 'Vapoursynth'-folder
downloaded 'VapourSynth64-Portable-R67' from https://github.com/vapoursynth/vapoursynth/releases
extract the Vapoursynth portable download into the 'Vapoursynth'-folder
download get-pip.py from https://bootstrap.pypa.io/get-pip.py and save it into the 'Vapoursynth'-folder
open a 'Windows Command Prompt'-window and navigate into the 'Vapoursynth'-folder
install pip by calling :
python get-pip.py
open the python312._pth in a text addition and added the following to lines above anything else in that file and saved the file
Scripts
Lib\site-packages
install wheel
python -m pip install wheel/VapourSynth-67-cp312-cp312-win_amd64.whl
installed pytorch using:
python -m pip install torch===2.2.0+cu121 torchvision===0.17.1 -f https://download.pytorch.org/whl/torch_stable.html

TensorRT

created a 'torch_dependencies/bin'-folder inside the Vapoursynth folder
download https://developer.download.nvidia.com/compute/machine-learning/tensorrt/10.0.0/zip/TensorRT-10.0.0.6.Windows10.win10.cuda-12.4.zip
extract (from the lib folder into the 'torch_dependencies/bin'-folder):

nvinfer.dll
nvinfer_builder_resource.dll
nvinfer_plugin.dll
nvinfervc__plugin.dll
nvonnxparser.dll

extract (from the python folder into the Vapoursynth folder: 'tensorrt-10.0.0b6-cp312-none-win_amd64.whl' and called:
python -m pip install tensorrt-10.0.0b6-cp312-none-win_amd64.whl
download https://developer.download.nvidia.com/compute/cuda/12.4.1/local_installers/cuda_12.4.1_551.78_windows.exe
extract:

cuda_cudart\cudart\bin\cudart64_12.dll
cuda_nvrtc\nvrtc\bin\nvrtc64_120_0.dll
cuda_nvrtc\nvrtc\bin\nvrtc-builtins64_124.dll
libcublas\cublas\bin\cublas64_12.dll
libcublas\cublas\bin\cublasLt64_12.dll
libcublas\cublas\bin\nvblas64_12.dll
libcufft\cufft\bin\cufft64_11.dll
libcufft\cufft\bin\cufftw64_11.dll
libcurand\curand\bin\curand64_10.dll
libcusolver\cusolver\bin\cusolver64_11.dll
libcusolver\cusolver\bin\cusolverMg64_11.dll
libcusparse\cusparse\bin\cusparse64_12.dll
libnvjpeg\nvjpeg\bin\nvjpeg64_12.dll

into the 'torch_dependencies/bin'-folder

download https://developer.download.nvidia.com/compute/cudnn/9.1.0/local_installers/cudnn_9.1.0_windows.exe
extract:

cudnn_cuda12.4\libcudnn\bin\12.4\cudnn64_9.dll
cudnn_cuda12.4\libcudnn\bin\12.4\cudnn_adv64_9.dll
cudnn_cuda12.4\libcudnn\bin\12.4\cudnn_cnn64_9.dll
cudnn_cuda12.4\libcudnn\bin\12.4\cudnn_ops64_9.dll

call set PATH=<PATH TO torch_dependencies>\bin;%PATH%
i.e.: set PATH=F:\Vapoursynth\torch_dependencies\bin;%PATH%;
install VSGAN (https://github.com/rlaPHOENiX/VSGAN)
python -m pip install -U VSGAN
installed DPIR (https://github.com/HolyWu/vs-dpir)

python -m pip install -U vsdpir
python -m pip install packaging
python -m pip install onnxruntime-gpu
python -m vsdpir[/code]

install Real-ESGRAN (https://github.com/HolyWu/vs-realesrgan)
python -m pip install -U vsrealesrgan
python -m vsrealesrgan
install vs-rife (https://github.com/HolyWu/vs-rife)
python -m pip install --upgrade vsrife
python -m vsrife
install GMFSS (https://github.com/HolyWu/vs-gmfss_fortuna)
python -m pip install -U vsgmfss-fortuna
install SwinIR (https://github.com/HolyWu/vs-swinir)
python -m pip install -U vsswinir
python -m vsswinir
install HINet (https://github.com/HolyWu/vs-hinet)
python -m pip install --upgrade vshinet
python -m vshinet
install AnimeSR (https://github.com/HolyWu/vs-animesr)
python -m pip install -U vsanimesr
install FeMaSR (https://github.com/HolyWu/vs-femasr)
python -m pip install -U vsfemasr
python -m vsfemasr
install SCUNet (https://github.com/HolyWu/vs-scunet)
python -m pip install -U vsscunet
python -m vsscunet
install CodeFormer (https://github.com/HolyWu/vs-codeformer)
python -m pip install -U vscodeformer
python -m vscodeformer
install GRLIR (https://github.com/HolyWu/vs-grlir)

python -m pip install antlr4-python3-runtime
python -m pip install wheel
python -m pip install -U vsgrlir
python -m vsgrlir
install BasicVSR++ (https://github.com/HolyWu/vs-basicvsrpp)
????


Sadly I have no clue how to BasicVSR++ (https://github.com/HolyWu/vs-basicvsrpp) installed, so far

set CUDA_HOME=F:\Vapoursynth\torch_dependencies\bin;
python -m pip install -U openmim==0.3.7

anything I tried so far after this fails :/
Seems like the problem is that there seems to be no mmcv with cp312 wheel for Windows.

Hope this is useful to anyone and if anyone finds a way to add BasicVSR++

Cu Selur

[Cary Knoop]

Thanks for that detailed description!

Life could be a lot easier if VapourSynth simply adhered to common standards.

More than a decade of VapourSynth and a simple "pip install VapourSynth" from a conda environment is still not supported. :(

[Selur]

Only 'fault' with Vapousynth is that it abandoned Python 3.11 and now requires Python 3.12, which other projects do not support atm.

[PatchWorKs]

Any (Adobe-backended) VideoGigaGAN (https://videogigagan.com/) test ?

Demo: https://videogigagan.github.io/assets/videos/demo.mp4

Project: https://videogigagan.github.io/

Git: https://github.com/danaigc/videoGigaGanHub

[Selur]

https://github.com/danaigc/videoGigaGanHub seems to lack the used models, so no way to reproduce the presented results (or test with other content).

[PatchWorKs]

...well, it seemed a bit strange that Adobe-backeded researchers pretended to be open (using Microsoft's GitHub, moreover)...

[PatchWorKs]

Fast-SRGAN (https://github.com/HasnainRaz/Fast-SRGAN#readme) (The goal of this repository is to enable real time super resolution for upsampling low resolution videos. [] For speed, the upsampling is done through pixel shuffle.) anyone ?

[Selur]

Isn't that like 5 years old,... with https://github.com/HolyWu/vs-realesrgan I get way faster speeds on 180x320 content, but then my hardware is also way faster than a m1.

[Selur]

Did anyone try ccrestoration (https://github.com/TensoRaws/ccrestoration) ?

[ReinerSchweinlin]

No - but the amount of usable models look very interesting.

It seems the open models site offers the models in pytorch, ONNX and NCNN Format - Isn´t hybrid already capable of using some of these ?

[Selur]

Yes, Hybrid can use those that VSGAN and VSMLRT support, but trying somthing new seems worth a look especially since some of the mentioned architectures https://github.com/TensoRaws/ccrestoration/blob/main/ccrestoration/type/arch.py are not supported by VSGAN and VSMLRT afaik. :)

[poisondeathray]

vs_liff - interesting resizing project . Works ok , but looks like only the baseline model so far

https://github.com/pifroggi/vs_liif

https://yinboc.github.io/liif/

[Selur]

Looked at it a while ago, but couldn't find an example where I liked the output :)

[ReinerSchweinlin]

Yes, Hybrid can use those that VSGAN and VSMLRT support, but trying somthing new seems worth a look especially since some of the mentioned architectures https://github.com/TensoRaws/ccrestoration/blob/main/ccrestoration/type/arch.py are not supported by VSGAN and VSMLRT afaik. :)
Tempting :)

[ReinerSchweinlin]

Looked at it a while ago, but couldn't find an example where I liked the output :)

For some Line Art it produces not so bad results and IMHO it could be usefull as a generic upscaler - while most GAN based algos are much better in cases where they are trained on the spcific footage and "recognize" the content and add plausible details - in cases where the don´t recognize anything, the often fall apart (zombie faces) - here the liif apporach could work better.

[asarian]

This is sad. :( I was hoping to use vs-basicvsr, to replace Oyster (as it no longer works); and now this one is gone altogether. And I really need a filter to get rid of those ugly compression artifacts.

[Selur]

vs-basicvsr was replaced with BasicVSR++ (https://github.com/HolyWu/vs-basicvsrpp) quite a while ago,...
(answered in your Oyster thread how to fix the Oyster.py to work with newer Vapoursynth versions)