vs-dpir onnxruntime portable?

[Selur]

Trying to get a portable Vapoursynth running with onnxruntime to use latest vs-dpir and vs-realesrgan:

Okay, here's how I setup Vapoursynth to get it working for CUDA (atm.):

I created a new empy Vapoursynth-folder
downloaded 'Windows embeddable package (64-bit)' from https://www.python.org/downloads/release/python-3912/
extracted the Python download into the download into the 'Vapoursynth'-folder
downloaded 'VapourSynth64-Portable-R57' from https://github.com/vapoursynth/vapoursynth/releases
extracted the Vapoursynth portable download into the 'Vapoursynth'-folder
downloaded get-pip.py from https://bootstrap.pypa.io/get-pip.py and save it into the 'Vapoursynth'-folder
opened a 'Windows Command Prompt'-window and navigate into the 'Vaporusynth'-folder
installed pip by calling :
pyhton get-pip.py
opened the python39._pth in a text addition and added the following to lines above anything else in that file and saved the file

Scripts
Lib\site-packages

installed VSGAN

python -m pip install vsgan
python -m pip install torch===1.11.0+cu113 torchvision==0.12.0 -f https://download.pytorch.org/whl/torch_stable.html

installed BASICVSR++

python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11/index.html
python -m pip install tqdm
python -m pip install opencv-python
python -m pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp

installted RIFE
python -m pip install --upgrade vsrife
installed SWINIR

python -m pip install --upgrade vsswinir
python -m vsswinir

installed DPIR and onnxruntime-gpu

python -m pip install --upgrade vsdpir
python -m pip install --upgrade onnxruntime-gpu

from cudnn-11.4-windows-x64-v8.2.4.15.zip and NVIDIA CUDA SDK 11.4.1 runtimes
I copied:
cublas64_11.dll
cublasLt64_11.dll
cudart64_110.dll
cudnn64_8.dll
cudnn_cnn_infer64_8.dll
cudnn_ops_infer64_8.dll
cufft64_10.dll
cufftw64_10.dll
into Vapoursynth/Lib/site-packages/onnxruntime/capi and then uninstalled the sdk and cudnn and cleared all __pycache__-folders inside the Vapoursynth subfolders.
downloaded the vsdpir modules
python -m vsdpir
installed REALESRGAN (which also uses onnxruntime)

python -m pip install --upgrade vsrealesrgan
python -m vsrealesrgan



When I add the path to "Vapoursynth/Lib/site-packages/onnxruntime/capi" where the dlls are to my systemwide PATH variable vsdpir works fine.
(I can also move the dlls into a separate folder, as long as that folder is listed in the PATH variable, it works.)

Problem is: I want it portable without having to change the PATH variable on the system.

I tried adding:
import os
import site
# Import scripts folder
os.environ["PATH"] += site.getsitepackages()[0]+'/Lib/site-packages/onnxruntime/capi'
to the start of my Vapoursynth scripts, but that didn't help.
I also tried to expand the PATH, just for the process that runs vsViewer (which I use to open the script), but that didn't help either.

=> does anyone have an idea how to get the onnxruntime working without having to adjust the PATH variable or copy the dlls to a path that is already in the PATH variable?

Cu Selur

Ps.: I'm struggling with this for a few days and since I can't find a solution, I thought may be someone here knows a way. :)

[WolframRhodium]

You could manually load these dlls in .vpy by

from ctypes import WinDLL
WinDLL(dll1)
WinDLL(dll2)
...

The ordering matters.

[Selur]

Thanks, I will give it a try.

[Selur]

import os
import site
# load libraries for onnxruntime
from ctypes import WinDLL
path = site.getsitepackages()[0]+'/onnxruntime_dlls/'
WinDLL(path+'cublas64_11.dll')
WinDLL(path+'cudart64_110.dll')
WinDLL(path+'cudnn64_8.dll')
WinDLL(path+'cudnn_cnn_infer64_8.dll')
WinDLL(path+'cudnn_ops_infer64_8.dll')
WinDLL(path+'cufft64_10.dll')
WinDLL(path+'cufftw64_10.dll')
seems to work! (moved the libraries to a separte folder 'onnxruntime_dlls')
Hurray!

Thanks !

Cu Selur

[Myrsloik]

Maybe calling this function is more elegant?
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-setdlldirectoryw

[Selur]

@mysloik: you are right using

import site
# Import libraries for onnxruntime
import ctypes
path = site.getsitepackages()[0]+'/onnxruntime_dlls/'
ctypes.windll.kernel32.SetDllDirectoryW(path)

is more elegant.

@all:
For TensorRT users:

import site
# Import libraries for onnxruntime
from ctypes import WinDLL
path = site.getsitepackages()[0]+'/onnxruntime_dlls/'
WinDLL(path+'cublas64_11.dll')
WinDLL(path+'cudart64_110.dll')
WinDLL(path+'cudnn64_8.dll')
WinDLL(path+'cudnn_cnn_infer64_8.dll')
WinDLL(path+'cudnn_ops_infer64_8.dll')
WinDLL(path+'cufft64_10.dll')
WinDLL(path+'cufftw64_10.dll')
WinDLL(path+'nvinfer.dll')
WinDLL(path+'nvinfer_plugin.dll')
WinDLL(path+'nvparsers.dll')
WinDLL(path+'nvonnxparser.dll')

seems to work.

Cu Selur

[Selur]

Argh,... vs-dpir in combination with vs-rife fails.

When using just vs-rife:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/DeinterlaceFilter/TIVTC/libtivtc.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/SourceFilter/d2vSource/d2vsource.dll")
# source: 'C:\Users\Selur\Desktop\VTS_01_1.VOB'
# current color space: YUV420P8, bit depth: 8, resolution: 720x480, fps: 29.97, color matrix: 470bg, yuv luminance scale: limited, scanorder: telecine
# Loading C:\Users\Selur\Desktop\VTS_01_1.VOB using D2VSource
clip = core.d2v.Source(input="E:/Temp/vob_941fdaaeda22090766694391cc4281d5_853323747.d2v")
# Setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=5)
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 29.970
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Deinterlacing using TIVTC
clip = core.tivtc.TFM(clip=clip)
clip = core.tivtc.TDecimate(clip=clip, mode=7, rate=10, dupThresh=0.04, vidThresh=3.50, sceneThresh=15.00)# new fps: 10
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
clip = core.misc.SCDetect(clip=clip,threshold=0.150)
from vsrife import RIFE
# adjusting color space from YUV420P8 to RGBS for VsTorchRIFE
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# adjusting frame count&rate with RIFE (torch)
clip = RIFE(clip, multi=3, device_type='cuda', device_index=0) # new fps: 20
# adjusting output color from: RGBS to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 30.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=30, fpsden=1)
# Output
clip.set_output()

everything works.
But when I add latest vs-dpir:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import os
import site
# Import libraries for onnxruntime
from ctypes import WinDLL
path = site.getsitepackages()[0]+'/onnxruntime_dlls/'
WinDLL(path+'cublas64_11.dll')
WinDLL(path+'cudart64_110.dll')
WinDLL(path+'cudnn64_8.dll')
WinDLL(path+'cudnn_cnn_infer64_8.dll')
WinDLL(path+'cudnn_ops_infer64_8.dll')
WinDLL(path+'cufft64_10.dll')
WinDLL(path+'cufftw64_10.dll')
WinDLL(path+'nvinfer.dll')
WinDLL(path+'nvinfer_plugin.dll')
WinDLL(path+'nvparsers.dll')
WinDLL(path+'nvonnxparser.dll')
# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/DeinterlaceFilter/TIVTC/libtivtc.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/SourceFilter/d2vSource/d2vsource.dll")
# source: 'C:\Users\Selur\Desktop\VTS_01_1.VOB'
# current color space: YUV420P8, bit depth: 8, resolution: 720x480, fps: 29.97, color matrix: 470bg, yuv luminance scale: limited, scanorder: telecine
# Loading C:\Users\Selur\Desktop\VTS_01_1.VOB using D2VSource
clip = core.d2v.Source(input="E:/Temp/vob_941fdaaeda22090766694391cc4281d5_853323747.d2v")
# Setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=5)
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 29.970
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Deinterlacing using TIVTC
clip = core.tivtc.TFM(clip=clip)
clip = core.tivtc.TDecimate(clip=clip, mode=7, rate=10, dupThresh=0.04, vidThresh=3.50, sceneThresh=15.00)# new fps: 10
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
from vsdpir import DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=15.000, task="denoise", provider=1, device_id=0)
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, matrix_s="470bg", range_s="limited")
clip = core.misc.SCDetect(clip=clip,threshold=0.150)
from vsrife import RIFE
# adjusting color space from YUV444P16 to RGBS for VsTorchRIFE
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# adjusting frame count&rate with RIFE (torch)
clip = RIFE(clip, multi=3, device_type='cuda', device_index=0) # new fps: 20
# adjusting output color from: RGBS to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 30.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=30, fpsden=1)
# Output
clip.set_output()


I get:

Python exception: [WinError 127] Die angegebene Prozedur wurde nicht gefunden. Error loading "I:\Hybrid\64bit\Vapoursynth\Lib/site-packages\torch\lib\cudnn_cnn_train64_8.dll" or one of its dependencies.

Using just vs-dpir:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import os
import site
# Import libraries for onnxruntime
from ctypes import WinDLL
path = site.getsitepackages()[0]+'/onnxruntime_dlls/'
WinDLL(path+'cublas64_11.dll')
WinDLL(path+'cudart64_110.dll')
WinDLL(path+'cudnn64_8.dll')
WinDLL(path+'cudnn_cnn_infer64_8.dll')
WinDLL(path+'cudnn_ops_infer64_8.dll')
WinDLL(path+'cufft64_10.dll')
WinDLL(path+'cufftw64_10.dll')
WinDLL(path+'nvinfer.dll')
WinDLL(path+'nvinfer_plugin.dll')
WinDLL(path+'nvparsers.dll')
WinDLL(path+'nvonnxparser.dll')
# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/DeinterlaceFilter/TIVTC/libtivtc.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/SourceFilter/d2vSource/d2vsource.dll")
# source: 'C:\Users\Selur\Desktop\VTS_01_1.VOB'
# current color space: YUV420P8, bit depth: 8, resolution: 720x480, fps: 29.97, color matrix: 470bg, yuv luminance scale: limited, scanorder: telecine
# Loading C:\Users\Selur\Desktop\VTS_01_1.VOB using D2VSource
clip = core.d2v.Source(input="E:/Temp/vob_941fdaaeda22090766694391cc4281d5_853323747.d2v")
# Setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=5)
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 29.970
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Deinterlacing using TIVTC
clip = core.tivtc.TFM(clip=clip)
clip = core.tivtc.TDecimate(clip=clip, mode=7, rate=10, dupThresh=0.04, vidThresh=3.50, sceneThresh=15.00)# new fps: 10
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
from vsdpir import DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=15.000, task="denoise", provider=1, device_id=0)
# adjusting output color from: RGBS to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 10.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=10, fpsden=1)
# Output
clip.set_output()

works fine.

-> Doe anyone have an idea how to could fix this?

Cu Selur

Ps.: also posted it to the vs-rife bug tracker https://github.com/HolyWu/vs-rife/issues/15
PPs.: Same issue happens when having vs-realesrgan and vs-rife in the same script.
PPPs.: using

import site
# Import libraries for onnxruntime
import ctypes
path = site.getsitepackages()[0]+'/onnxruntime_dlls/'
ctypes.windll.kernel32.SetDllDirectoryW(path)

instead of loading the libraries one at a time doesn help.

[_Al_]

... wrong thread

[Selur]

No clue, what I did to my system, but now vs-dpir and vs-rife work together,...

[Selur]

Small warning to others: Current onnxruntime 3.11 (https://github.com/microsoft/onnxruntime) does not support Python 3.10 so if you want to use vs-dpir or vs-realesrgan do not switch to Vapoursynth R58 (which uses Python 3.10) until onnxruntime 3.12 (which should support Pyhton 3.10) is out.

Cu Selur

[ChaosKing]

I thought R58 supports 3.10 and 3.8 ?

[Selur]

Yes, you are right. One might get it working with Python 3.8, but since I used 3.9 before with Vapoursynth R53 (switched to 3.10 since 3.9 isn't supported by Vapoursynth R54), I don't really plan to downgrade Python.

Cu Selur

[knumag]

Had to change it to this, to make it work with R57 and python 3.9:

python -m pip install torch===1.11.0+cu113 torchvision==0.12.0+cu113 -f https://download.pytorch.org/whl/torch_stable.html
python -m pip install vsrife==1.3.0
python -m pip install vsdpir==1.6.0
python -m pip install vsrealesrgan==1.2.0

With R58 and python 3.8 I got the newest versions of vsrife, vsdpir and vsrealesrgan with onnx to work.

Same FPS when converting in both installs, but GPU use is 90-100% with latest versions and 10% with the above versions..
Do not understand why it is so slow and why the GPU usage differs so much when fps is the same.
With Topaz and other similar programs it is around 10x as fast as this.
And also got sharper ouput from VSGAN with same model. Very strange. Some other filters working there?

[Selur]

With R58 and python 3.8 I got the newest versions of vsrife, vsdpir and vsrealesrgan with onnx to work.
Good to know. :)

No clue about speed&co

[Selur]

btw. R58&R59 with Python 3.10 work fine with the ort-nightly, so instead of using:
python -m pip install --upgrade onnxruntime-gpu
using
python -m pip install --upgrade ort-nightly
works fine so far. :)

[PatchWorKs]

...a Colab Notebook would be great...

https://github.com/AlphaAtlas/VapourSynthColab
https://github.com/kodxana/VapourSynthColab