does vs-rife need cuda toolkit?

[Milardo]

Hi I posted this here:

https://github.com/HolyWu/vs-rife/issues/24

Getting an error "RIFE: cuda is not available"

Someone has mentioned that I need to install cuda toolkit.

However it's also mentioned that the latest vs-rife includes necessary runtime libraries.

Anyone has used this encounter this error and what to do?

I have followed this and tried to use it with mpc-be but keep encountering the above error.

https://www.svp-team.com/forum/viewtopic.php?pid=81456#p81456

Thanks for any good replies and i can provide more info if needed.

[Selur]

https://github.com/styler00dollar/VapourSynth-RIFE-ncnn-Vulkan would probably be easier to use,...

[Milardo]

https://github.com/styler00dollar/VapourSynth-RIFE-ncnn-Vulkan would probably be easier to use,...

Hi, I have a gtx 1660 and i read that vs-rife could have better performance than vs-mlrt.

So I'd like to try it

[Selur]

Okay, then make sure to:
a. set CUDA_MODULE_LOADING to "LAZY"
b. load the dependencies like "CUDA Toolkit 11.7", "cuDNN 8.6", "TensorRT 8.5.2.2"
In my (portable) setup, a script where I use vs-rife looks like:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import site
import os
import ctypes
# Adding torch dependencies to PATH
path = site.getsitepackages()[0]+'/torch_dependencies/bin/'
ctypes.windll.kernel32.SetDllDirectoryW(path)
path = path.replace('\\', '/')
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]
os.environ["CUDA_MODULE_LOADING"] = "LAZY"
# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/fmtconv.dll") # for color conversions
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll") # for scene change detection
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll") # to load the source
# source: 'G:\TestClips&Co\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/test.avi", format="YUV420P8", stream_index=0, cache=0, fpsnum=25, prefer_hw=0)
# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip, _Matrix=5)
# Setting color transfer info (470bg), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
# Setting color primaries info (), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
from vsrife import RIFE
# adjusting color space from YUV420P8 to RGBH for vsTorchRIFE
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="470bg", range_s="limited")
# adjusting frame count&rate with RIFE (torch)
clip = RIFE(clip, model="4.6", sc_threshold=0.150) # new fps: 50
# Resizing using 10 - bicubic spline
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, range_s="limited")
clip = core.fmtc.resample(clip=clip, kernel="spline16", w=1280, h=704, interlaced=False, interlacedd=False) # resolution 1280x704
# adjusting output color from: RGBS to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited", dither_type="error_diffusion")
# set output frame rate to 50fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=50, fpsden=1)
# Output
clip.set_output()
where, the 'torch_dependencies/bin'-folder contains, all the need dependencies:
cublas64_11.dll
cublasLt64_11.dll
cudart64_110.dll
cudnn64_8.dll
cudnn_adv_infer64_8.dll
cudnn_cnn_infer64_8.dll
cudnn_ops_infer64_8.dll
cufft64_10.dll
cufftw64_10.dll
curand64_10.dll
cusolver64_11.dll
cusolverMg64_11.dll
cusparse64_11.dll
nvblas64_11.dll
nvinfer.dll
nvinfer_builder_resource.dll
nvinfer_plugin.dll
nvonnxparser.dll (iirc. this isn't needed for vs-rife, but for some other filter I use)
nvparsers.dll
nvrtc-builtins64_117.dll
nvrtc64_112_0.dll
zlibwapi.dll

However it's also mentioned that the latest vs-rife includes necessary runtime libraries.
Where is that mentioned? Never saw it and at least for me that isn't the case.

Cu Selur

[Milardo]

Okay, then make sure to:
a. set CUDA_MODULE_LOADING to "LAZY"
b. load the dependencies like "CUDA Toolkit 11.7", "cuDNN 8.6", "TensorRT 8.5.2.2"
In my (portable) setup, a script where I use vs-rife looks like:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import site
import os
import ctypes
# Adding torch dependencies to PATH
path = site.getsitepackages()[0]+'/torch_dependencies/bin/'
ctypes.windll.kernel32.SetDllDirectoryW(path)
path = path.replace('\\', '/')
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]
os.environ["CUDA_MODULE_LOADING"] = "LAZY"
# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/Support/fmtconv.dll") # for color conversions
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll") # for scene change detection
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll") # to load the source
# source: 'G:\TestClips&Co\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/test.avi", format="YUV420P8", stream_index=0, cache=0, fpsnum=25, prefer_hw=0)
# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip, _Matrix=5)
# Setting color transfer info (470bg), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
# Setting color primaries info (), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive
from vsrife import RIFE
# adjusting color space from YUV420P8 to RGBH for vsTorchRIFE
clip = core.resize.Bicubic(clip=clip, format=vs.RGBH, matrix_in_s="470bg", range_s="limited")
# adjusting frame count&rate with RIFE (torch)
clip = RIFE(clip, model="4.6", sc_threshold=0.150) # new fps: 50
# Resizing using 10 - bicubic spline
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, range_s="limited")
clip = core.fmtc.resample(clip=clip, kernel="spline16", w=1280, h=704, interlaced=False, interlacedd=False) # resolution 1280x704
# adjusting output color from: RGBS to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited", dither_type="error_diffusion")
# set output frame rate to 50fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=50, fpsden=1)
# Output
clip.set_output()
where, the 'torch_dependencies/bin'-folder contains, all the need dependencies:
cublas64_11.dll
cublasLt64_11.dll
cudart64_110.dll
cudnn64_8.dll
cudnn_adv_infer64_8.dll
cudnn_cnn_infer64_8.dll
cudnn_ops_infer64_8.dll
cufft64_10.dll
cufftw64_10.dll
curand64_10.dll
cusolver64_11.dll
cusolverMg64_11.dll
cusparse64_11.dll
nvblas64_11.dll
nvinfer.dll
nvinfer_builder_resource.dll
nvinfer_plugin.dll
nvonnxparser.dll (iirc. this isn't needed for vs-rife, but for some other filter I use)
nvparsers.dll
nvrtc-builtins64_117.dll
nvrtc64_112_0.dll
zlibwapi.dll


Where is that mentioned? Never saw it and at least for me that isn't the case.

Cu Selur

It's on the main github page for vs-rife:

"
README.md
RIFE

Real-Time Intermediate Flow Estimation for Video Frame Interpolation, based on https://github.com/hzwer/Practical-RIFE.
Dependencies

NumPy
PyTorch 1.13
VapourSynth R55+

trt requires additional runtime libraries:

CUDA Toolkit 11.7
cuDNN 8.6
TensorRT 8.5.2.2

For ease of installation on Windows, you can download the 7z file on Releases which contains required runtime libraries and Python wheel file. Either add the unzipped directory to your system PATH or copy the DLL files to a directory which is already in your system PATH. Finally pip install the Python wheel file."


Also, how do you set "CUDA_MODULE_LOADING to "LAZY" ?

[Selur]

Also, how do you set "CUDA_MODULE_LOADING to "LAZY" ?
look at my script.

About the runtime files: CUDA-11.7_cuDNN-8.6.0_TensorRT-8.5.2.2_win64.7z contains them, you just need to be sure that they are used. Since I didn't want to copy them into my path, I extended the PATH variable in my script.

[Milardo]

look at my script.

About the runtime files: CUDA-11.7_cuDNN-8.6.0_TensorRT-8.5.2.2_win64.7z contains them, you just need to be sure that they are used. Since I didn't want to copy them into my path, I extended the PATH variable in my script.


Where do I put the "torch_dependencies/bin" folder so it can be found?

[Selur]

I set it to:
path = site.getsitepackages()[0]+'/torch_dependencies/bin/'
where 'site.getsitepackages()[0]' is usually the Vapoursynth home folder, but you can put it everywhere, by changing the assignment.
Using:

import vapoursynth as vs
core = vs.core
import site
clip = core.std.BlankClip(format=vs.RGB24, color=[255, 255, 255])
clip = core.text.Text(clip, site.getsitepackages()[0])
clip.set_output()

you can see where 'site.getsitepackages()[0]' points to on your system.

I kept the 'torch_dependencies/bin' folder inside the Vapoursynth-folder to keep it portable. :)

Cu Selur

[Milardo]

I set it to:
path = site.getsitepackages()[0]+'/torch_dependencies/bin/'
where 'site.getsitepackages()[0]' is usually the Vapoursynth home folder, but you can put it everywhere, by changing the assignment.
Using:

import vapoursynth as vs
core = vs.core
import site
clip = core.std.BlankClip(format=vs.RGB24, color=[255, 255, 255])
clip = core.text.Text(clip, site.getsitepackages()[0])
clip.set_output()

you can see where 'site.getsitepackages()[0]' points to on your system.

I kept the 'torch_dependencies/bin' folder inside the Vapoursynth-folder to keep it portable. :)

Cu Selur

I have a portable python setup with portable vapoursynth, it's not recognizing the dependencies folder.

Is the vapoursynth home folder in the python folder if it is portable?

Here is a screenshot of what I am getting as an error without the dependency folder being recognized.

[Selur]

Can't see the image since it's pending for approval.
The script I posted should show the first entry in the side packages.
Placing the 'torch_dependencies/bin' folder there and using:
import site
import ctypes
path = site.getsitepackages()[0]+'/torch_dependencies/bin/'
ctypes.windll.kernel32.SetDllDirectoryW(path)
should do the job.

Cu Selur

[Milardo]

Can't see the image since it's pending for approval.
The script I posted should show the first entry in the side packages.
Placing the 'torch_dependencies/bin' folder there and using:
import site
import ctypes
path = site.getsitepackages()[0]+'/torch_dependencies/bin/'
ctypes.windll.kernel32.SetDllDirectoryW(path)
should do the job.

Cu Selur

I don't know why but it's not paying attention the standalone dependencies folder.

Here is the image i tried posting yesterday

https://i.ibb.co/vYQFR8B/2023-05-20-21-36-36-cvt-gw1-avi-MPC-BE-x64-1-5-8.jpg (https://ibb.co/Dp8JTnb)
gif upload (https://imgbb.com/)

[Selur]

What does:
import vapoursynth as vs
core = vs.core
import site
clip = core.std.BlankClip(format=vs.RGB24, color=[255, 255, 255])
clip = core.text.Text(clip, site.getsitepackages()[0])
clip.set_output()
show for you?
What does your Vapoursynth script look like?
lib/site-packages normally isn't in the PATH, so unless you add it to the 'SetDllDirectoryW' this won't work.
Seems like you are trying to load those dlls as if they were normal filters.
(also make sure you use the latest NVIDIA drivers)

Cu Selur

[Milardo]

What does:
import vapoursynth as vs
core = vs.core
import site
clip = core.std.BlankClip(format=vs.RGB24, color=[255, 255, 255])
clip = core.text.Text(clip, site.getsitepackages()[0])
clip.set_output()
show for you?
What does your Vapoursynth script look like?
lib/site-packages normally isn't in the PATH, so unless you add it to the 'SetDllDirectoryW' this won't work.
Seems like you are trying to load those dlls as if they were normal filters.
(also make sure you use the latest NVIDIA drivers)

Cu Selur

The output is the screenshot below:

https://i.ibb.co/7NdLdKG/2023-05-22-19-44-10-cvt-gw1-avi-MPC-BE-x64-1-5-8.jpg (https://ibb.co/3Y9Q9c0)

Here is my vapoursynth script:



import vapoursynth as vs
from vsrife import RIFE

core = vs.core
import site
import os

# Adding torch dependencies to PATH
import ctypes
path = site.getsitepackages()[0]+'/torch_dependencies/bin/'
ctypes.windll.kernel32.SetDllDirectoryW(path)
path = path.replace('\\', '/')
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]
os.environ["CUDA_MODULE_LOADING"] = "LAZY"
ret = VpsFilterSource

ret = core.resize.Bicubic(clip=ret, format=vs.RGBS, matrix_in_s="470bg", range_s="full")

ret = RIFE(clip=ret, model='4.6', trt=True, factor_num=2, factor_den=1)

output_ret = core.resize.Bicubic(clip=ret, format=vs.YUV420P8, matrix_s="709")

output_ret.set_output()




VpsFilterSource is from here:

https://github.com/CrendKing/avisynth_filter

I have the latest nvidia gpu drivers.

[Selur]

Okay, so the first site package points to 'C:\Users\StarFox\Palemoondownload\testing\pything-3.10.9-embeded-am64' and since you used 'getsitepackages()[0]+'/torch_dependencies/bin/'
your 'torch_dependencies/bin'-folder should be under 'C:\Users\StarFox\Palemoondownload\testing\pything-3.10.9-embeded-am64/torch_dependencies/bin/', is this the case?
Since your RIFE call contains 'trt=True', did you install the pytorch and the TensorRT whl ?
To install the whl, copy the tensorrt-8.5.2.2-cp310-none-win_amd64.whl to your Vapoursynth folder, open a terminal in it and call:
python -m pip install tensorrt-8.5.2.2-cp310-none-win_amd64.whl
(I assumed you installed pip)

Cu Selur