vs-basicvsr

[poisondeathray]

Yet another contribution from HolyWu!

Thanks

https://github.com/HolyWu/vs-basicvsr

This one is true temporal SR for video, not single images

Also has tiling option for low GPU memory vs. task

Nice!

[Zarxrax]

What is this difference between vs-basicvsr, vs-realesrgan, and VSGAN?
As far as I can see, it looks like vs-basicvsr & vs-realesrgan have built-in models that you can use, wheras VSGAN lets you just use whatever model you want.
Is there something else that I am missing?

[poisondeathray]

What is this difference between vs-basicvsr, vs-realesrgan, and VSGAN?
As far as I can see, it looks like vs-basicvsr & vs-realesrgan have built-in models that you can use, wheras VSGAN lets you just use whatever model you want.
Is there something else that I am missing?

Yes,

other differences - vsgan implementation of ESRGAN does not have tiling (image is split up with padding and stitched up back togethe) - this helps when your GPU has low memory compared to image size; some other ESRGAN implementations have tiling. Quality wise you cannot tell the difference with large enough padding even zoomed in on still images - although it is detectable with amplified differences.

basicvsr is temporal - in general there is less aliasing and temporal flicker. It's meant for video and works on more than 1 frame. The others are meant for still images (except tecogan, but that does not have a VPY implementation). Single image based SR has less temporal consistency. Unfortunately I'm finding the 3 models provided for basicvsr generally only work for fairly clean sources. Just like with other GAN's you generally have to pre and post process to "massage" the input for a suitable model. It would be nice if you had a specific model for everything and thousands of different situations - I don't see it happening

But users can train their own models for basicvsr and realesrgan - the training code is published - but there isn't a public repository for them - only the published paper models are readily available. In theory you should be able to use other pre-trained models too, with minor adjustment to the script

[aegisofrime]

HolyWu is on a roll, holy molly.

Hoping to see VS implementations of AMD FSR or Intel XeSS next :D

[kedautinh12]

HolyWu is on a roll, holy molly.

Hoping to see VS implementations of AMD FSR or Intel XeSS next :D

I think Nvidia dlss is better but sad it's close source

[aegisofrime]

I think Nvidia dlss is better but sad it's close source

Yeah, that was the reason I didn't mention DLSS, unless nvidia decides to open source it due to competitive pressure. :D

[edcrfv94]

Yeah, that was the reason I didn't mention DLSS, unless nvidia decides to open source it due to competitive pressure. :D

DLSS need motion vector.

[kedautinh12]

DLSS need motion vector.

Yes, DLSS only for RTX card

[Quadratic]

HolyWu is on a roll, holy molly.

Hoping to see VS implementations of AMD FSR or Intel XeSS next :D

https://github.com/GPUOpen-Effects/FidelityFX-FSR/blob/master/ffx-fsr/ffx_fsr1.h
AMD FSR (EASU) is quite interesting, I would imagine one would probably be able to approximate the results with existing tools - but an actual implementation would be nice.

[Julek]

DLSS even if open would not be easy to port, the models are trained specifically for each game, there is no generic model.
But if you have an RTX card, you can at least use DLISR, via the Akarin plugin.
https://github.com/AkarinVS/vapoursynth-plugin#dlisr

[aegisofrime]

DLSS even if open would not be easy to port, the models are trained specifically for each game, there is no generic model.
But if you have an RTX card, you can at least use DLISR, via the Akarin plugin.
https://github.com/AkarinVS/vapoursynth-plugin#dlisr

OMG, I have been looking for a DLSS VS plugin for ages now, but I guess I either didn't bother Googling for it or my Google skills suck. Thanks for linking this! I wonder how many other plugins lie under my nose...

[Selur]

vs-basicvsr
Yet another contribution from HolyWu!
Nice, and there goes the weekend,.. ;)

[edcrfv94]

DLSS even if open would not be easy to port, the models are trained specifically for each game, there is no generic model.
But if you have an RTX card, you can at least use DLISR, via the Akarin plugin.
https://github.com/AkarinVS/vapoursynth-plugin#dlisr

DLSS 2.0 is generic model, but still need motion vector.
This means it cannot be used for video.

[aegisofrime]

DLSS 2.0 is generic model, but still need motion vector.
This means it cannot be used for video.

According to the plugin description, it is using NGX though which is indeed designed for video applications.

[kedautinh12]

According to the plugin description, it is using NGX though which is indeed designed for video applications.

I think DLSS don't same DLISR

[Selur]

Btw. has anyone an example where BasicVSR show some impressive results when starting from a 480p or 576p source?
Normal resizing using i.e. Lanczos and adding some contrast sharpening seems to produce more impressive results.
Only thing that I found is that it does do well is upscaling QCIF content (https://gleitz.info/forum/index.php?thread/48279-super-resolution-erfahrungen/&postID=465168#post465168), and I wonder whether that is it's goal.

Cu Selur

[ChaosKing]

@Selur I think the link is broken.

[Selur]

Sorry, fixed now.

[kedautinh12]

The link work for me but almost German

[Reel.Deel]

The link work for me but almost German

That is because is a German forum :p

[Selur]

The interessting part are the screenshots, so the language shouldn't be an issue. Not much text so, Google Translate or similar should work too. ;)

Cu Selur

[poisondeathray]

Btw. has anyone an example where BasicVSR show some impressive results when starting from a 480p or 576p source?
Normal resizing using i.e. Lanczos and adding some contrast sharpening seems to produce more impressive results.
Only thing that I found is that it does do well is upscaling QCIF content (https://gleitz.info/forum/index.php?thread/48279-super-resolution-erfahrungen/&postID=465168#post465168), and I wonder whether that is it's goal.



BasicVSR works ok in simular situations to where the ESRGAN base model works ok - ie. fairly clean, downsampled sources - really an oversampled SD source... Which is NOT what people typically have - it's usually some noisy, soft DVD source, or SD crappy home video that people want to upscale. But those clean datasets are what most publically available algorithms were trained on: Clean HR ground truth source, downscaled by 4. Usually with bicubic. BasicVSR used selections from the Reds4 and Vimeo90k datasets. qcif "suzie" is really an oversampled source. If you take some ok quality UHD, and downscale it, that will work too. What's needed is training of specific models that take account for noise, degredation, compression artifacts like mpeg2 artifacts, etc... stuff like bad YT compression and degredation after the downscaling step. There are specific datasets that include noise - but they are synthetic noise - and they weren't used for these default models .

So in those situations where the model is applicable - BasicVSR is going to produce more detailed images than , say lanczos3 + any form of sharpen, but less detailed results than default ESRGAN or ESRGAN+ models on single image examination. On single image, ESRGAN+ is going to look more "wow" in applicable source situations. The main difference is BasicVSR is more consistent temporally, when you look at actual video, or more than 1 image. It resembles tecogan in that regard, but produces better results detail wise than the default tecogan model (which isn't the same as the tecogan published paper model). For all single image SR approaches you often need to use some type of temporal smoothing to make it decent for video purposes (and that smoothing reduces the fine details too) , otherwise you are prone to flickering aliasing artifacts, objects changing shapes (as single image prediction is slightly different on different frames) .

So the question is can you use something like ESRGAN, add some QTGMC in progressive mode - to get similar temporally consistent results to BasicVSR, but hopefully keep more fine details? The answer is generally no. The artifacts are too "baked" in - that's the strength of basicvsr

[poisondeathray]

Some apng comparisons. They should animate and loop in most browsers

1:1 non-AR corrected crops, unfiltered. 720x480 => 2880x1920.

Note the temporal flickering artifacts (aliasing flicker, and prediction flicker).

You can see the earrings look like random lightning in the ESRGAN+ results.

https://i.postimg.cc/PxD42rQR/shimmer-compare-apng.png (https://postimages.org/)

No amount of sharpening will transform lanczos3 into as detailed as BasicVSR. No amount of temporal filtering will transform lanczos3, or ESRGAN+ results into as temporally consistent results as BasicVSR

https://i.postimg.cc/FsNZVvdT/stillface-compare-apng.png (https://postimages.org/)


The license plate, you can make out the numbers/letters with lanczos, but it's much more clear in the BasicVSR. It doesn't take a stretch of the imagination that slightly smaller or a more distant plate might be illegible when processed with lanczos , but partially readable by BasicVSR

https://i.postimg.cc/qgVxPfjv/compare-licenseplate-apng.png (https://postimages.org/)

ESRGAN and psnr model mess up letters/numbers/text (many GAN, NN models do), I won't include in the animated temporal demo, but here is a still image compare animated demo. ESRGAN default does good on BG foliage textures, but messes up text. None of the GAN scalers have "fuzzy" lines or bad aliasing - that's what they excel at. Early gen NNEDI/1/2/3 is NN (neural net) based, but signficantly worse in terms of aliasing artifacts for upscaling 4x than modern GAN scalers


https://i.postimg.cc/mr4BbVmX/stillcompare-licenseplate-apng.png (https://postimages.org/)


But these are cleanly downsample source examples, much like the training set. ie. They don't have MPEG2 macroblocks or or grain plate added on top like you might with some film DVD

Some trained degraded jpeg ESRGAN models work ok for single image jpeg interpolation - I would like to see more BasicVSR models that utilize some typical degraded aspects of common video

[lansing]

The car plate image is amazing

[poisondeathray]

Keep in mind those are "textbook" clean sources . Normal dashcam / security cam footage would have a bunch of data loss, compression artifacts. The training set for the 3 base models do not account for that

There is an improved version, BasicVSR++ , but the code has not been released yet . I hope HolyWu brings it to vapoursynth when it's released

[Selur]

@poisondeathray: Thanks for the explaination, that explains why I didn't get and impressive results from my usual content. :)

[Selur]

There is an improved version, BasicVSR++ ,..
Looks impressive: https://ckkelvinchan.github.io/projects/BasicVSR++/

I hope HolyWu brings it to vapoursynth when it's released
fingers crossed ;)

Cu Selur

[edcrfv94]

Keep in mind those are "textbook" clean sources . Normal dashcam / security cam footage would have a bunch of data loss, compression artifacts. The training set for the 3 base models do not account for that

There is an improved version, BasicVSR++ , but the code has not been released yet . I hope HolyWu brings it to vapoursynth when it's released

released 10 days ago
https://github.com/open-mmlab/mmediting/blob/master/mmedit/models/backbones/sr_backbones/basicvsr_pp.py

MMEditing v0.10.0 Release
Support LIIF-RDN (CVPR'2021)
Support BasicVSR++ (NTIRE'2021)
https://github.com/open-mmlab/mmediting/releases/tag/v0.10.0

[poisondeathray]

released 10 days ago
https://github.com/open-mmlab/mmediting/blob/master/mmedit/models/backbones/sr_backbones/basicvsr_pp.py

MMEditing v0.10.0 Release
Support LIIF-RDN (CVPR'2021)
Support BasicVSR++ (NTIRE'2021)
https://github.com/open-mmlab/mmediting/releases/tag/v0.10.0

Nice!! :D

[aegisofrime]

@poisondeathray: Could I ask what sort of speed are you getting in your test? Upscaling just from 720x480 -> 1280x720 I'm getting like 0.05fps :scared: with a Ryzen 5950X + RTX 3060Ti, and I'm wondering if I'm doing something wrong.

[poisondeathray]

@poisondeathray: Could I ask what sort of speed are you getting in your test? Upscaling just from 720x480 -> 1280x720 I'm getting like 0.05fps :scared: with a Ryzen 5950X + RTX 3060Ti, and I'm wondering if I'm doing something wrong.

Similar speeds - it's very slow.

[gugglu]

Hi folks, i am trying to test the vsbasicVSR but whenever i try to load the BasicVSR it says cuda not available. any hint what i am missing, tia.

script using to testimport vapoursynth as vs
import numpy
import torch
from vsbasicvsr import BasicVSR

core = vs.get_core()

src = core.dgdecodenv.DGSource(r'C:\Users\User89\Desktop\test\00000.dgi')
src = core.fmtc.bitdepth(src, bits=32)
src = core.fmtc.resample(src, kernel='spline36', css='444')
src = core.fmtc.matrix(src, mat='709', col_fam=vs.RGB)
ret = BasicVSR(src)

ret.set_output()

I get this error when checking the above scriptTraceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 2242, in vapoursynth.vpy_evaluateScript
File "src\cython\vapoursynth.pyx", line 2243, in vapoursynth.vpy_evaluateScript
File "C:/Users/User89/Desktop/VapourSynthEditor-r19-64bit/Untitled.vpy", line 12, in
src = core.fmtc.matrix(src, mat='709', col_fam=vs.RGB)
File "C:\Users\User89\AppData\Local\Programs\Python\Python39\lib\site-packages\vsbasicvsr\__init__.py", line 56, in BasicVSR
raise vs.Error('BasicVSR: CUDA is not available')
vapoursynth.Error: BasicVSR: CUDA is not available

[poisondeathray]

vapoursynth.Error: BasicVSR: CUDA is not available

Drivers ?

Did you install pytorch with cuda ?

Does cuda work ok for you in other applications, or vapoursynth ? eg. try eedi2cuda
https://github.com/AmusementClub/VapourSynth-EEDI2CUDA

[gugglu]

Drivers ?

Did you install pytorch with cuda ?

Does cuda work ok for you in other applications, or vapoursynth ? eg. try eedi2cuda
https://github.com/AmusementClub/VapourSynth-EEDI2CUDA

Hi poisondeathray, i've installed cuda (cuda_11.0.2_451.48_win10) but on its own here are some scrn.shots of process please have a look.

Cuda installation:
https://prnt.sc/1qyobff
https://prnt.sc/1qyocvh
https://prnt.sc/1qyodye
https://prnt.sc/1qyp184
https://prnt.sc/1qyp3ly
installed Drivers:
https://prnt.sc/1qyp9ae
and pytorch was installed with vsbasicvsr via pip as instructed on a github .
https://prnt.sc/1qyrk22
yes EEDi2CUDA works fine
https://prnt.sc/1qyrd78

:thanks:

[poisondeathray]

H
and pytorch was installed with vsbasicvsr via pip as instructed on a github .
https://prnt.sc/1qyrk22



You're missing pytorch with cuda . You only have pytorch cpu

There should be a folder torch-xxx-cuxxx

Where "xxx" is the version number, cu is for cuda

If you go here, and select the boxes that apply to you, it will give you command line to install correct version with cuda
https://pytorch.org/get-started/locally/

[Selur]

Also, if you want I can send you a link to the 'addon' for Hybrid I just uploaded, it's basically a portable Vapoursynth&Python install with:

vsbasicvsr (https://github.com/HolyWu/vs-basicvsr)
vsdpir (https://github.com/HolyWu/vs-dpir)
vsffdnet (https://github.com/HolyWu/vs-ffdnet)
vsgan (https://github.com/rlaPHOENiX/VSGAN) + a bunch of vsgan_models (https://upscale.wiki/wiki/Model_Database)
vsrealesgran (https://github.com/HolyWu/vs-realesrgan/)
vsrife (https://github.com/HolyWu/vs-rife)

packed size ~9GB
if you want a link to the download send me a pm.

Cu Selur

[gugglu]

You're missing pytorch with cuda . You only have pytorch cpu

There should be a folder torch-xxx-cuxxx

Where "xxx" is the version number, cu is for cuda

If you go here, and select the boxes that apply to you, it will give you command line to install correct version with cuda
https://pytorch.org/get-started/locally/

thanks a lot poisondeathray its working after downloading pytorch with cuda with windows command line from a pytorch link

Microsoft Windows [Version 10.0.19042.1165]
(c) Microsoft Corporation. All rights reserved.

C:\Users\User89\AppData\Local\Programs\Python\Python39\Scripts>pip3 install torch==1.9.0+cu102 torchvision==0.10.0+cu102 torchaudio===0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch==1.9.0+cu102
Downloading https://download.pytorch.org/whl/cu102/torch-1.9.0%2Bcu102-cp39-cp39-win_amd64.whl (1440.4 MB)
|████████████████████████████████| 1440.4 MB 2.8 kB/s
Collecting torchvision==0.10.0+cu102
Downloading https://download.pytorch.org/whl/cu102/torchvision-0.10.0%2Bcu102-cp39-cp39-win_amd64.whl (2.0 MB)
|████████████████████████████████| 2.0 MB 3.3 MB/s
Collecting torchaudio===0.9.0
Using cached torchaudio-0.9.0-cp39-cp39-win_amd64.whl (215 kB)
Requirement already satisfied: typing-extensions in c:\users\user89\appdata\local\programs\python\python39\lib\site-packages (from torch==1.9.0+cu102) (3.10.0.2)
Requirement already satisfied: pillow>=5.3.0 in c:\users\user89\appdata\local\programs\python\python39\lib\site-packages (from torchvision==0.10.0+cu102) (8.3.1)
Requirement already satisfied: numpy in c:\users\user89\appdata\local\programs\python\python39\lib\site-packages (from torchvision==0.10.0+cu102) (1.21.2)
Installing collected packages: torch, torchvision, torchaudio
Attempting uninstall: torch
Found existing installation: torch 1.9.0
Uninstalling torch-1.9.0:
Successfully uninstalled torch-1.9.0
Attempting uninstall: torchvision
Found existing installation: torchvision 0.10.0
Uninstalling torchvision-0.10.0:
Successfully uninstalled torchvision-0.10.0
Successfully installed torch-1.9.0+cu102 torchaudio-0.9.0 torchvision-0.10.0+cu102

[gugglu]

Also, if you want I can send you a link to the 'addon' for Hybrid I just uploaded, it's basically a portable Vapoursynth&Python install with:

vsbasicvsr (https://github.com/HolyWu/vs-basicvsr)
vsdpir (https://github.com/HolyWu/vs-dpir)
vsffdnet (https://github.com/HolyWu/vs-ffdnet)
vsgan (https://github.com/rlaPHOENiX/VSGAN) + a bunch of vsgan_models (https://upscale.wiki/wiki/Model_Database)
vsrealesgran (https://github.com/HolyWu/vs-realesrgan/)
vsrife (https://github.com/HolyWu/vs-rife)

packed size ~9GB
if you want a link to the download send me a pm.

Cu Selur

that seems quite interesting, Selur thank you :)

[poisondeathray]

:D

Thanks HolyWu

vs-BasicVSR++
https://github.com/HolyWu/vs-basicvsrpp

[Selur]

nice!

Support either x4 upsampling (for model 0-2) or same size output (for model 3-5).
For model 0-2, input resolution must be at least 64.
For model 3-5, input resolution must be at least 256 and mod-4.
Parameters:
clip: Clip to process. Only planar format with float sample type of 32 bit depth is supported.
model: Model to use.
0 = REDS
1 = Vimeo-90K (BI)
2 = Vimeo-90K (BD)
3 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 1
4 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 2
5 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 3

Is there a difference between BasicVSR and BasicVSR++ if model 0-2 are used, or is it the same resizing as BasicVSR (model 0-2) and additional models for cleaning?

[Selur]

Also did anyone manage to install this in a portable Vapoursynth environment on Windows?
Calling:
python -m pip install --upgrade vsbasicvsrpp
first failed with
ERROR: Could not find a version that satisfies the requirement vapoursynth==54 (from versions: 39, 40, 41, 42, 43, 44, 45, 46, 47, 47.1, 47.2, 48, 49, 50, 51)
ERROR: No matching distribution found for vapoursynth==54
after renaming the dummy 'VapourSynth-53.dist-info', I created to install VSGAN, to 'VapourSynth-54.dist-info', calling:
python -m pip install --upgrade vsbasicvsrpp
failed with:
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
calling:
set CUDA_HOME=I:/Hybrid/64bit/Vapoursynth/Lib/site-packages/torch/cuda (not toally sure this is correct)
and then:
python -m pip install --upgrade vsbasicvsrpp
it fails with:
I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\cpp_extension.py:305: UserWarning: Error checking compiler version for cl: [WinError 2] Das System kann die angegebene Datei nicht finden
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
I get the same error when calling:
python -m pip install mmcv-full==1.3.12 -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.9.0/index.htm

-> that's the point where I gave up, so if anyone figures out how to install vsbasicvsrpp in a protable Vapoursynth environment please let me know. :)

Cu Selur

[poisondeathray]

Is there a difference between BasicVSR and BasicVSR++ if model 0-2 are used, or is it the same resizing as BasicVSR (model 0-2) and additional models for cleaning?

I've only done a few tests so far , but some early observations/comments - basicvsrpp is marginally better with the same model interval size compared to basicvsr. Not a major difference. The default interval size is different, 30 for ++, vs 7*2+1=15

Models 3-5 are from the NTIRE 2021 Quality enhancement of heavily compressed videos Challenge , which take HEVC compressed videos using fixed qp and low bitrate encodings - so those pre-trained models should factor in some compression degredation (at least HEVC type, not necessarily MPEG2, or AVC). It' s nice to see some other types of degradation training and models, but 3 and 5 tend to be very smooth (ie. no detail) . 4 has more detail but more artifacts. Models 3-5 don't upscale

I haven't done enough testing to see if using a much larger interval size helps or hinders in general. It appears a very small interval size is worse. Larger sizes take more memory and are slower

[poisondeathray]

-> that's the point where I gave up, so if anyone figures out how to install vsbasicvsrpp in a protable Vapoursynth environment please let me know. :)




I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\cpp_extension.py:305: UserWarning: Error checking compiler version for cl: [WinError 2] Das System kann die angegebene Datei nicht finden
warnings.warn(f'Error checking compiler version for {compiler}: {error}')


Not sure, I used installed environment, but I had problems at first. My errors msg was slightly different - it needed MS Visual C++ compiler to build the "wheels" to install other components. I'm wondering how the compiler is accessed in a "portable" environment ?

[Selur]

I'm wondering how the compiler is accessed in a "portable" environment ?
No clue either, haven't run into the problem before. :)

Cu Selur

[poisondeathray]

HolyWu added update a few hours ago and made install "easier" on Windows. Maybe try this new one
https://github.com/HolyWu/vs-basicvsrpp


Installing mmcv-full on Windows is a bit complicated as it requires Visual Studio and other tools to compile CUDA ops. So I have uploaded the built file compiled with CUDA 11.1 for Windows users and you can install it by executing the following command.


pip install https://github.com/HolyWu/vs-basicvsrpp/releases/download/v1.0.0/mmcv_full-1.3.12-cp39-cp39-win_amd64.whl

[Selur]

Thanks ! using that call it works for me too. :)

Cu Selur

[Selur]

Has anyone tried https://github.com/HolyWu/vs-swinir ? (didn't want to create a new thread ;))
-> man this is too slow on my machine to be useful for normal usage on my gpu (Geforce GTX 1070ti)

[PatchWorKs]

Normal resizing using i.e. Lanczos and adding some contrast sharpening seems to produce more impressive results.
HolyWu has just ported - after my request - SwinIR (https://github.com/JingyunLiang/SwinIR) to VS, can someone make some "real world" test with it ?

https://github.com/HolyWu/vs-swinir

https://raw.githubusercontent.com/JingyunLiang/SwinIR/main/figs/real_world_image_sr.png

man this is too slow on my machine to be useful for normal usage on my gpu (Geforce GTX 1070ti)
For individual frames (= pics) you can test it here (https://colab.research.google.com/gist/JingyunLiang/a5e3e54bc9ef8d7bf594f6fee8208533/swinir-demo-on-real-world-image-sr.ipynb), but a video-oriented colab notebook like this (https://colab.research.google.com/github/rlaPHOENiX/VSGAN/blob/master/VSGAN.ipynb) would be great (I don't own a discrete GPU at all) !


EDIT
Out of curiosity: do you think the new Apple chips (M1 Pro / Max (https://www.anandtech.com/show/17024/apple-m1-max-performance-review)) could speed up operations?

[Selur]

Out of curiosity: do you think the new Apple chips (M1 Pro / Max) could speed up operations?
Without:
a. pytorch support
b. rewriting of the exitistn plugins
-> no

For individual frames (= pics)
I can run it for single pics fine, but I get like 0.005fps for sd->hd on my system, which simply is too slow for me to be usable.

[ReinerSchweinlin]

I can run it for single pics fine, but I get like 0.005fps for sd->hd on my system, which simply is too slow for me to be usable.
thanx for testing :) How much vram did it use in your case? Is it really that GPU demanding or might the slow-down caused by not enough VRAM ? If you have something a unskilled person like me could use and test, I could throw it into a 12GB VRAM card and see what happens...

[PatchWorKs]

Without:
a. pytorch support
b. rewriting of the exitistn plugins
-> no
Well, pytorch support SEEMS on the go:
https://github.com/pytorch/pytorch/issues/47702


I can run it for single pics fine, but I get like 0.005fps for sd->hd on my system, which simply is too slow for me to be usable.
Of course (that's why a colab can help), but can you please post some visual results ?

Thx !

[Selur]

here are a few examples:
https://i.ibb.co/Pg3nJBm/SwinIR.png (https://ibb.co/2dmQRXn)
used:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DeinterlaceFilter/TIVTC/libtivtc.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/d2vSource/d2vsource.dll")
# source: 'E:\clips\VTS_02_1-Sample-Beginning.demuxed.m2v'
# current color space: YUV420P8, bit depth: 8, resolution: 720x480, fps: 29.97, color matrix: 470bg, yuv luminance scale: limited, scanorder: telecine
# Loading E:\clips\VTS_02_1-Sample-Beginning.demuxed.m2v using D2VSource
clip = core.d2v.Source(input="E:/Temp/m2v_5d36292e1f7f53fd6e26be51d50bbf8c_853323747.d2v")
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 29.97
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# Deinterlacing using TIVTC
clip = core.tivtc.TFM(clip=clip)
clip = core.tivtc.TDecimate(clip=clip)# new fps: 23.976
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
# DEBUG: vsTIVTC changed scanorder to: progressive
# cropping the video to 704x480
clip = core.std.CropRel(clip=clip, left=8, right=8, top=0, bottom=0)
from vsswinir import SwinIR
# adjusting color space from YUV420P8 to RGBS for VsSwinIR
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# resizing using SwinIR
clip = SwinIR(clip=clip, task="real_sr_large", scale=4, tile_x=352, tile_y=240, tile_pad=16, device_type="cuda", device_index=0) # 2816x1920
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1474, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()
https://i.ibb.co/dpykKQ0/Swin-IR-clip-task-classical-sr-scale-4.png (https://ibb.co/z2wPfSh)
https://i.ibb.co/k2D9LtV/Swin-IR-clip-task-lightweight-sr-scale-4.png (https://ibb.co/whQwnfZ)
https://i.ibb.co/51Qb71F/Swin-IR-clip-task-real-sr-scale-4.png (https://ibb.co/vcN0tcY)
https://i.ibb.co/pryQgfC/Swin-IR-clip-task-real-sr-large-scale-4.png (https://ibb.co/3Yzvwpt)

some more using RealSR_large:
https://i.ibb.co/YR6nGnf/swinir-real-sr-large-4.png (https://ibb.co/mDmZ7Zh)
https://i.ibb.co/M5TF29C/swinir-real-sr-large-4-2.png (https://ibb.co/rfPD7vw)
https://i.ibb.co/MSSQt2m/swinir-real-sr-large-4-4.png (https://ibb.co/C88DY1f)

Cu Selur

[PatchWorKs]

Very nice results (especially on the faces), even if - of course - not yet optimal for everything...

...btw I hope to see a SwinIR version optimized for videos too.

[Selur]

not yet optimal for everything...
Espeically the last example shows some real issues ;)

...btw I hope to see a SwinIR version optimized for videos too.
first a way faster version would be needed

[PatchWorKs]

first a way faster version would be needed
Already asked, of course: https://github.com/JingyunLiang/SwinIR/issues/47

Note: I've also just "fed" @HolyWu with this awesome collection, let's see if other interesting "VS-ports" will come out...

[ReinerSchweinlin]

Very nice results (especially on the faces), even if - of course - not yet optimal for everything...

...btw I hope to see a SwinIR version optimized for videos too.
Is a video version planed?

for upscaling, algos like esrgan (single image) are not very suitable for real-life content. Too much flickering. So unless SwinIR doesn´t get some extensions for multi-frame usage / flow detection /whatever, one will always get flickering / stutters / inkonsistent movement...

[aegisofrime]

Nvidia just open sourced Nvidia image scaling... Would this be a candidate for the next filter?

[Selur]

Has anyone tested https://github.com/HolyWu/vs-hinet ?

Here are a few screen shots: (not sure what to make of them and for what content this is really useful)

Mode: Deblur GoPro
https://i.ibb.co/0MjBKN3/deblur-Go-Pro.png (https://ibb.co/QNF8kBG)
Mode: Deblur REDS
https://i.ibb.co/WyWdJyG/deblur-REDS.png (https://ibb.co/xjScTjX)
Mode: denoise
https://i.ibb.co/2SP8LJ7/denoise.png (https://ibb.co/pQZWsqv)
Mode: derain
https://i.ibb.co/zP0j9sH/derain.png (https://ibb.co/wpx3Gyr)

[PatchWorKs]

According to your tests on that frame, the highest fidelity seems to be achieved by derain model, btw here are some questions:

how fast is it ?
how does it performs (in terms of both speed and fidelity) compared to xClean (https://forum.doom9.org/showthread.php?t=183245) ?
its description claims that it perform "restoring" (aka denoising ?) function, but since is BasicSR-based does it upscale too ?


Last but not least (even if OT): did you tried RIFE ?
https://github.com/HolyWu/vs-rife

[Selur]

speed: ~2-3fps for sd content, so not that slow
xClean: no clue about xClean, haven't played around with it too many options for my taste (+ would need to add znedi3 and nnedi3cl support to it)
upscale: at least the current interface offers no upscaling and the method does not upscale
rife: yes, I like it (with sceneChange added). Waiting for FrameRateConverter to properly support it

[lansing]

I don't understand, what was the filter supposed to do??

[Selur]

I'm not sure either which is why is simply took a clip and made some screen shots and asked whether someone has experience with it,..from what I have seen so far each of the models has a different effect.
So far I would say:
deblur GoPro: seems to mainly sharpen and 'add details' (seems to causes artifacts on cartoons)
deblur REDS: strop denosing and sharpening (seems to only suited for cartoon content)
denoise: also seems to be useful for really noise vhs sources or cartoons
derain: milder denoise/degrain, unsure where this could be useful, seems to remove less when used on more detailed content,...
-> so no real clue

[poisondeathray]

for hinet: the main purpose is single image deblurring . It's not a temporal algorithm. The type of blur it's most suited to - is mostly camera motion blur / shutter blur.

"derain" is for removing "rain" (it's not "degrain") . ie. short streaks of white, like rain drops. It actually works ok on random dark film scratches (not persistent , long scratches) if you invert the src.

Another semi popular deblurring algorithm is deblurGANv2 (also works best on camera motion blur)

[lansing]

for hinet: the main purpose is single image deblurring . It's not a temporal algorithm. The type of blur it's most suited to - is mostly camera motion blur / shutter blur.

"derain" is for removing "rain" (it's not "degrain") . ie. short streaks of white, like rain drops. It actually works ok on random dark film scratches (not persistent , long scratches) if you invert the src.

Another semi popular deblurring algorithm is deblurGANv2 (also works best on camera motion blur)

Ok thanks. So it's targeting high resolution photo images, not videos.

[poisondeathray]

Ok thanks. So it's targeting high resolution photo images, not videos.

Not necessarily; the gopro model is trained on gopro video . The resolution isn't "high", though

All the single image algorithms tend to have some temporal inconsistencies in the end result when used on video .

YMMV for all these research projects

[PatchWorKs]

Investigating Tradeoffs in Real-World Video Super-Resolution (aka RealBasicVSR) (https://github.com/ckkelvinchan/RealBasicVSR) has just been released:

https://user-images.githubusercontent.com/7676947/143370499-9fe4069b-46cc-4f12-b6ff-5595e8e5e0b8.mp4

https://user-images.githubusercontent.com/7676947/143370350-91f751f3-0f33-4ee4-9b1a-b9279bf41c18.mp4

https://user-images.githubusercontent.com/7676947/143370556-9e7019d4-e718-46af-859f-54d5576cd370.mp4

https://user-images.githubusercontent.com/7676947/143370859-e0293b97-f962-476f-acf8-14fad27cea77.mp4

[kedautinh12]

Wow, interesting results

[Selur]

Doesn't really look better than BasicVSR++ to me atm. but in case someone ports it to Vapoursynth we will see more. :)

[Gravitator]

At the end, stripes appear under the left arm in Titanic.

[PatchWorKs]

Doesn't really look better than BasicVSR++ to me atm. but in case someone ports it to Vapoursynth we will see more. :)
Some other "real world" comparisons done with Colab Demo (https://colab.research.google.com/drive/1JzWRUR34hpKvtCHm84IGx6nv35LCv20J?usp=sharing):
https://user-images.githubusercontent.com/7676947/144365675-5c558a89-8b7f-4af9-a696-228bc73ab56e.png

[Selur]

Here's a nice example what BasicVSR++ can do:
https://i.ibb.co/WHyMhX5/Basic-VSR-Model1.png (https://ibb.co/HKzbSy4)

Cu Selur

[aegisofrime]

Can I get a bit of help here? After finally setting up the dependencies (they are a bit convoluted for this plugin honestly), I keep getting the following error:

vpy [error]: CUDA out of memory. Tried to allocate 1.24 GiB (GPU 0; 8.00 GiB total capacity; 5.08 GiB already allocated; 0 bytes free; 5.21 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation occurred while getting frame 0
x265 [error]: unable to open input file <P:\ISO\MainMovie\ME_021\OME021_temp\OME021.vpy>

My Script is very simple,


clip = core.avs.DGSource(r"D:\ISO\Test.dgi")
#core.max_cache_size=16384
#core.num_threads = 8
import havsfunc as haf
core.max_cache_size=16384
clip = haf.QTGMC(clip, Preset='Placebo', TFF=True, Denoiser="KNLMeansCL")
from vsbasicvsrpp import BasicVSRPP
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
clip = BasicVSRPP(clip,model=1)
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P12, matrix_s="470bg", range_s="limited")
clip = core.resize.Spline36(clip, 1440,960)
clip.set_output()


Input is just a 720x480 DVD video, while my GPU is a RTX 3060Ti, so it only has got 8GB of RAM. It should still be enough right? How can I tweak the arguments so that this works at all?

[Selur]

adding:
,fp16=True
and/or
,tile_x=360,tile_y=480
to the BasicVSRPP code should help.

Cu Selur

[PatchWorKs]

For those - like me - who don't own powerful GPUs, I just asked kodxana (aka Mediator2011) (https://github.com/kodxana):

https://github.com/kodxana/VapourSynthColab/issues/1

[PatchWorKs]

Hi there, can someone make some (performances, particularry) tests of VapourSynth Single Image Super-Resolution Generative Adversarial Network (GAN) (https://github.com/rlaPHOENiX/VSGAN) ?
(note: I tried their colab (https://colab.research.google.com/github/rlaphoenix/VSGAN/blob/master/VSGAN.ipynb) without success)

Thanks in advance !

[Selur]

What do you have in mind?

[PatchWorKs]

What do you have in mind?
Oh, well, it would be great to have a colab "service" where upload (old) SD stuff and download (x265-10bit encoded) 1080p...

[Selur]

Ah okay, sounded like wanted some benachmarks. No clue about colab. :)

[PatchWorKs]

Ah okay, sounded like wanted some benachmarks. No clue about colab. :)
Oh, well, of course it would be great to have some tests of newest algos:

https://github.com/Weepingchestnut/Awesome-Video-Super-Resolution

[poisondeathray]

PP-MSVSR featured in PaddleGAN looks like the next improvement over basicvsr++ . I'm trying to get it to work, but no luck so far with modifying config files and paths

https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US/tutorials/video_super_resolution.md
https://arxiv.org/pdf/2112.02828.pdf

[PatchWorKs]

PP-MSVSR featured in PaddleGAN looks like the next improvement over basicvsr++ . I'm trying to get it to work, but no luck so far with modifying config files and paths
...to me could be VERY interesting too:

OverNet: Lightweight multi-scale super-resolution with overscaling network (https://openaccess.thecvf.com/content/WACV2021/papers/Behjati_OverNet_Lightweight_Multi-Scale_Super-Resolution_With_Overscaling_Network_WACV_2021_paper.pdf)

https://arxiv.org/abs/2008.02382

Unfortunally their GH seems empty:
https://github.com/pbehjatii/OverNet

[ReinerSchweinlin]

thanx for mentioning it,m looking good :)

there is hope, GITHUB says:

This repository is an official PyTorch implementation of the paper "OverNet: Lightweight multi-scale super-resolution with overscaling network" from WACV 2021. The code will be released soon.

[poisondeathray]

I requested VRT: A Video Restoration Transformer

https://github.com/JingyunLiang/VRT

https://github.com/HolyWu/vs-basicvsrpp/issues/21

[PatchWorKs]

Here's another kind of enhancer that claims to make HDR from SDR:
https://github.com/nothinglo/Deep-Photo-Enhancer

https://raw.githubusercontent.com/nothinglo/Deep-Photo-Enhancer/master/images/teaser.png

Results on video:
https://www.youtube.com/watch?v=d7OXb2sqoec

Pytorch implementation:
https://github.com/mtics/deep-photo-enhancer

[Selur]

HolyWu posted an update for vs-dpir. I'm not sure what provider uses what onxruntime.
Okay, got that cleared up, see: https://github.com/HolyWu/vs-dpir/issues/20

CUDA:
encoded 429 frames, 8.83 fps, 1117.06 kb/s
DIRECTML:
encoded 429 frames, 6.03 fps, 1117.74 kb/s
CPU:
encoded 429 frames, 0.49 fps, 1116.08 kb/s

(can't test CUDA TensorRT since I only got an old Geforce GTX 1070ti)
-> seems that onnxruntime-gpu is the way to go for CUDA cards.

[cubicibo]

Here's another kind of enhancer that claims to make HDR from SDR:
https://github.com/nothinglo/Deep-Photo-Enhancer

[Image]

I wouldn't say this is an improvement. Look at the halo around the "PUMA" on the shirt.

[Selur]

Did anyone get vs-dpir installed when using Vapoursynth R58-RC1? (https://github.com/HolyWu/vs-dpir/issues/22)
vs-basicvsrpp also doesn't work since mmcv_full is meant for Python 3.9 which isn't supported in Vapoursynth R58 (only 3.8 and 3.10) (https://github.com/HolyWu/vs-basicvsrpp/issues/23)
Similar for vs-rife and vs-realesrgan.

Cu Selur

[Selur]

Thanks to HolyWu I got vs-basicvsrpp working using:
python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11/index.html
python -m pip install tqdm
python -m pip install opencv-python
python -m pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp
:)

Cu Selur

[ReinerSchweinlin]

Thanks to HolyWu I got vs-basicvsrpp working using:
python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11/index.html
python -m pip install tqdm
python -m pip install opencv-python
python -m pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp
:)

Cu Selur

Nice :) Thanx to you both :)

[PatchWorKs]

Thanks to HolyWu I got vs-basicvsrpp working using:
python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11/index.html
python -m pip install tqdm
python -m pip install opencv-python
python -m pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp
:)

Cu Selur

Holy & Slur contributions & testing tandem rockz !

[PatchWorKs]

How about (new ?) VRT - A Video Restoration Transformer ?

https://github.com/JingyunLiang/VRT

Tests would be great.

[Selur]

Maybe vs-mlrt (https://github.com/AmusementClub/vs-mlrt) will add support for it, based onhttps://docs.microsoft.com/en-us/windows/ai/windows-ml/tutorials/pytorch-convert-model conversion from pth to onnx models might be possible.

[WolframRhodium]

Sadly, ther non-supported operations (deformable convolution and others) in VRT prevent the conversion to the standard onnx format.

[Selur]

argh,.. good to know, thanks for looking into it. :)

[~ VEGETA ~]

is there any good use of this filter on high quality anime encode releases?

I mean if sharpness and detail preservation are wanted, then descaling is the way to go if can be done. Plus some high quality denoisers and careful masked debanding can yield best results.

what can this filter offer?

[Selur]

Assuming you are referring to BasicVSR++, it's meant for restoration so using it on a 'high quality' sounds like wanting to give a new car to a restaurator,...
As a side note: There are tons of Anime trained models (https://upscale.wiki/wiki/Model_Database) for VSGAN which might be more suited, detail preservation totally depends on the model used.
Using ml stuff on high resolution content is really slow with current hardware, so if high quality also means high source resolution be prepared to wait.

Cu Selur

[poisondeathray]

I've modified __init__.py in the vsbasicvsrpp folder to include the NTIRE 4x VSR model if anyone is interested

https://www.mediafire.com/file/k0bbai32yrk7y05/__init__(modified+bvsrpp+to+incl+NTIRE+VSR).zip/file

https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/basicvsr_plusplus

Extract __init__.py from the zip archive ,and download the model and place in same vsbasicvsrpp directory
https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c128n25_ntire_vsr_20210311-1ff35292.pth


model: Model to use.
0 = REDS
1 = Vimeo-90K (BI)
2 = Vimeo-90K (BD)
3 = NTIRE 2021 Video Super-Resolution
4 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 1
5 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 2
6 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 3


I inserted it as model "3" because models 0 to 2 are 4x models, and NTIRE 2021 Video Super-Resolution is a 4x model too; and that bumps the NTIRE 1x models down

The model uses mid_channels 128 and num_blocks 25 (instead of 64 and 7), as written in the pth name, and in the py file, so I modified those entries and the other relevant ones
https://github.com/open-mmlab/mmediting/blob/master/configs/restorers/basicvsr_plusplus/basicvsr_plusplus_c128n25_600k_ntire_vsr.py

It seems to work ok, and results are similar to REDS model, maybe slightly better or on some sources. The py says it was trained on REDS data set with spynet_20210409-c6c1bd09.pth pretrain

[Selur]

Nice.Thanks!
(added a request in the vs-basicvsrpp bugtacker to add this)

[Selur]

btw. HolyWu just release vs-dpir-ncnn version :)

python -m pip install -U vsdpir_ncnn
python -m pip install --upgrade https://github.com/HolyWu/ncnn/releases/download/1.0.20220910/ncnn-1.0.20220910-cp310-cp310-win_amd64.whl
python -m vsdpir_ncnn

usage:
from vsdpir_ncnn import dpir

ret = dpir(clip)
options:
def dpir(
clip: vs.VideoNode,
strength: float | vs.VideoNode | None = None,
task: str = 'denoise',
tile_w: int = 0,
tile_h: int = 0,
tile_pad: int = 8,
gpu_id: int | None = None,
fp16: bool = True,
) -> vs.VideoNode:
"""
DPIR: Deep Plug-and-Play Image Restoration
Parameters:
clip: Clip to process. Only RGB and GRAY formats with float sample type of 32 bit depth are supported.
strength: Strength for deblocking/denoising. Defaults to 50.0 for 'deblock', 5.0 for 'denoise'.
Also accepts a GRAY8/GRAYS clip for varying strength.
task: Task to perform. Must be 'deblock' or 'denoise'.
tile_w, tile_h: Tile width and height, respectively. As too large images result in the out of GPU memory issue,
so this tile option will first crop input images into tiles, and then process each of them.
Finally, they will be merged into one image. 0 denotes for do not use tile.
tile_pad: The pad size for each tile, to remove border artifacts.
gpu_id: The GPU ID.
fp16: Enable FP16 mode.
"""

------------------------------


If someone compares this to https://github.com/HolyWu/vs-dpir let us know whether there are some speed differences between the two.
Thanks!

On my Geforce GTX 1070ti on 640x352 content I get:

vs-dpir:
clip = DPIR(clip=clip, strength=5.000, task="denoise", provider=1, device_id=0, dual=True)
Output 429 frames in 96.04 seconds (4.47 fps) (without dual=True, I get 4.29fps)

clip = DPIR(clip=clip, strength=5.000, task="denoise", provider=1, device_id=0)
Output 429 frames in 96.04 seconds (4.47 fps)

vs-dpir-ncnn:
clip = dpir(clip=clip, strength=5.000, task="denoise", gpu_id=0)
Output 429 frames in 117.23 seconds (3.66 fps)

So for me it's better to stick with normal vs-dpir, but I wonder whether this is true for folks with other cards.

Cu Selur

[Selur]

update to vs-dpir-ncnn:
v2.0.0
Repository: HolyWu/vs-dpir-ncnn · Tag: v2.0.0 · Commit: 0938c18 · Released by: HolyWu
Turn dpir function into a class so as to separate model initialization and inference. It's more memory friendly when you run the same task more than once (such as on different clips or with different strengths) since the same model will be initialized only once.

Cu Selur

[ReinerSchweinlin]

Thanx for mentioning, good news :) Did you by chance already put it into hybrid? Haven´t got much time lately, I remember you had a "big" version of hybrid, containing a lot of upscaling filters and additions. If so, I´d be happy to try it out and benchmark to add some numbers to you above test.

[Selur]

Did not add it to Hybrids torchAddon and atm. I'm not planning to add it since it's like 20% slower on my machine.

[AIisCool]

Has anyone found the super resolution of BasicVSR++ to cause strange effects on certain video?

Before:
https://i.imgur.com/Z79WIhe.png

After:
https://i.imgur.com/V6ns97O.png


It doesn't matter which I use, REDS, Vimeo-90K (BI), Vimeo-90K (BD), or even the NTIRE 2021 Video Super-Resolution, it creates harsh lines and distortions on this part of the video, and oddly only this part.

[markanini]

How do I modify the google colab to use my video?

[Selur]

@AIisCool: got a short sample of the source, so others can try to reproduce this?
@markanini: sorry, no clue, since I'm not using the colab thing.

[gmail123]

The order in which I installed BasicVSR++:
1、I have installed：Anaconda3-2022.10-Windows-x86_64、cuda_11.6.0_511.23_windows、python-3.10.8-amd64、VapourSynth64-R59
2、PyTorch:
conda create -n PyTorch python=3.10
https://pytorch.org/get-started/locally/
conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia
3、INSTALLING NUMPY
conda create -n my-env
conda activate my-env
4、mmcv-full ：
pip install -U openmim
mim install mmcv-full
but：
5、pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp
worng:https://imgloc.com/i/DQKrp
https://i.328888.xyz/2022/12/26/DQKrp.md.jpeg

[gmail123]

Where is the installation error?
How to install it correctly?
THX

[poisondeathray]

No module named "vapoursynth" suggests vapoursynth is not installed. I'm guessing it might be an issue with anaconda

[poisondeathray]

Anyone try RealBasicVSR yet? It seems to produce oversmooth ,blurry, reduced detail results .

VRT SR is temporal and slightly higher metrics than basicvsr++ in the papers . In my short tests, it produces similar results to basicvsr++, but significantly slower than basicvsr++ . The provided pretrained models were trained on similar sets, and you get similar artifacts under similar conditions

There is a new one RVRT, but I can't get it to run

[Selur]

Haven't tried RealBasicVSR, will look at it once it can be used through Vapoursynth.
btw. I just saw https://github.com/HolyWu/vs-femasr (https://github.com/chaofengc/FeMaSR), but haven't played around with it so far. :)

Cu Selur

[gmail123]

No module named "vapoursynth" suggests vapoursynth is not installed. I'm guessing it might be an issue with anaconda

But i have installed vapoursynth, and i can use vapoursynth:sly:

[poisondeathray]

But i have installed vapoursynth, and i can use vapoursynth:sly:

Can you run vapoursynth in the anaconda environment ? I remember some people had problems before.

[Selur]

Did a quick test of vs-femasr and in general the results seem nice, but atm. it does create some strange highlights.
https://i.ibb.co/qWf08g1/grafik.png (https://ibb.co/4YGsLSN)
Also TensorRT support does not work fo rme: https://github.com/HolyWu/vs-femasr/issues/2

Suzi:
https://i.ibb.co/XYP4M98/grafik.png (https://ibb.co/JRn48hK)

Cu Selur

[Selur]

https://github.com/raywzy/Bringing-Old-Films-Back-to-Life looks interesting.

[gmail123]

Thanks to HolyWu I got vs-basicvsrpp working using:
python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11/index.html
python -m pip install tqdm
python -m pip install opencv-python
python -m pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp
:)

Cu Selur

I ran the same command, but it failed.
Can you explain what software and operating environment has been installed?
Thank you very much!

[Selur]

I don't really get the question. But maybe reading [INFO] Portable Vapoursynth (Windows) with pytorch&co (https://forum.selur.net/thread-2918.html) helps, those are the steps I used back then to set up my portable Vapoursynth.
I can also send you a link (via pm) to a download of my latest portable Vapoursynth that I use in Hybrid, if that would help.

Cu Selur

[gmail123]

I don't really get the question. But maybe reading [INFO] Portable Vapoursynth (Windows) with pytorch&co (https://forum.selur.net/thread-2918.html) helps, those are the steps I used back then to set up my portable Vapoursynth.
I can also send you a link (via pm) to a download of my latest portable Vapoursynth that I use in Hybrid, if that would help.

Cu Selur
I will try it.I've PM you. thanks a lot:thanks:

[gmail123]

@Selur Special thanks to you! I have successfully installed vs-basicvsrpp.
Test video resolution: 2064*1416, 3070, 0.84 fps:D

script:
from vsbasicvsrpp import BasicVSRPP
ret = core.resize.Bicubic(ret, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
ret = BasicVSRPP(ret,model=3,fp16=True,tile_x=360,tile_y=480)
ret = core.resize.Bicubic(ret, format=vs.YUV420P12, matrix_s="470bg", range_s="limited")
Can it be optimized?:rolleyes:

[Selur]

Can it be optimized?
For speed, probably not. (aside from a hardware upgrade, or maybe lower the interval, but that will also influence the processing strength)

Cu
Selur

[kedautinh12]

Yeah, you're right, Cuda plugins very slow now

Cu
Kedautinh12 :D

[Selur]

Cuda plugins very slow now
Not really Cuda, more like most of the ai model based stuff. ;)

[kedautinh12]

You forgot "Cu Selur" :D

[Selur]

¯\_(ツ)_/¯

[Selur]

A few more examples of vs-femasr:
not bad:
https://imgsli.com/MTQ1ODI3
https://imgsli.com/MTQ1ODI4
https://imgsli.com/MTQ1ODI5
https://imgsli.com/MTQ1ODM1
nice detail recovery, but partially too sharp (too much contrast) for my liking, and if you have noise&co that might get enhanced too.
https://imgsli.com/MTQ1ODM5

Cu
Selur

[Selur]

High Performance GMFSS with RIFE and GAN for Anime Video Frame Interpolation: https://github.com/HolyWu/vs-gmfss_union

[kedautinh12]

Wow, VS development very fast. When i think Avs+ can reach to VS but after Pytorch, NumPy releases VS development more than Avs+ very much

[ChaosKing]

Some quick performance numbers with RTX 3070 TI & 720x480 clip (noisy source):
tested with vsedit, so number should be slighlty higher bcs of the editors small overhead.

gmfss_union(clip) # ~14fps
gmfss_union(clip, trt=True) # ~17fps, tensor cache took a long time to build
gmfss_union(clip, num_streams=2) # ~17.5fps
gmfss_union(clip, num_streams=3) # ~18.9fps
gmfss_union(clip, trt=True, num_streams=2) # ~21.5fps
gmfss_union(clip, trt=True, num_streams=3) # ~22.5fps

[Selur]

What speed do you get with RIFE and the anime model?

[ChaosKing]

https://github.com/HolyWu/vs-rife runs with 175 fps
clip = RIFE(clip)

https://github.com/HomeOfVapourSynthEvolution/VapourSynth-RIFE-ncnn-Vulkan
rife.RIFE(clip, model=3) # ~78fps

Look at this perfect frame :devil:
https://i.imgur.com/r2GAnRI.jpeg

[Selur]

Okay, so it's really slow. :)
(for RIFE enabling sc and setting sc_threshold to something like 0.1 might help a bit)

...

Played around with it and probably won't add this to Hybrid since I don't see any gain from it over RIFE&co atm.

[~ VEGETA ~]

tried installing vsbasicvsr++ but got this error: https://pastebin.com/6VWwaXRd

how can i solve it?

[Selur]

Probably an issue with the cuda dependencies,
here's what I did last time I setup the environment:

I created a new empy Vapoursynth-folder
downloaded 'Windows embeddable package (64-bit)' from https://www.python.org/downloads/release/python-3108/ (3.11 isn't supported atm. by Vapoursynth)
extracted the Python download into the download into the 'Vapoursynth'-folder
downloaded 'VapourSynth64-Portable-R60.7z' from https://github.com/vapoursynth/vapoursynth/releases
extracted the Vapoursynth portable download into the 'Vapoursynth'-folder
downloaded get-pip.py from https://bootstrap.pypa.io/get-pip.py and save it into the 'Vapoursynth'-folder
opened a 'Windows Command Prompt'-window and navigate into the 'Vaporusynth'-folder
installed pip by calling :
pyhton get-pip.py
opened the python310._pth in a text addition and added the following to lines above anything else in that file and saved the file
Scripts
Lib\site-packages

installed VSGAN

python -m pip install vsgan
python -m pip install torch===1.13.0+cu117 torchvision==0.14.0 -f https://download.pytorch.org/whl/torch_stable.html
installed BASICVSR++

python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu117/torch1.13/index.html
python -m pip install tqdm
python -m pip install opencv-python
python -m pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp


installted RIFE

downloaded https://github.com/HolyWu/vs-realesrgan/releases/download/v4.0.1/CUDA-11.7_cuDNN-8.6.0_TensorRT-8.5.2.2_win64.7z (there's a newer versio now)

extractedtensorrt-8.5.2.2-cp310-none-win_amd64.whl into my Vapoursynth folder
python -m pip install tensorrt-8.5.2.2-cp310-none-win_amd64.whl
deleted tensorrt-8.5.2.2-cp310-none-win_amd64.whl
python -m pip install --upgrade vsrife
created a folder 'vsrife_dependencies' inside the Vapoursynth folder and copied the other files form the CUDA-11.7_cuDNN-8.6.0_TensorRT-8.5.1.7_win64.7z into it. This folder need to be added to when using vs-rife using:
import site
# add vs-rife dependencies to the path
path = site.getsitepackages()[0]+'/torch_dependencies/'
path = path.replace('\\', '/')
import os
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]
downloaded the models using:
set PATH=i:\Hybrid\64bit\Vapoursynth\torch_dependencies\bin;%PATH
python -m vsrife

installed SWINIR

python -m pip install --upgrade vsswinir
python -m vsswinir


installed DPIR and onnxruntime-gpu

python -m pip install --upgrade vsdpir
python -m pip install --upgrade onnxruntime-gpu
python -m vsdpir

created a onnxruntime_dlls folder inside the Vapoursynth folder.
I copied from cudnn-11.4-windows-x64-v8.2.4.15.zip and NVIDIA CUDA SDK 11.4.1 runtimes the following files into the folder:

cublas64_11.dll
cublasLt64_11.dll
cudart64_110.dll
cudnn_cnn_infer64_8.dll
cudnn_ops_infer64_8.dll
cudnn64_8.dll
cufft64_10.dll
cufftw64_10.dll
msvcp140.dll
nvinfer.dll
nvinfer_builder_resource.dll
nvinfer_plugin.dll
nvonnxparser.dll
nvparsers.dll
vcruntime140.dll
vcruntime140_1.dll

When using vsdir or realesrgan in a Vapoursynth script:

# Import libraries for onnxruntime
import site
import ctypes
path = site.getsitepackages()[0]+'/onnxruntime_dlls/'
ctypes.windll.kernel32.SetDllDirectoryW(path)
needs to be added to the script to load the dlls.
installed REALESRGAN (which also uses onnxruntime)

python -m pip install --upgrade vsrealesrgan
python -m vsrealesrgan

Installed HINet

python -m pip install --upgrade vshinet
python -m vshinet



Cu Selur

Ps.: also send you a link to my current Hybrid dev and torch addon, going to bed now. ;) You might want to checkout

[Selur]

Anyone managed to get BasicVSR++ v2.0.0 running in a portable Vapoursynth setup?
I'm stuck https://github.com/HolyWu/vs-basicvsrpp/issues/29 installing mmcv-full. :(

Cu Selur

[Selur]

HolyWu could help, using:
python -m pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117
python -m pip install -U openmim
python -m mim install "mmcv-full>=1.7.1"
python -m pip install -U vsbasicvsrpp
python -m vsbasicvsrpp
worked :)

Cu Selur

[Selur]

With the release of Vapoursynth R62, it's either Python 3.11 or Python 3.8. (Python 3.8 is still there to support old Windows versions and Python 3.10 support was dropped not to having to support multiple Python versions.)
So Python 3.11 is probably what most folks want to go for, but since there are no cp311 pytorch versions over at https://download.pytorch.org/whl/torch/ for Windows I was wondering if someone got a way to get all the ai stuff working with Python 3.11 and Vapoursynth R62?

Cu Selur

[Selur]

btw. did anyone test https://github.com/HolyWu/vs-codeformer ?
https://imgsli.com/MTY2MTM5
Suzie: https://imgsli.com/MTY2MTQz

[ChaosKing]

lol they get entirely new faces :eek:

[Selur]

Yes, but in combination with realesrgan this might have its uses. :)
=> I have to check what happens if this is applied to different actors,...
https://imgsli.com/MTY2MTQ0
https://imgsli.com/MTY2MTQ2
https://imgsli.com/MTY2MTQ3
https://imgsli.com/MTY2MTQ5
seems more like a gimmick filter than really usable atm., but its impressive.

Cu Selur

[Dogway]

Some people mix it with GFPGAN or other face models, haven't played with NN but it seems there's more work to it than a simple button push.
Usually whenever I use NN upscaling I always follow a downscale, to hide the pristine AI look and blend everything together.
Very excited to get into this AI craze soon.

[HolyWu]

Yes, but in combination with realesrgan this might have its uses. :)
=> I have to check what happens if this is applied to different actors,...
https://imgsli.com/MTY2MTQ0
https://imgsli.com/MTY2MTQ2
https://imgsli.com/MTY2MTQ3
https://imgsli.com/MTY2MTQ5
seems more like a gimmick filter than really usable atm., but its impressive.

Cu Selur

You should also try weight=1 if you prefer a higher-fidelity result.

[Selur]

Will do.
Main issue I have atm. with it, is that when the head covers, most of the image, only parts of the head seem to be processed.
This kind of ruins the image if the person has longer hair.
https://i.ibb.co/3c2jNcC/grafik.png (https://ibb.co/Xt1KstC)
(using detector = 1, doesn't help either)
https://imgsli.com/MTY2MjEw Suzie with weight=1

[ChaosKing]

At least the face now looks like it belongs to the same person :devil:

[Selur]

Here's an example of weight 0 and weight 1: https://imgsli.com/MTY2Mjc4

[Selur]

Did some further testing and if upscale is > 1 (weight=1) and the upscaling is later undone it basically can be used as a 'face detail enhancer' which might be interesting.
Also in combination with FeMaSR (with weight<1) it can be used to fix the face distortions FeMaSR does.
https://i.ibb.co/1TZ0yRs/grafik.png (https://ibb.co/Fz6hZ8s)https://i.ibb.co/JqnSm6n/grafik.png (https://ibb.co/G97rnY7)https://i.ibb.co/3m74tKM/grafik.png (https://ibb.co/k2x5YZ9)
This does not fix the issue with half enhanced heads, but seems to be a good way to use the filter. :)
SwinIRx4+vsCodeFormer: https://imgsli.com/MTY2NDA3, looks nice frame-by-frame, but isn't really temporally stable (https://forum.videohelp.com/attachments/70126-1680433345/SwinIR+CodeFormer_lossless.mkv) => adding 3xQTGMC_type3 (https://forum.videohelp.com/attachments/70128-1680434869/3xQTGMC_type3.mp4).

Cu Selur

Ps.: got a nice example of what weight 0 can do: https://imgsli.com/MTY2NDI2,...

[Selur]

Okay, small update in regard to Vapoursynth R62:
TensorRT doesn't support Python 3.11 atm.
Latest TensorRT-package (TensorRT-8.6.0.12.Windows10.x86_64.cuda-11.8.zip (https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/8.6.0/zip/TensorRT-8.6.0.12.Windows10.x86_64.cuda-11.8.zip)) from https://developer.nvidia.com/tensorrt does not contain (TensorRT-8.6.0.12.Windows10.x86_64.cuda-11.8.zip\TensorRT-8.6.0.12\python) a cp311 file. :(
So all the filters that use TensorRT won't work. :(
=> Hoping NVIDIA will release a Python 3.11 file in the near (who knows) future.

Cu Selur

[Selur]

Maybe interesting to some:
https://github.com/HolyWu/vs-grlir uses real-world image super-resolution model of GRL for Image Restoration (https://github.com/ofsoundof/GRL-Image-Restoration)
https://imgsli.com/MTcxMDA1

Cu Selur

[ReinerSchweinlin]

Thanx for all the info.. I think its about time to check out your extended version of hybrid, including all the nice "AI" stuff not included in the normal release :)

[Selur]

@RainerSchweinlin: Send you a pm with links to my current dev + torchAddon. :)

[ReinerSchweinlin]

@RainerSchweinlin: Send you a pm with links to my current dev + torchAddon. :)

Merci :)

[Selur]

btw. vs-grlir just got updated and not supports multiple models, cuda_graphs, fp16. :)
:param model: Model to use.
0 = Blind Image SR
1 = Defocus Deblurring
2 = Motion Deblurring (GoPro)
3 = Motion Deblurring (RealBlur-J)
4 = Motion Deblurring (RealBlur-R)
5 = Demosaicking
6 = Denoising (sigma 15)
7 = Denoising (sigma 25)
8 = Denoising (sigma 50)
9 = JPEG compression artifact removal (quality 10)
10 = JPEG compression artifact removal (quality 20)
11 = JPEG compression artifact removal (quality 30)
12 = JPEG compression artifact removal (quality 40)
13 = Classical Image SR (scale 2)
14 = Classical Image SR (scale 3)
15 = Classical Image SR (scale 4) source: https://github.com/HolyWu/vs-grlir/blob/master/vsgrlir/__init__.py
Models 0, 12-15 are for upscaling, the others are for filtering. :)
Nice!

Cu Selur

[Selur]

Has anyone tried https://github.com/tongyuantongyu/vs-NNVISR/ ?

[Selur]

Tried to test it by extracing everything into "F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR" ending with:
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cublas64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cublasLt64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudart64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_adv_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_adv_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_cnn_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_cnn_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_ops_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_ops_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvblas64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_builder_resource.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_dispatch.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_lean.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_plugin.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_vc_plugin.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvonnxparser.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvrtc-builtins64_121.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvrtc64_120_0.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\vs-nnvisr.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\fe_n2_2x2_l4_yuv1-1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\fe_n4_4x4_l1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\ff_n2_2x2_l4_yuv1-1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\ff_n4_4x4_l1.onnx

Problem is when i use:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core

# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll")

# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\files\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/files/test.avi", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)

# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip, _Matrix=5)
# Setting color transfer info (470bg), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
# Setting color primaries info (), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive


clip = core.nnvisr.Super(clip, scale_factor=2, use_fp16=True, model="fe_n2_2x2_l4_yuv1-1.onnx", model_path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR")



# set output frame rate to 25fps (progressive)
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

I get:
Failed to evaluate the script:
Python exception: Failed to load F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll. GetLastError() returned 1114.

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 3115, in vapoursynth._vpy_evaluate
File "src\cython\vapoursynth.pyx", line 3116, in vapoursynth._vpy_evaluate
File "C:\Users\Selur\Desktop\test_1.vpy", line 8, in
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll")
File "src\cython\vapoursynth.pyx", line 2847, in vapoursynth.Function.__call__
vapoursynth.Error: Failed to load F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll. GetLastError() returned 1114.
So the loading of the dll already fails. :(

-> does anyone have an idea what I'm missing/overlooking?

Cu Selur

[Selur]

Got it working.
dll folder needs to explicitly loaded:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import site
import os
import ctypes
# Adding torch dependencies to PATH
path = site.getsitepackages()[0]+'/../vsfilters/ResizeFilter/NNVISR'
ctypes.windll.kernel32.SetDllDirectoryW(path)
path = path.replace('\\', '/')
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]
and Primaries, Transfer and Matrix all have to be BT709 (for RGB only Primaries and Transfer have to be bt709).

Cu Selur
Ps.: the explicit loading is fixed in 'v1.0.0-pre3'

[ReinerSchweinlin]

Cool you got it working :)

If I understand correctly, this enables multi-frame aware networks to be run, not only single-frame ones. Correct?

[Selur]

Yes, vs-NNVISR support models which use mutliple frames for their calculation, main problem is finding such models (or training them yourself).
We are not aiming at being an alternative of vs-mlrt, but focus mainly on models that do multiple frames to multiple frames enhancement, which vs-mlrt currently not able to do, or at least not easily. source: issue #1 (https://github.com/tongyuantongyu/vs-NNVISR/issues/1#issuecomment-1585733868)

Side note: VSGAN also supports EGVSR models, which are 'temporal aware'.

Current available models only run with specific values.
"'scale_factor': 2, 'input_count': 2,..., 'interpolation': True" means the model always does scale by 2, takes two frames as input and interpolates the frames by 2.
Also, the YUV420 1/1/1 means that it only works if color fromat is YUV420(P8), _Transfer, _Primaries and _Matrix are bt709.
see: https://github.com/tongyuantongyu/vs-NNVISR/blob/main/docs/models.md

So to me, it's nice to see something different is being developed, but atm. I won't use it since it seems too much of a hassle. ;)

Cu Selur

Ps.: Seems like the 'Enhance Everything' folks are revamping https://upscale.wiki/wiki/Model_Database into https://openmodeldb.info/

[mastrboy]

Ps.: Seems like the 'Enhance Everything' folks are revamping https://upscale.wiki/wiki/Model_Database into https://openmodeldb.info/

Looks good, the wiki format doesn't really work that well for content like that...

vsbasicvsrpp does temporal I think, with the length parameter: https://github.com/HolyWu/vs-basicvsrpp/blob/master/vsbasicvsrpp/__init__.py#L50

[Selur]

Yes, BasicVSR++ does, but most models you load in VSGAN/vsMLRT do not.

[Selur]

Trying to set up a new portable Vapoursynth R64 with BasicVSR++&Co I'm stuck.
What I did so far to create a portable Vapoursynth with pytorch&co:

created a new empty Vapoursynth-folder
downloaded 'Windows embeddable package (64-bit)' from https://www.python.org/downloads/release/python-3111/
extracted the Python download into the download into the 'Vapoursynth'-folder
downloaded 'VapourSynth64-Portable-R64' from https://github.com/vapoursynth/vapoursynth/releases
extracted the Vapoursynth portable download into the 'Vapoursynth'-folder
downloaded get-pip.py from https://bootstrap.pypa.io/get-pip.py and save it into the 'Vapoursynth'-folder
opened a 'Windows Command Prompt'-window and navigate into the 'Vapoursynth'-folder
installed pip by calling :
python get-pip.py
opened the python311._pth in a text addition and added the following to lines above anything else in that file and saved the file
Scripts
Lib\site-packages
installed pytorch using:
python -m pip install torch===2.1.0+cu121 torchvision===0.16.0 -f https://download.pytorch.org/whl/torch_stable.html

installed VSGAN (https://github.com/rlaPHOENiX/VSGAN)
python -m pip install vsgan
installed DPIR (https://github.com/HolyWu/vs-dpir)

downloaded https://github.com/HolyWu/vs-rife/releases/download/v4.0.0/CUDA-12.2.2_cuDNN-8.9.5_TensorRT-8.6.1.6_win64.7z extracted tensorrt-8.6.1-cp311-none-win_amd64.whl file into the Vapoursynth-folder.
extracted the other dlls into a 'torch_dependencies/bin'-folder inside the Vapoursynth folder
called set PATH=<PATH TO torch_dependencies>\bin;%PATH%
installed vsdir and its dependencies
python -m pip install tensorrt-8.6.1-cp311-none-win_amd64.whl
python -m pip install -U vsdpir
python -m pip install packaging
python -m vsdpir

installed Real-ESGRAN (https://github.com/HolyWu/vs-realesrgan)
python -m pip install -U vsrealesrgan
python -m vsrealesrgan
installed vs-rife (https://github.com/HolyWu/vs-rife)
python -m pip install --upgrade vsrife
python -m vsrife
installed GMFSS (https://github.com/HolyWu/vs-gmfss_fortuna)
python -m pip install -U vsgmfss-fortuna
installed SwinIR (https://github.com/HolyWu/vs-swinir)
python -m pip install -U vsswinir
python -m vsswinir
installe HINet (https://github.com/HolyWu/vs-hinet)
python -m pip install --upgrade vshinet
python -m vshinet
installed AnimeSR (https://github.com/HolyWu/vs-animesr)
python -m pip install -U vsanimesr
installed FeMaSR (https://github.com/HolyWu/vs-femasr)
python -m pip install -U vsfemasr
python -m vsfemasr
installed SCUNet (https://github.com/HolyWu/vs-scunet)
python -m pip install -U vsscunet
python -m vsscunet
installed CodeFormer (https://github.com/HolyWu/vs-codeformer)
python -m pip install -U vscodeformer
python -m vscodeformer
installed GRLIR (https://github.com/HolyWu/vs-grlir)
python -m pip install -U vsgrlir
python -m vsgrlir

Not I wanted to call:
python -m pip install -U openmim
mim install "mmcv>=2.0.0"
python -m pip install -U vsbasicvsrpp
python -m vsbasicvsrpp

to install BasicVSR++, problem is when calling:
python -m pip install -U openmim
it fails with:
Collecting aliyun-python-sdk-core>=2.13.12 (from oss2~=2.17.0->openxlab->opendatalab->openmim)
Using cached aliyun-python-sdk-core-2.14.0.tar.gz (443 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [7 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "C:\Users\Selur\AppData\Local\Temp\pip-install-t2r9eu4j\aliyun-python-sdk-core_b912f61c599e490b859580d9c77cbbd2\setup.py", line 41, in <module>
VERSION = __import__(PACKAGE).__version__
^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'aliyunsdkcore'
[end of output]
see: https://github.com/HolyWu/vs-basicvsrpp/issues/37

I also tried installing aliyun-python-sdk-core
with:
python -m pip install aliyunsdkcore
and
python -m pip install aliyun-python-sdk-core-v3
but both calls end the same way, with the same error I got above.

=> does anyone have an idea how to fix this?

Cu Selur

[Selur]

Seems to be related to the openmim version, when using:
python -m pip install -U openmim==0.3.7
(0.3.7 was the version I had installed in a previous portable setup)
openmim does install without an issue, but then
set PATH=f:\Hybrid\64bit\Vapoursynth\Scripts;%PATH%
mim install "mmcv>=2.0.0"
fails with:

Looking in links: https://download.openmmlab.com/mmcv/dist/cu121/torch2.1.0/index.html
Collecting mmcv>=2.0.0
Using cached mmcv-2.0.1.tar.gz (469 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
C:\Users\Selur\AppData\Local\Temp\pip-install-xs23k5n_\mmcv_a93dd8751cad4bf1bafcae67f7e00b43\setup.py:5: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
from pkg_resources import DistributionNotFound, get_distribution, parse_version
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "C:\Users\Selur\AppData\Local\Temp\pip-install-xs23k5n_\mmcv_a93dd8751cad4bf1bafcae67f7e00b43\setup.py", line 466, in <module>
ext_modules=get_extensions(),
^^^^^^^^^^^^^^^^
File "C:\Users\Selur\AppData\Local\Temp\pip-install-xs23k5n_\mmcv_a93dd8751cad4bf1bafcae67f7e00b43\setup.py", line 423, in get_extensions
ext_ops = extension(
^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\cpp_extension.py", line 1076, in CUDAExtension
library_dirs += library_paths(cuda=True)
^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\cpp_extension.py", line 1210, in library_paths
paths.append(_join_cuda_home(lib_dir))
^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\cpp_extension.py", line 2416, in _join_cuda_home
raise OSError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "f:\Hybrid\64bit\Vapoursynth\Scripts\mim.exe\__main__.py", line 7, in <module>
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\mim\commands\install.py", line 73, in cli
exit(exit_code)
^^^^
NameError: name 'exit' is not defined

¯\(°_o)/¯
(same when using openmim==0.3.8, openmim==0.3.9 fails at the install of openmim)

Cu Selur

[Selur]

Pre-built package for PyTorch 2.1+cu121 is probably not available. File an issue to MMCV if you really hurry to have it. source: https://github.com/HolyWu/vs-basicvsrpp/issues/37#issuecomment-1749183229
=> so there's hope, but will have to wait

[ReinerSchweinlin]

Sorry, I can´t help, thats beyond my capabilities :)

But I just wanted to say "thanks" for all you do !!

[Selur]

btw. did anyone test NNVISR ? (https://github.com/tongyuantongyu/vs-NNVISR)

[lansing]

btw. did anyone test NNVISR ? (https://github.com/tongyuantongyu/vs-NNVISR)

I couldn't get it working on vsedit2 or vd2 or vsedit_mod. I manual install the plugin and its dependencies. Somehow the plugin won't auto load when I put it in my roaming folder, so I'll have to put everything in Vapoursynth's installation plugin folder. But everything just crashed after some loading time.


import vapoursynth as vs

core = vs.core

clip = core.bs.VideoSource(r'E:\video samples\hand wave.mkv')

model_config = {'scale_factor': 2, 'input_count': 2, 'feature_count': 64, 'extraction_layers': 4, 'interpolation': True, 'extra_frame': True, 'double_frame': True, 'model': 'cycmunet/vimeo90k-deblur'}

clip = core.nnvisr.Super(clip,
model_path=r"C:\Program Files\VapourSynth\plugins", **model_config)

clip.set_output()

[Selur]

When I first tested this, I used this file structure
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cublas64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cublasLt64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudart64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_adv_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_adv_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_cnn_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_cnn_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_ops_infer64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\cudnn_ops_train64_8.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvblas64_12.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_builder_resource.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_dispatch.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_lean.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_plugin.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvinfer_vc_plugin.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvonnxparser.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvrtc-builtins64_121.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\nvrtc64_120_0.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\test_1.vpy
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\timing.cache
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\vs-nnvisr.dll
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\timing.cache
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet\vimeo90k-deblur
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet\vimeo90k-deblur\yuv_1_1_1
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet\vimeo90k-deblur\yuv_1_1_1\fe_640x352_2x2_b1_l4_yuv1-1.engine
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\8601\NVIDIA GeForce RTX 4080\cycmunet\vimeo90k-deblur\yuv_1_1_1\ff_n2a+_640x352_2x2_b1_l4_yuv1-1.engine
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet\vimeo90k-deblur
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet\vimeo90k-deblur\yuv_1_1_1
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet\vimeo90k-deblur\yuv_1_1_1\fe_640x352_2x2_b1_l4_yuv1-1.engine
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\engines\cycmunet\vimeo90k-deblur\yuv_1_1_1\ff_n2a+_640x352_2x2_b1_l4_yuv1-1.engine
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet\vimeo90k-deblur
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet\vimeo90k-deblur\yuv_1_1_1
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet\vimeo90k-deblur\yuv_1_1_1\fe_n2_2x2_l4_yuv1-1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\cycmunet\vimeo90k-deblur\yuv_1_1_1\ff_n2_2x2_l4_yuv1-1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo\vimeo90k
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo\vimeo90k\rgb_1_1
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo\vimeo90k\rgb_1_1\fe_n4_4x4_l1.onnx
F:\Hybrid\64bit\vsfilters\ResizeFilter\NNVISR\models\yogo\vimeo90k\rgb_1_1\ff_n4_4x4_l1.onnx
and the following script:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import site
import os
import ctypes
# Adding torch dependencies to PATH
path = site.getsitepackages()[0]+'/../vsfilters/ResizeFilter/NNVISR'
ctypes.windll.kernel32.SetDllDirectoryW(path)
path = path.replace('\\', '/')
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]


# Loading Plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR/vs-nnvisr.dll")

# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\files\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/files/test.avi", format="YUV420P8", stream_index=0, cache=0, prefer_hw=0)

# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip, _Matrix=5)
# Setting color transfer info (470bg), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
# Setting color primaries info (), when it is not set
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0) # progressive

clip = core.resize.Bicubic(clip=clip, matrix_in_s="470bg", matrix_s="709", range_in=0, range=0) # converting to bt709 for NNVISR
clip = core.std.SetFrameProps(clip, _Transfer=1)
clip = core.std.SetFrameProps(clip, _Primaries=1)

model_config = {'scale_factor': 2, 'input_count': 2, 'feature_count': 64, 'extraction_layers': 4, 'interpolation': True, 'extra_frame': True, 'double_frame': True, 'model': 'cycmunet/vimeo90k-deblur'}
clip = core.nnvisr.Super(clip, model_path="F:/Hybrid/64bit/vsfilters/ResizeFilter/NNVISR", **model_config)

# Output
clip.set_output()
in a portable Vapoursynth setup. But I had my struggles too (https://github.com/tongyuantongyu/vs-NNVISR/issues/2).
Maybe this helps you getting it working.

Cu Selur

Ps.: iirc. it took ages to get engine files created on my setup when I tested it.

[julius666]

Does anyone know what happened to vs-femasr? The github repository is not available anymore.

[kedautinh12]

HolyWu was deleted it (maybe)

[Selur]

Yup, like a few other repos HolyWu deleted it. (or set it private)

[julius666]

Yup, like a few other repos HolyWu deleted it. (or set it private)

I thought so but do we have any information regarding why he did that? It was one of the best (if not THE best) upscalers in vapoursynth.

[Selur]

No (at least not that I know of), you would have to ask him (maybe by posting an issue in one of this other projects https://github.com/HolyWu/).
Probably because there hasn't been an update to it, not much feedback and keeping it working with latest pytorch&co was too much trouble. :)

[Selur]

btw. for those who need colorization of b&w content https://github.com/HolyWu/vs-ddcolor might be worth a try:
https://i.ibb.co/rwGVJr8/grafik.png (https://ibb.co/d50RyYC)
did some small tests which do seem good so far.

Cu Selur

[kedautinh12]

Wow, respect HolyWu :D

[Selur]

https://github.com/piddnad/DDColor should get the main respect, HolyWu 'just' wrote a Vapoursynth Wrapper for it.

[kedautinh12]

Ok, piddnad too

[Selur]

Anyone has an idea how to stabilize the colorization output of ddcolor? (see: https://forum.selur.net/thread-3532.html)

[Selur]

https://github.com/dan64/vs-deoldify might be interesting for those interested in colorization
https://i.ibb.co/p3Xm0P1/grafik.png (https://ibb.co/gDgnj7r)

[ReinerSchweinlin]

Thank you.

I have a somewhat amateur question:

If this is based on pytorch - what relevance does the used GPU have in this case? Some hybrid filters are clearly based on specific stuff like vulkan or CUDA (CDNN) or some other filters use OPENCL... How does it work here?

I have been playing around with some StableDiffusion GUIs and here It seems (as far as I understand it), all that needs to be modified is how Pytorch passes the calls from the GUI to the underlying inferences. So if I want to use DirectML, I need the py-directML stuff, if i have Nvidia, I stick to the tensorRT or CUDA, hence must install pytor...cuda... etc... So basically, as long as some backend in pytorch for my specific hardware exists, the filter itself will run...

Could you help straighten out my somewhat fuzzy thinking of how it is related ?

Thanx :)

[Selur]

afaik vs-deoldify uses cudnn as backend (see: https://github.com/dan64/vs-deoldify/blob/main/vsdeoldify/deoldify/_device.py)

[ReinerSchweinlin]

afaik vs-deoldify uses cudnn as backend (see: https://github.com/dan64/vs-deoldify/blob/main/vsdeoldify/deoldify/_device.py)
Thanx for pointing me in the right direction :)

[Selur]

Side note for those using tensorrt: atm. there isn't a Python 3.12 version of the bindings (tensorrt-8.6.1-cp311-none-win_amd64.whl is the latest currently), so switching to Vapoursynth R66 will not work.

Cu Selur

[PatchWorKs]

Just out of curiosity: does anyone tested vs-mlrt (https://github.com/AmusementClub/vs-mlrt) performances on Intel CPU+iGPU ?

[Selur]

Only tested that it works with my A380, but never did any speed test since my RTX 4080 is clearly faster.

Cu Selur

[PatchWorKs]

Only tested that it works with my A380, but never did any speed test since my RTX 4080 is clearly faster.

Cu Selur
Ok, I'd like to understand how feasible (video) inferences are WITHOUT a discrete video card...

EDIT
I forgot to mention that I've collected some "AI" (= deep learning/neural nets) video tools in this GitHub page (https://github.com/forart/HyMPS/blob/main/V_AI.md).
Suggestions - and direct contribution too - are welcome.

[Selur]

Did anyone manage to get BasicVSR++ working with R66, now R67? (both requiring Python 3.12)

[Selur]

(since a user asked me via pm)
For those interested, here's how I atm. would create a portable Vapoursynth 67 with pytorch&co:

creat a new empty Vapoursynth-folder
download 'Windows embeddable package (64-bit)' from https://www.python.org/downloads/release/python-3122/
extract the Python download into the download into the 'Vapoursynth'-folder
downloaded 'VapourSynth64-Portable-R67' from https://github.com/vapoursynth/vapoursynth/releases
extract the Vapoursynth portable download into the 'Vapoursynth'-folder
download get-pip.py from https://bootstrap.pypa.io/get-pip.py and save it into the 'Vapoursynth'-folder
open a 'Windows Command Prompt'-window and navigate into the 'Vapoursynth'-folder
install pip by calling :
python get-pip.py
open the python312._pth in a text addition and added the following to lines above anything else in that file and saved the file
Scripts
Lib\site-packages
install wheel
python -m pip install wheel/VapourSynth-67-cp312-cp312-win_amd64.whl
installed pytorch using:
python -m pip install torch===2.2.0+cu121 torchvision===0.17.1 -f https://download.pytorch.org/whl/torch_stable.html

TensorRT

created a 'torch_dependencies/bin'-folder inside the Vapoursynth folder
download https://developer.download.nvidia.com/compute/machine-learning/tensorrt/10.0.0/zip/TensorRT-10.0.0.6.Windows10.win10.cuda-12.4.zip
extract (from the lib folder into the 'torch_dependencies/bin'-folder):

nvinfer.dll
nvinfer_builder_resource.dll
nvinfer_plugin.dll
nvinfervc__plugin.dll
nvonnxparser.dll

extract (from the python folder into the Vapoursynth folder: 'tensorrt-10.0.0b6-cp312-none-win_amd64.whl' and called:
python -m pip install tensorrt-10.0.0b6-cp312-none-win_amd64.whl
download https://developer.download.nvidia.com/compute/cuda/12.4.1/local_installers/cuda_12.4.1_551.78_windows.exe
extract:

cuda_cudart\cudart\bin\cudart64_12.dll
cuda_nvrtc\nvrtc\bin\nvrtc64_120_0.dll
cuda_nvrtc\nvrtc\bin\nvrtc-builtins64_124.dll
libcublas\cublas\bin\cublas64_12.dll
libcublas\cublas\bin\cublasLt64_12.dll
libcublas\cublas\bin\nvblas64_12.dll
libcufft\cufft\bin\cufft64_11.dll
libcufft\cufft\bin\cufftw64_11.dll
libcurand\curand\bin\curand64_10.dll
libcusolver\cusolver\bin\cusolver64_11.dll
libcusolver\cusolver\bin\cusolverMg64_11.dll
libcusparse\cusparse\bin\cusparse64_12.dll
libnvjpeg\nvjpeg\bin\nvjpeg64_12.dll

into the 'torch_dependencies/bin'-folder

download https://developer.download.nvidia.com/compute/cudnn/9.1.0/local_installers/cudnn_9.1.0_windows.exe
extract:

cudnn_cuda12.4\libcudnn\bin\12.4\cudnn64_9.dll
cudnn_cuda12.4\libcudnn\bin\12.4\cudnn_adv64_9.dll
cudnn_cuda12.4\libcudnn\bin\12.4\cudnn_cnn64_9.dll
cudnn_cuda12.4\libcudnn\bin\12.4\cudnn_ops64_9.dll

call set PATH=<PATH TO torch_dependencies>\bin;%PATH%
i.e.: set PATH=F:\Vapoursynth\torch_dependencies\bin;%PATH%;
install VSGAN (https://github.com/rlaPHOENiX/VSGAN)
python -m pip install -U VSGAN
installed DPIR (https://github.com/HolyWu/vs-dpir)

python -m pip install -U vsdpir
python -m pip install packaging
python -m pip install onnxruntime-gpu
python -m vsdpir[/code]

install Real-ESGRAN (https://github.com/HolyWu/vs-realesrgan)
python -m pip install -U vsrealesrgan
python -m vsrealesrgan
install vs-rife (https://github.com/HolyWu/vs-rife)
python -m pip install --upgrade vsrife
python -m vsrife
install GMFSS (https://github.com/HolyWu/vs-gmfss_fortuna)
python -m pip install -U vsgmfss-fortuna
install SwinIR (https://github.com/HolyWu/vs-swinir)
python -m pip install -U vsswinir
python -m vsswinir
install HINet (https://github.com/HolyWu/vs-hinet)
python -m pip install --upgrade vshinet
python -m vshinet
install AnimeSR (https://github.com/HolyWu/vs-animesr)
python -m pip install -U vsanimesr
install FeMaSR (https://github.com/HolyWu/vs-femasr)
python -m pip install -U vsfemasr
python -m vsfemasr
install SCUNet (https://github.com/HolyWu/vs-scunet)
python -m pip install -U vsscunet
python -m vsscunet
install CodeFormer (https://github.com/HolyWu/vs-codeformer)
python -m pip install -U vscodeformer
python -m vscodeformer
install GRLIR (https://github.com/HolyWu/vs-grlir)

python -m pip install antlr4-python3-runtime
python -m pip install wheel
python -m pip install -U vsgrlir
python -m vsgrlir
install BasicVSR++ (https://github.com/HolyWu/vs-basicvsrpp)
????


Sadly I have no clue how to BasicVSR++ (https://github.com/HolyWu/vs-basicvsrpp) installed, so far

set CUDA_HOME=F:\Vapoursynth\torch_dependencies\bin;
python -m pip install -U openmim==0.3.7

anything I tried so far after this fails :/
Seems like the problem is that there seems to be no mmcv with cp312 wheel for Windows.

Hope this is useful to anyone and if anyone finds a way to add BasicVSR++

Cu Selur

[Cary Knoop]

Thanks for that detailed description!

Life could be a lot easier if VapourSynth simply adhered to common standards.

More than a decade of VapourSynth and a simple "pip install VapourSynth" from a conda environment is still not supported. :(

[Selur]

Only 'fault' with Vapousynth is that it abandoned Python 3.11 and now requires Python 3.12, which other projects do not support atm.

[PatchWorKs]

Any (Adobe-backended) VideoGigaGAN (https://videogigagan.com/) test ?

Demo: https://videogigagan.github.io/assets/videos/demo.mp4

Project: https://videogigagan.github.io/

Git: https://github.com/danaigc/videoGigaGanHub

[Selur]

https://github.com/danaigc/videoGigaGanHub seems to lack the used models, so no way to reproduce the presented results (or test with other content).

[PatchWorKs]

...well, it seemed a bit strange that Adobe-backeded researchers pretended to be open (using Microsoft's GitHub, moreover)...

[PatchWorKs]

Fast-SRGAN (https://github.com/HasnainRaz/Fast-SRGAN#readme) (The goal of this repository is to enable real time super resolution for upsampling low resolution videos. [] For speed, the upsampling is done through pixel shuffle.) anyone ?

[Selur]

Isn't that like 5 years old,... with https://github.com/HolyWu/vs-realesrgan I get way faster speeds on 180x320 content, but then my hardware is also way faster than a m1.

[Selur]

Did anyone try ccrestoration (https://github.com/TensoRaws/ccrestoration) ?

[ReinerSchweinlin]

No - but the amount of usable models look very interesting.

It seems the open models site offers the models in pytorch, ONNX and NCNN Format - Isn´t hybrid already capable of using some of these ?

[Selur]

Yes, Hybrid can use those that VSGAN and VSMLRT support, but trying somthing new seems worth a look especially since some of the mentioned architectures https://github.com/TensoRaws/ccrestoration/blob/main/ccrestoration/type/arch.py are not supported by VSGAN and VSMLRT afaik. :)

[poisondeathray]

vs_liff - interesting resizing project . Works ok , but looks like only the baseline model so far

https://github.com/pifroggi/vs_liif

https://yinboc.github.io/liif/

[Selur]

Looked at it a while ago, but couldn't find an example where I liked the output :)

[ReinerSchweinlin]

Yes, Hybrid can use those that VSGAN and VSMLRT support, but trying somthing new seems worth a look especially since some of the mentioned architectures https://github.com/TensoRaws/ccrestoration/blob/main/ccrestoration/type/arch.py are not supported by VSGAN and VSMLRT afaik. :)
Tempting :)

[ReinerSchweinlin]

Looked at it a while ago, but couldn't find an example where I liked the output :)

For some Line Art it produces not so bad results and IMHO it could be usefull as a generic upscaler - while most GAN based algos are much better in cases where they are trained on the spcific footage and "recognize" the content and add plausible details - in cases where the don´t recognize anything, the often fall apart (zombie faces) - here the liif apporach could work better.

[asarian]

This is sad. :( I was hoping to use vs-basicvsr, to replace Oyster (as it no longer works); and now this one is gone altogether. And I really need a filter to get rid of those ugly compression artifacts.

[Selur]

vs-basicvsr was replaced with BasicVSR++ (https://github.com/HolyWu/vs-basicvsrpp) quite a while ago,...
(answered in your Oyster thread how to fix the Oyster.py to work with newer Vapoursynth versions)

[PatchWorKs]

@Selur how you judge MIA-VSR ?

Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention
Recently, Vision Transformer has achieved great success in recovering missing details in low-resolution sequences, i.e., the video super-resolution (VSR) task. Despite its superiority in VSR accuracy, the heavy computational burden as well as the large memory footprint hinder the deployment of Transformer-based VSR models on constrained devices. In this paper, we address the above issue by proposing a novel feature-level masked processing framework: VSR with Masked Intra and inter-frame Attention (MIA-VSR). The core of MIA-VSR is leveraging featurelevel temporal continuity between adjacent frames to reduce redundant computations and make more rational use of previously enhanced SR features. Concretely, we propose an intra-frame and inter-frame attention block which takes the respective roles of past features and input features into consideration and only exploits previously enhanced features to provide supplementary information. In addition, an adaptive block-wise mask prediction module is developed to skip unimportant computations according to feature similarity between adjacent frames. We conduct detailed ablation studies to validate our contributions and compare the proposed method with recent state-of-the-art VSR approaches. The experimental results demonstrate that MIAVSR improves the memory and computation efffciency over state-of-the-art methods, without trading off PSNR accuracy.

https://raw.githubusercontent.com/LabShuHangGU/MIA-VSR/refs/heads/main/assets/Results.png


Git: https://github.com/LabShuHangGU/MIA-VSR
Paper: https://arxiv.org/abs/2401.06312

[Selur]

Haven't tried it. Since it's flownet based, maybe styler00dollar or HolyWu will write a Vapoursynth wrapper for it in the future.

[poisondeathray]

Maybe styler00dollar or HolyWu can port evtexture , currently the best 4x video superres by PSNR - REDS4 32.93, On Vid4 29.78

https://github.com/dachunkai/evtexture
https://arxiv.org/abs/2406.13457

I can't get it to run on your own video data set - I can't get the event voxel flow grid generation step modified correctly. The author is supposed to provide an inference script on user's own video data, but has not posted it yet

[ReinerSchweinlin]

This looks promising - at least from the example videos they show...

[PatchWorKs]

Some other upcoming interesting video-enhancing papers in CVPR-2024 (https://github.com/liuzhen03/awesome-video-enhancement?tab=readme-ov-file#cvpr-2024) and ECCV-2024 (https://github.com/liuzhen03/awesome-video-enhancement?tab=readme-ov-file#eccv-2024)...

[poisondeathray]

Some other upcoming interesting video-enhancing papers in CVPR-2024 (https://github.com/liuzhen03/awesome-video-enhancement?tab=readme-ov-file#cvpr-2024) and ECCV-2024 (https://github.com/liuzhen03/awesome-video-enhancement?tab=readme-ov-file#eccv-2024)...

Thx for the heads up

I got FMA-Net from CVPR-2024 to work . The only pretrained model provided was trained on Reds
https://github.com/KAIST-VICLab/FMA-Net

FWIW here is suzie in FFV1 in RGB (bgr0)
https://www.mediafire.com/file/f47ytaqkfx9cw6v/suzie_FMA-Net_Reds_ffv1.mkv/file

FMA-Net is signifcantly faster than something like basicvsr++ or vrt/rvrt - but more aliasing and temporal flickering (many of the metrics commonly used like PSNR/SSIM don't measure temporal characteristics like temporal consistency artifacts)

One quirk I can't figure out is you lose the first and last frames

I couldn't get the other ones that have code published to work on your own datasets

[PatchWorKs]

More VSR, more fun !

StableVSR (https://github.com/claudiom4sir/StableVSR#readme) - Enhancing Perceptual Quality in Video Super-Resolution through Temporally-Consistent Detail Synthesis using Diffusion Models (ECCV 2024)

[PatchWorKs]

[I've deleted my previous reply with some other links]

OK, since I don't wanna use this 3ad "as a notepad" more, I listed those VSR cited here (and some other) under VIDEO (https://github.com/FORARTfe/HyMPS#-1) \ AI-based page (https://github.com/FORARTfe/HyMPS/blob/main/Video/AI-based.md#--) \ Upscalers (https://github.com/FORARTfe/HyMPS/blob/main/Video/AI-based.md#upscalers-): feel free to add or fork it.

Some "real use" (video) shootout would be cool... :thanks:

[Selur]

If anyone tries Distance Ratio Based Adjuster for Animeinter (DRBA) (https://github.com/routineLife1/DRBA) through cvffi (https://github.com/TensoRaws/ccvfi?tab=readme-ov-file) let me know how it compares to newer RIFE models.
Thanks!

Cu Selur

[Z2697]

DRBA is "a control mechanism for Video Frame Interpolation (VFI) networks specifically tailored for anime", it's meant to be used with RIFE or other VFI nets.
Judging by the demo video on github, it's not only tailored for anime, but even a quite specific type of scenes in anime: the background is moving and the forgound character is "semi-static" (she's changing pose but not actually moving, and is the typical shots in anime that has duplicated "frames").

[Selur]

Thanks for clearing that up.

[PatchWorKs]

Better late than never: Happy New Year !

Just discovered that Intel has "own" (server-oriented ?) open source Video Super Resolution library:
Intel Library for Video Super Resolution consist of a few different algorithms including machine learning and deep learning implementations to offer a balance between quality and performance.

We have enhanced the public RAISR (Rapid and Accurate Image Super Resolution), an AI based Super Resolution algorithm https://arxiv.org/pdf/1606.01299.pdf, to achieve better visual quality and beyond real-time performance for 2x and 1.5x upscaling on Intel® Xeon® platforms and Intel® GPUs. Enhanced RAISR provides better quality results than standard (bicubic) algorithms and a good performance vs quality trade-off as compared to compute intensive DL-based algorithms.

Enhanced RAISR is provided as an FFmpeg plugin inside of a Docker container(Docker container only for CPU) to help ease testing and deployment burdens. This project is developed using C++ and takes advantage of Intel® Advanced Vector Extension 512 (Intel® AVX-512) on Intel® Xeon® Scalable Processor family and OpenCL support on Intel® GPUs.
https://private-user-images.githubusercontent.com/89970744/363394862-e28b52c2-67c7-44a9-a66f-df8b355735f9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDAwNDc5OTIsIm5iZiI6MTc0MDA0NzY5MiwicGF0aCI6Ii84OTk3MDc0NC8zNjMzOTQ4NjItZTI4YjUyYzItNjdjNy00NGE5LWE2NmYtZGY4YjM1NTczNWY5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMjAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjIwVDEwMzQ1MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPThiYTgxYWUwNjJhNTU3MTRkZDYyNTk3MmRjZWZjYmY2NDkyMzU4MThiMjUzOWU4MTFhNDk5MTZhMWU5ODExYTQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.-jWF59M1V5ygFWHag8XX-lT2PDj5swqWiCe6Quwq8DA

Check out this interesting doc about performance/usage (https://github.com/OpenVisualCloud/Video-Super-Resolution-Library/blob/main/docs/performance.md) too.

Git: https://github.com/OpenVisualCloud/Video-Super-Resolution-Library#readme

[PatchWorKs]

Hi everyone, just discovered (and tested on a single image) S3Diff that looks VERY promising:
Diffusion-based image super-resolution (SR) methods have achieved remarkable success by leveraging large pre-trained text-to-image diffusion models as priors. However, these methods still face two challenges: the requirement for dozens of sampling steps to achieve satisfactory results, which limits efficiency in real scenarios, and the neglect of degradation models, which are critical auxiliary information in solving the SR problem. In this work, we introduced a novel one-step SR model, which significantly addresses the efficiency issue of diffusion-based SR methods. Unlike existing fine-tuning strategies, we designed a degradation-guided Low-Rank Adaptation (LoRA) module specifically for SR, which corrects the model parameters based on the pre-estimated degradation information from low-resolution images. This module not only facilitates a powerful data-dependent or degradation-dependent SR model but also preserves the generative prior of the pre-trained diffusion model as much as possible. Furthermore, we tailor a novel training pipeline by introducing an online negative sample generation strategy. Combined with the classifier-free guidance strategy during inference, it largely improves the perceptual quality of the super-resolution results. Extensive experiments have demonstrated the superior efficiency and effectiveness of the proposed model compared to recent state-of-the-art methods.

Demo results: https://github.com/ArcticHare105/S3Diff#visual_comparison

Git: https://github.com/ArcticHare105/S3Diff#readme

[Selur]

Since it's aiming at images not video, maybe we will end up with a onnx mask. :)

[PatchWorKs]

Bump.

SeedVR2 (https://iceclear.github.io/projects/seedvr2/) anyone ?

https://www.ainvfx.com/blog/one-step-4k-video-upscaling-and-beyond-for-free-in-comfyui-with-seedvr2/

https://youtu.be/I0sl45GMqNg

[takla]

Demo results: https://github.com/ArcticHare105/S3Diff#visual_comparison
https://imgsli.com/MzAzNjQ1


Such massive bullshit. There is NO WAY you'd ever get that level of detail restoration on any image that wasn't part of the training set.

[ReinerSchweinlin]

Such massive bullshit. There is NO WAY you'd ever get that level of detail restoration on any image that wasn't part of the training set.

Correct.
to be fair - thats the case for EVERY Demo of any algo... No one publishes suboptimal results with footage that does not werk well on the specific trained model.

[Selur]

VS-DistilDRBA (https://github.com/routineLife1/VS-DistilDRBA) might be interesting for some. :)

[Selur]

Such massive bullshit. There is NO WAY you'd ever get that level of detail restoration on any image that wasn't part of the training set.
did a few short tests with a small 3b model:
test1: https://www.mediafire.com/folder/d2sysyer1xe0l/seedvr2_test1
test2: https://www.mediafire.com/folder/wcaihu4kacasl/seedvr2_test2
test3: https://www.mediafire.com/folder/4wq2qdngb25en/seedvr2_test3
each test contains the source I used and the output I got.

=> Nice, but with my current gpu (Geforce RTX 4080 16GB VRAM) this is way too slow to be usable
(second clip took 30min for 520 frames to process, so roughly 17 frames per minute, so 0.28fps for sd to hd)

Cu Selur

[ReinerSchweinlin]

did a few short tests with a small 3b model:
test1: https://www.mediafire.com/folder/d2sysyer1xe0l/seedvr2_test1
test2: https://www.mediafire.com/folder/wcaihu4kacasl/seedvr2_test2
test3: https://www.mediafire.com/folder/4wq2qdngb25en/seedvr2_test3
each test contains the source I used and the output I got.

=> Nice, but with my current gpu (Geforce RTX 4080 16GB VRAM) this is way too slow to be usable
(second clip took 30min for 520 frames to process, so roughly 17 frames per minute, so 0.28fps for sd to hd)

Cu Selur

Nice, thank you :)

Diffusion based Video Enhancers are nice :) The speed seems roughly comparable to what the commercial Topaz Video does in a similar test scenario...

Getting my hopes up... You did not - by any chance - fiddle SeedVR2 into hybrid ?

:)

CU

[Selur]

Getting my hopes up... You did not - by any chance - fiddle SeedVR2 into hybrid ?
No, don't really see a meaningful way to do it atm.
I used it through ComfyUI following https://www.youtube.com/watch?v=MBtWYXq_r60
Just wanted to show that it really produces nice results if you feed it a clean source. :)
Haven't played around with it much simply due to the resource hunger of the whole thing. :)

Cu Selur

[ReinerSchweinlin]

No, don't really see a meaningful way to do it atm.
I used it through ComfyUI following https://www.youtube.com/watch?v=MBtWYXq_r60
Just wanted to show that it really produces nice results if you feed it a clean source. :)
Haven't played around with it much simply due to the resource hunger of the whole thing. :)


:D
I did not really expect it :)

Yes, its very hungry... I tried to use multiple GPUs to speed it up - no luck at the moment...

The diffusion ability indeed is able to introduce new details, very fascinating to see.

[Selur]

The diffusion ability indeed is able to introduce new details, very fascinating to see.
Yup, it's impressive (https://www.mediafire.com/folder/pqr3kma7keqpg/seedvr2_test4) and I'm hoping someone will come up with a way to leverage this in Vapoursynth. :)

[Selur]

Just saw RVRT (Recurrent Video Restoration Transformer) (https://github.com/Lyra-Vhess/vs-rvrt), which atm. doesn't work with Python 3.12+, but I thought it might be interesting for someone here. :)
vs-rvrt is based on https://github.com/JingyunLiang/RVRT

Update 1:
might work with Python 3.12+ now, but fails on portable environment since it can't find cuda,...

Cu Selur

[Selur]

vs-vsrvrt is working now. :) (will probably add it to Hybrids dev next weekend)

[ReinerSchweinlin]

VERY COOL, thank you !

do you have a rough speed estimate from your tests? are 10 x 5090 enough for 1 hour / frame ?

[Selur]

Just tested with
clip = vsrvrt.SuperRes(clip, scale=4, model="reds",preview_mode=True)
and get:
F:\Hybrid\64bit\Vapoursynth>VSPipe.exe c:\Users\Selur\Desktop\test3.vpy -c y4m NUL --progress
RVRT: Using preview mode (lazy chunk processing)
Warning: F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\native\TensorShape.cpp:4316.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

Loading model from: C:\Users\Selur\AppData\Local\vsrvrt\models\001_RVRT_videosr_bi_REDS_30frames.pth
RVRT: Preview mode - 429 frames in 9 chunk(s)
RVRT: Chunk size=64, processing on-demand
Script evaluation done in 2.14 seconds
RVRT: Processing chunk 1/9 (frames 0-63) for preview
RVRT: Auto-tiling: (11, 256, 256)
RVRT: Available VRAM: 14.5 GB, Estimated: 14.5 GB
RVRT: Processing chunk 2/9 (frames 48-111) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 13.2 GB
RVRT: Processing chunk 3/9 (frames 96-159) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 4/9 (frames 144-207) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 5/9 (frames 192-255) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 6/9 (frames 240-303) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 7/9 (frames 288-351) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 8/9 (frames 336-399) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 9/9 (frames 384-428) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
Output 429 frames in 1034.04 seconds (0.41 fps)

using:
clip = vsrvrt.SuperRes(clip, scale=4, model="reds",preview_mode=False)
I get:
F:\Hybrid\64bit\Vapoursynth>VSPipe.exe c:\Users\Selur\Desktop\test3.vpy -c y4m NUL --progress
RVRT: Using chunked processing (video too long for single-pass)
Warning: F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\native\TensorShape.cpp:4316.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

Loading model from: C:\Users\Selur\AppData\Local\vsrvrt\models\001_RVRT_videosr_bi_REDS_30frames.pth
RVRT: Processing 429 frames in 9 chunk(s)
RVRT: Chunk size=64, overlap=16
RVRT: Processing chunk 1/9 (frames 0-63)
RVRT: Auto-tiling: (11, 256, 256)
RVRT: Available VRAM: 14.5 GB, Estimated: 14.5 GB
RVRT: Processing chunk 2/9 (frames 48-111)
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 13.2 GB
RVRT: Processing chunk 3/9 (frames 96-159)
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 4/9 (frames 144-207)
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 5/9 (frames 192-255)
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 6/9 (frames 240-303)
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 7/9 (frames 288-351)
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 8/9 (frames 336-399)
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 9/9 (frames 384-428)
RVRT: Padded chunk from 45 to 64 frames
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Final output shape: torch.Size([1, 429, 3, 1408, 2560])
Script evaluation done in 1050.36 seconds
Output 429 frames in 6.47 seconds (66.28 fps)

so 0.4fps seems to be the correct speed.

Also tested:
clip = vsrvrt.Denoise(clip)
and got:
F:\Hybrid\64bit\Vapoursynth>VSPipe.exe c:\Users\Selur\Desktop\test3.vpy -c y4m NUL --progress
Information: VideoSource track #0 index progress 7%
RVRT: Using chunked processing (video too long for single-pass)
Warning: F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\native\TensorShape.cpp:4316.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

Loading model from: C:\Users\Selur\AppData\Local\vsrvrt\models\006_RVRT_videodenoising_DAVIS_16frames.pth
RVRT: Processing 429 frames in 9 chunk(s)
RVRT: Chunk size=64, overlap=16
RVRT: Processing chunk 1/9 (frames 0-63)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.5 GB, Estimated: 8.7 GB
RVRT: Processing chunk 2/9 (frames 48-111)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 3/9 (frames 96-159)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 4/9 (frames 144-207)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 5/9 (frames 192-255)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 6/9 (frames 240-303)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 7/9 (frames 288-351)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 8/9 (frames 336-399)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 9/9 (frames 384-428)
RVRT: Padded chunk from 45 to 64 frames
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Final output shape: torch.Size([1, 429, 3, 352, 640])
Script evaluation done in 63.90 seconds
Output 429 frames in 0.21 seconds (2090.79 fps)

clip = vsrvrt.Denoise(clip, preview_mode=True)
and got:
F:\Hybrid\64bit\Vapoursynth>VSPipe.exe c:\Users\Selur\Desktop\test3.vpy -c y4m NUL --progress
RVRT: Using preview mode (lazy chunk processing)
Warning: F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\native\TensorShape.cpp:4316.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

Loading model from: C:\Users\Selur\AppData\Local\vsrvrt\models\006_RVRT_videodenoising_DAVIS_16frames.pth
RVRT: Preview mode - 429 frames in 9 chunk(s)
RVRT: Chunk size=64, processing on-demand
Script evaluation done in 2.03 seconds
RVRT: Processing chunk 1/9 (frames 0-63) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.5 GB, Estimated: 8.7 GB
RVRT: Processing chunk 2/9 (frames 48-111) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 3/9 (frames 96-159) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 4/9 (frames 144-207) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 5/9 (frames 192-255) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 6/9 (frames 240-303) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 7/9 (frames 288-351) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 8/9 (frames 336-399) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 9/9 (frames 384-428) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
Output 429 frames in 61.37 seconds (6.99 fps)

for:
clip = vsrvrt.Deblur(clip, preview_mode=True)
I got:
F:\Hybrid\64bit\Vapoursynth>VSPipe.exe c:\Users\Selur\Desktop\test3.vpy -c y4m NUL --progress
RVRT: Using preview mode (lazy chunk processing)
Warning: F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\native\TensorShape.cpp:4316.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

Loading model from: C:\Users\Selur\AppData\Local\vsrvrt\models\005_RVRT_videodeblurring_GoPro_16frames.pth
RVRT: Preview mode - 429 frames in 9 chunk(s)
RVRT: Chunk size=64, processing on-demand
Script evaluation done in 2.14 seconds
RVRT: Processing chunk 1/9 (frames 0-63) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.5 GB, Estimated: 8.7 GB
RVRT: Processing chunk 2/9 (frames 48-111) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.5 GB, Estimated: 8.7 GB
RVRT: Processing chunk 3/9 (frames 96-159) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.5 GB, Estimated: 8.7 GB
RVRT: Processing chunk 4/9 (frames 144-207) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 5/9 (frames 192-255) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 6/9 (frames 240-303) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 7/9 (frames 288-351) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 8/9 (frames 336-399) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 9/9 (frames 384-428) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
Output 429 frames in 60.19 seconds (7.13 fps)

Cu Selur

[ReinerSchweinlin]

thanx. 16gb vram? which gpu did you use?

[Selur]

my Geforce RTX 4080

[Selur]

Uploading a new dev&torch atm., which will be up in 1 1/2 hours which has basic support for RVRT.

Cu Selur

[ReinerSchweinlin]

Uploading a new dev&torch atm., which will be up in 1 1/2 hours which has basic support for RVRT.

Cu Selur
nice :)
thanx !

[Selur]

side note: you might want to enable 'Lazy Chunking'/'preview_mode' and lower the chunking values, if you use the filter in a preview. :)
Without 'Lazy Chunking'/'preview_mode' all frames get processed before outputting a frame,..

[ReinerSchweinlin]

side note: you might want to enable 'Lazy Chunking'/'preview_mode' and lower the chunking values, if you use the filter in a preview. :)
Without 'Lazy Chunking'/'preview_mode' all frames get processed before outputting a frame,..
good to know :) Thanx !

[ReinerSchweinlin]

tried it on my 4060TI 16GB ...

The deblur model works fine, but with the Super resoultion I get no visible difference (enabled DIFF in preview). Running a 1x "clean up pass"

Its a MPEG2 File as MKV, just deinterlaced prior to running RVRT.

I remember in the past that some of the resizers needed actual resize to be enabled in the resize tab - tried that, made no difference.

ah, another thing I noticed:

the tiled processing (deblur in this case) seems to produce a fine line sometimes visible

[Selur]

Resizing does give quite different results for me, depending on the input. Like results are not as aggresive as for example RealESRGAN, but they are there. To to me that seems to work fine.
Since the changes are more subtile, you might loose them depending on the additional resizing that is done to archive your target resolution.
About tiling: I'll probably also add the tile_overlap control to the gui, which should allow to lower / circumvent this effect.

Cu Selur

[Selur]

Added tile_overlap controls to Hybrid, which should help with those lines.

[ReinerSchweinlin]

Resizing does give quite different results for me, depending on the input.

Thanx for reminding - I do "resizing to the optical resolution" quite oten, in my tests here, I just threw the original DVD into hybrid and was wondering why "nothing happened"... After lowering the resolution of the input file, I get usefull resoults :) Thanx for the nice addition :)

Normaly, I prepare files externaly (use some VD stuff I am used to for decades now) and also do the resizing - then throw this prepared file into whatever I want to use next for Upscaling / AI processing..

Is there a way to pre-downscale inside hybrid? Id would be nice if all the filtering, deinterlacing etc.. could be done in hybrid, then lower the resolution and then let the Scalers do their thing (I bet there is one, I am just not bright enough to see it :) )

[Selur]

Is there a way to pre-downscale inside hybrid? Id would be nice if all the filtering, deinterlacing etc.. could be done in hybrid, then lower the resolution and then let the Scalers do their thing (I bet there is one, I am just not bright enough to see it )
Yes.
For deinterlacing and resizing there are special controls under "Filtering->Vapoursynth->Misc->Script->Lower res. before resize / Lower res. before deinterlace", for general filtering you can enable "Filtering->Vapoursynth->Misc->UI->Show 'Gimmick'-controls" which aside from other options, also adds an option to resize before the filter (and undo the resizing if wanted)
https://i.ibb.co/gFT5qm41/grafik.png (https://ibb.co/0VMg4msN)

Cu Selur

Ps.: "Filtering->Vapoursynth->Misc->UI" allows to add quite a few additional controls.

[ReinerSchweinlin]

Great, missed that one so far, will check it out, very cool !!