vs-basicvsr

[ReinerSchweinlin]

Thanx for mentioning, good news :) Did you by chance already put it into hybrid? Haven´t got much time lately, I remember you had a "big" version of hybrid, containing a lot of upscaling filters and additions. If so, I´d be happy to try it out and benchmark to add some numbers to you above test.

[Selur]

Did not add it to Hybrids torchAddon and atm. I'm not planning to add it since it's like 20% slower on my machine.

[AIisCool]

Has anyone found the super resolution of BasicVSR++ to cause strange effects on certain video?

Before:
https://i.imgur.com/Z79WIhe.png

After:
https://i.imgur.com/V6ns97O.png


It doesn't matter which I use, REDS, Vimeo-90K (BI), Vimeo-90K (BD), or even the NTIRE 2021 Video Super-Resolution, it creates harsh lines and distortions on this part of the video, and oddly only this part.

[markanini]

How do I modify the google colab to use my video?

[Selur]

@AIisCool: got a short sample of the source, so others can try to reproduce this?
@markanini: sorry, no clue, since I'm not using the colab thing.

[gmail123]

The order in which I installed BasicVSR++:
1、I have installed：Anaconda3-2022.10-Windows-x86_64、cuda_11.6.0_511.23_windows、python-3.10.8-amd64、VapourSynth64-R59
2、PyTorch:
conda create -n PyTorch python=3.10
https://pytorch.org/get-started/locally/
conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia
3、INSTALLING NUMPY
conda create -n my-env
conda activate my-env
4、mmcv-full ：
pip install -U openmim
mim install mmcv-full
but：
5、pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp
worng:https://imgloc.com/i/DQKrp
https://i.328888.xyz/2022/12/26/DQKrp.md.jpeg

[gmail123]

Where is the installation error?
How to install it correctly?
THX

[poisondeathray]

No module named "vapoursynth" suggests vapoursynth is not installed. I'm guessing it might be an issue with anaconda

[poisondeathray]

Anyone try RealBasicVSR yet? It seems to produce oversmooth ,blurry, reduced detail results .

VRT SR is temporal and slightly higher metrics than basicvsr++ in the papers . In my short tests, it produces similar results to basicvsr++, but significantly slower than basicvsr++ . The provided pretrained models were trained on similar sets, and you get similar artifacts under similar conditions

There is a new one RVRT, but I can't get it to run

[Selur]

Haven't tried RealBasicVSR, will look at it once it can be used through Vapoursynth.
btw. I just saw https://github.com/HolyWu/vs-femasr (https://github.com/chaofengc/FeMaSR), but haven't played around with it so far. :)

Cu Selur

[gmail123]

No module named "vapoursynth" suggests vapoursynth is not installed. I'm guessing it might be an issue with anaconda

But i have installed vapoursynth, and i can use vapoursynth:sly:

[poisondeathray]

But i have installed vapoursynth, and i can use vapoursynth:sly:

Can you run vapoursynth in the anaconda environment ? I remember some people had problems before.

[Selur]

Did a quick test of vs-femasr and in general the results seem nice, but atm. it does create some strange highlights.
https://i.ibb.co/qWf08g1/grafik.png (https://ibb.co/4YGsLSN)
Also TensorRT support does not work fo rme: https://github.com/HolyWu/vs-femasr/issues/2

Suzi:
https://i.ibb.co/XYP4M98/grafik.png (https://ibb.co/JRn48hK)

Cu Selur

[Selur]

https://github.com/raywzy/Bringing-Old-Films-Back-to-Life looks interesting.

[gmail123]

Thanks to HolyWu I got vs-basicvsrpp working using:
python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11/index.html
python -m pip install tqdm
python -m pip install opencv-python
python -m pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp
:)

Cu Selur

I ran the same command, but it failed.
Can you explain what software and operating environment has been installed?
Thank you very much!

[Selur]

I don't really get the question. But maybe reading [INFO] Portable Vapoursynth (Windows) with pytorch&co (https://forum.selur.net/thread-2918.html) helps, those are the steps I used back then to set up my portable Vapoursynth.
I can also send you a link (via pm) to a download of my latest portable Vapoursynth that I use in Hybrid, if that would help.

Cu Selur

[gmail123]

I don't really get the question. But maybe reading [INFO] Portable Vapoursynth (Windows) with pytorch&co (https://forum.selur.net/thread-2918.html) helps, those are the steps I used back then to set up my portable Vapoursynth.
I can also send you a link (via pm) to a download of my latest portable Vapoursynth that I use in Hybrid, if that would help.

Cu Selur
I will try it.I've PM you. thanks a lot:thanks:

[gmail123]

@Selur Special thanks to you! I have successfully installed vs-basicvsrpp.
Test video resolution: 2064*1416, 3070, 0.84 fps:D

script:
from vsbasicvsrpp import BasicVSRPP
ret = core.resize.Bicubic(ret, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
ret = BasicVSRPP(ret,model=3,fp16=True,tile_x=360,tile_y=480)
ret = core.resize.Bicubic(ret, format=vs.YUV420P12, matrix_s="470bg", range_s="limited")
Can it be optimized?:rolleyes:

[Selur]

Can it be optimized?
For speed, probably not. (aside from a hardware upgrade, or maybe lower the interval, but that will also influence the processing strength)

Cu
Selur

[kedautinh12]

Yeah, you're right, Cuda plugins very slow now

Cu
Kedautinh12 :D

[Selur]

Cuda plugins very slow now
Not really Cuda, more like most of the ai model based stuff. ;)

[kedautinh12]

You forgot "Cu Selur" :D

[Selur]

¯\_(ツ)_/¯

[Selur]

A few more examples of vs-femasr:
not bad:
https://imgsli.com/MTQ1ODI3
https://imgsli.com/MTQ1ODI4
https://imgsli.com/MTQ1ODI5
https://imgsli.com/MTQ1ODM1
nice detail recovery, but partially too sharp (too much contrast) for my liking, and if you have noise&co that might get enhanced too.
https://imgsli.com/MTQ1ODM5

Cu
Selur

[Selur]

High Performance GMFSS with RIFE and GAN for Anime Video Frame Interpolation: https://github.com/HolyWu/vs-gmfss_union

[kedautinh12]

Wow, VS development very fast. When i think Avs+ can reach to VS but after Pytorch, NumPy releases VS development more than Avs+ very much

[ChaosKing]

Some quick performance numbers with RTX 3070 TI & 720x480 clip (noisy source):
tested with vsedit, so number should be slighlty higher bcs of the editors small overhead.

gmfss_union(clip) # ~14fps
gmfss_union(clip, trt=True) # ~17fps, tensor cache took a long time to build
gmfss_union(clip, num_streams=2) # ~17.5fps
gmfss_union(clip, num_streams=3) # ~18.9fps
gmfss_union(clip, trt=True, num_streams=2) # ~21.5fps
gmfss_union(clip, trt=True, num_streams=3) # ~22.5fps

[Selur]

What speed do you get with RIFE and the anime model?

[ChaosKing]

https://github.com/HolyWu/vs-rife runs with 175 fps
clip = RIFE(clip)

https://github.com/HomeOfVapourSynthEvolution/VapourSynth-RIFE-ncnn-Vulkan
rife.RIFE(clip, model=3) # ~78fps

Look at this perfect frame :devil:
https://i.imgur.com/r2GAnRI.jpeg

[Selur]

Okay, so it's really slow. :)
(for RIFE enabling sc and setting sc_threshold to something like 0.1 might help a bit)

...

Played around with it and probably won't add this to Hybrid since I don't see any gain from it over RIFE&co atm.

[~ VEGETA ~]

tried installing vsbasicvsr++ but got this error: https://pastebin.com/6VWwaXRd

how can i solve it?

[Selur]

Probably an issue with the cuda dependencies,
here's what I did last time I setup the environment:

I created a new empy Vapoursynth-folder
downloaded 'Windows embeddable package (64-bit)' from https://www.python.org/downloads/release/python-3108/ (3.11 isn't supported atm. by Vapoursynth)
extracted the Python download into the download into the 'Vapoursynth'-folder
downloaded 'VapourSynth64-Portable-R60.7z' from https://github.com/vapoursynth/vapoursynth/releases
extracted the Vapoursynth portable download into the 'Vapoursynth'-folder
downloaded get-pip.py from https://bootstrap.pypa.io/get-pip.py and save it into the 'Vapoursynth'-folder
opened a 'Windows Command Prompt'-window and navigate into the 'Vaporusynth'-folder
installed pip by calling :
pyhton get-pip.py
opened the python310._pth in a text addition and added the following to lines above anything else in that file and saved the file
Scripts
Lib\site-packages

installed VSGAN

python -m pip install vsgan
python -m pip install torch===1.13.0+cu117 torchvision==0.14.0 -f https://download.pytorch.org/whl/torch_stable.html
installed BASICVSR++

python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu117/torch1.13/index.html
python -m pip install tqdm
python -m pip install opencv-python
python -m pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp


installted RIFE

downloaded https://github.com/HolyWu/vs-realesrgan/releases/download/v4.0.1/CUDA-11.7_cuDNN-8.6.0_TensorRT-8.5.2.2_win64.7z (there's a newer versio now)

extractedtensorrt-8.5.2.2-cp310-none-win_amd64.whl into my Vapoursynth folder
python -m pip install tensorrt-8.5.2.2-cp310-none-win_amd64.whl
deleted tensorrt-8.5.2.2-cp310-none-win_amd64.whl
python -m pip install --upgrade vsrife
created a folder 'vsrife_dependencies' inside the Vapoursynth folder and copied the other files form the CUDA-11.7_cuDNN-8.6.0_TensorRT-8.5.1.7_win64.7z into it. This folder need to be added to when using vs-rife using:
import site
# add vs-rife dependencies to the path
path = site.getsitepackages()[0]+'/torch_dependencies/'
path = path.replace('\\', '/')
import os
os.environ["PATH"] = path + os.pathsep + os.environ["PATH"]
downloaded the models using:
set PATH=i:\Hybrid\64bit\Vapoursynth\torch_dependencies\bin;%PATH
python -m vsrife

installed SWINIR

python -m pip install --upgrade vsswinir
python -m vsswinir


installed DPIR and onnxruntime-gpu

python -m pip install --upgrade vsdpir
python -m pip install --upgrade onnxruntime-gpu
python -m vsdpir

created a onnxruntime_dlls folder inside the Vapoursynth folder.
I copied from cudnn-11.4-windows-x64-v8.2.4.15.zip and NVIDIA CUDA SDK 11.4.1 runtimes the following files into the folder:

cublas64_11.dll
cublasLt64_11.dll
cudart64_110.dll
cudnn_cnn_infer64_8.dll
cudnn_ops_infer64_8.dll
cudnn64_8.dll
cufft64_10.dll
cufftw64_10.dll
msvcp140.dll
nvinfer.dll
nvinfer_builder_resource.dll
nvinfer_plugin.dll
nvonnxparser.dll
nvparsers.dll
vcruntime140.dll
vcruntime140_1.dll

When using vsdir or realesrgan in a Vapoursynth script:

# Import libraries for onnxruntime
import site
import ctypes
path = site.getsitepackages()[0]+'/onnxruntime_dlls/'
ctypes.windll.kernel32.SetDllDirectoryW(path)
needs to be added to the script to load the dlls.
installed REALESRGAN (which also uses onnxruntime)

python -m pip install --upgrade vsrealesrgan
python -m vsrealesrgan

Installed HINet

python -m pip install --upgrade vshinet
python -m vshinet



Cu Selur

Ps.: also send you a link to my current Hybrid dev and torch addon, going to bed now. ;) You might want to checkout

[Selur]

Anyone managed to get BasicVSR++ v2.0.0 running in a portable Vapoursynth setup?
I'm stuck https://github.com/HolyWu/vs-basicvsrpp/issues/29 installing mmcv-full. :(

Cu Selur

[Selur]

HolyWu could help, using:
python -m pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117
python -m pip install -U openmim
python -m mim install "mmcv-full>=1.7.1"
python -m pip install -U vsbasicvsrpp
python -m vsbasicvsrpp
worked :)

Cu Selur

[Selur]

With the release of Vapoursynth R62, it's either Python 3.11 or Python 3.8. (Python 3.8 is still there to support old Windows versions and Python 3.10 support was dropped not to having to support multiple Python versions.)
So Python 3.11 is probably what most folks want to go for, but since there are no cp311 pytorch versions over at https://download.pytorch.org/whl/torch/ for Windows I was wondering if someone got a way to get all the ai stuff working with Python 3.11 and Vapoursynth R62?

Cu Selur

[Selur]

btw. did anyone test https://github.com/HolyWu/vs-codeformer ?
https://imgsli.com/MTY2MTM5
Suzie: https://imgsli.com/MTY2MTQz

[ChaosKing]

lol they get entirely new faces :eek:

[Selur]

Yes, but in combination with realesrgan this might have its uses. :)
=> I have to check what happens if this is applied to different actors,...
https://imgsli.com/MTY2MTQ0
https://imgsli.com/MTY2MTQ2
https://imgsli.com/MTY2MTQ3
https://imgsli.com/MTY2MTQ5
seems more like a gimmick filter than really usable atm., but its impressive.

Cu Selur

[Dogway]

Some people mix it with GFPGAN or other face models, haven't played with NN but it seems there's more work to it than a simple button push.
Usually whenever I use NN upscaling I always follow a downscale, to hide the pristine AI look and blend everything together.
Very excited to get into this AI craze soon.

[HolyWu]

Yes, but in combination with realesrgan this might have its uses. :)
=> I have to check what happens if this is applied to different actors,...
https://imgsli.com/MTY2MTQ0
https://imgsli.com/MTY2MTQ2
https://imgsli.com/MTY2MTQ3
https://imgsli.com/MTY2MTQ5
seems more like a gimmick filter than really usable atm., but its impressive.

Cu Selur

You should also try weight=1 if you prefer a higher-fidelity result.

[Selur]

Will do.
Main issue I have atm. with it, is that when the head covers, most of the image, only parts of the head seem to be processed.
This kind of ruins the image if the person has longer hair.
https://i.ibb.co/3c2jNcC/grafik.png (https://ibb.co/Xt1KstC)
(using detector = 1, doesn't help either)
https://imgsli.com/MTY2MjEw Suzie with weight=1

[ChaosKing]

At least the face now looks like it belongs to the same person :devil:

[Selur]

Here's an example of weight 0 and weight 1: https://imgsli.com/MTY2Mjc4

[Selur]

Did some further testing and if upscale is > 1 (weight=1) and the upscaling is later undone it basically can be used as a 'face detail enhancer' which might be interesting.
Also in combination with FeMaSR (with weight<1) it can be used to fix the face distortions FeMaSR does.
https://i.ibb.co/1TZ0yRs/grafik.png (https://ibb.co/Fz6hZ8s)https://i.ibb.co/JqnSm6n/grafik.png (https://ibb.co/G97rnY7)https://i.ibb.co/3m74tKM/grafik.png (https://ibb.co/k2x5YZ9)
This does not fix the issue with half enhanced heads, but seems to be a good way to use the filter. :)
SwinIRx4+vsCodeFormer: https://imgsli.com/MTY2NDA3, looks nice frame-by-frame, but isn't really temporally stable (https://forum.videohelp.com/attachments/70126-1680433345/SwinIR+CodeFormer_lossless.mkv) => adding 3xQTGMC_type3 (https://forum.videohelp.com/attachments/70128-1680434869/3xQTGMC_type3.mp4).

Cu Selur

Ps.: got a nice example of what weight 0 can do: https://imgsli.com/MTY2NDI2,...

[Selur]

Okay, small update in regard to Vapoursynth R62:
TensorRT doesn't support Python 3.11 atm.
Latest TensorRT-package (TensorRT-8.6.0.12.Windows10.x86_64.cuda-11.8.zip (https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/8.6.0/zip/TensorRT-8.6.0.12.Windows10.x86_64.cuda-11.8.zip)) from https://developer.nvidia.com/tensorrt does not contain (TensorRT-8.6.0.12.Windows10.x86_64.cuda-11.8.zip\TensorRT-8.6.0.12\python) a cp311 file. :(
So all the filters that use TensorRT won't work. :(
=> Hoping NVIDIA will release a Python 3.11 file in the near (who knows) future.

Cu Selur

[Selur]

Maybe interesting to some:
https://github.com/HolyWu/vs-grlir uses real-world image super-resolution model of GRL for Image Restoration (https://github.com/ofsoundof/GRL-Image-Restoration)
https://imgsli.com/MTcxMDA1

Cu Selur

[ReinerSchweinlin]

Thanx for all the info.. I think its about time to check out your extended version of hybrid, including all the nice "AI" stuff not included in the normal release :)

[Selur]

@RainerSchweinlin: Send you a pm with links to my current dev + torchAddon. :)

[ReinerSchweinlin]

@RainerSchweinlin: Send you a pm with links to my current dev + torchAddon. :)

Merci :)

[Selur]

btw. vs-grlir just got updated and not supports multiple models, cuda_graphs, fp16. :)
:param model: Model to use.
0 = Blind Image SR
1 = Defocus Deblurring
2 = Motion Deblurring (GoPro)
3 = Motion Deblurring (RealBlur-J)
4 = Motion Deblurring (RealBlur-R)
5 = Demosaicking
6 = Denoising (sigma 15)
7 = Denoising (sigma 25)
8 = Denoising (sigma 50)
9 = JPEG compression artifact removal (quality 10)
10 = JPEG compression artifact removal (quality 20)
11 = JPEG compression artifact removal (quality 30)
12 = JPEG compression artifact removal (quality 40)
13 = Classical Image SR (scale 2)
14 = Classical Image SR (scale 3)
15 = Classical Image SR (scale 4) source: https://github.com/HolyWu/vs-grlir/blob/master/vsgrlir/__init__.py
Models 0, 12-15 are for upscaling, the others are for filtering. :)
Nice!

Cu Selur