vs-basicvsr

[PatchWorKs]

Without:
a. pytorch support
b. rewriting of the exitistn plugins
-> no
Well, pytorch support SEEMS on the go:
https://github.com/pytorch/pytorch/issues/47702


I can run it for single pics fine, but I get like 0.005fps for sd->hd on my system, which simply is too slow for me to be usable.
Of course (that's why a colab can help), but can you please post some visual results ?

Thx !

[Selur]

here are a few examples:
https://i.ibb.co/Pg3nJBm/SwinIR.png (https://ibb.co/2dmQRXn)
used:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DeinterlaceFilter/TIVTC/libtivtc.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/d2vSource/d2vsource.dll")
# source: 'E:\clips\VTS_02_1-Sample-Beginning.demuxed.m2v'
# current color space: YUV420P8, bit depth: 8, resolution: 720x480, fps: 29.97, color matrix: 470bg, yuv luminance scale: limited, scanorder: telecine
# Loading E:\clips\VTS_02_1-Sample-Beginning.demuxed.m2v using D2VSource
clip = core.d2v.Source(input="E:/Temp/m2v_5d36292e1f7f53fd6e26be51d50bbf8c_853323747.d2v")
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 29.97
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# Deinterlacing using TIVTC
clip = core.tivtc.TFM(clip=clip)
clip = core.tivtc.TDecimate(clip=clip)# new fps: 23.976
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
# DEBUG: vsTIVTC changed scanorder to: progressive
# cropping the video to 704x480
clip = core.std.CropRel(clip=clip, left=8, right=8, top=0, bottom=0)
from vsswinir import SwinIR
# adjusting color space from YUV420P8 to RGBS for VsSwinIR
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# resizing using SwinIR
clip = SwinIR(clip=clip, task="real_sr_large", scale=4, tile_x=352, tile_y=240, tile_pad=16, device_type="cuda", device_index=0) # 2816x1920
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1474, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()
https://i.ibb.co/dpykKQ0/Swin-IR-clip-task-classical-sr-scale-4.png (https://ibb.co/z2wPfSh)
https://i.ibb.co/k2D9LtV/Swin-IR-clip-task-lightweight-sr-scale-4.png (https://ibb.co/whQwnfZ)
https://i.ibb.co/51Qb71F/Swin-IR-clip-task-real-sr-scale-4.png (https://ibb.co/vcN0tcY)
https://i.ibb.co/pryQgfC/Swin-IR-clip-task-real-sr-large-scale-4.png (https://ibb.co/3Yzvwpt)

some more using RealSR_large:
https://i.ibb.co/YR6nGnf/swinir-real-sr-large-4.png (https://ibb.co/mDmZ7Zh)
https://i.ibb.co/M5TF29C/swinir-real-sr-large-4-2.png (https://ibb.co/rfPD7vw)
https://i.ibb.co/MSSQt2m/swinir-real-sr-large-4-4.png (https://ibb.co/C88DY1f)

Cu Selur

[PatchWorKs]

Very nice results (especially on the faces), even if - of course - not yet optimal for everything...

...btw I hope to see a SwinIR version optimized for videos too.

[Selur]

not yet optimal for everything...
Espeically the last example shows some real issues ;)

...btw I hope to see a SwinIR version optimized for videos too.
first a way faster version would be needed

[PatchWorKs]

first a way faster version would be needed
Already asked, of course: https://github.com/JingyunLiang/SwinIR/issues/47

Note: I've also just "fed" @HolyWu with this awesome collection, let's see if other interesting "VS-ports" will come out...

[ReinerSchweinlin]

Very nice results (especially on the faces), even if - of course - not yet optimal for everything...

...btw I hope to see a SwinIR version optimized for videos too.
Is a video version planed?

for upscaling, algos like esrgan (single image) are not very suitable for real-life content. Too much flickering. So unless SwinIR doesn´t get some extensions for multi-frame usage / flow detection /whatever, one will always get flickering / stutters / inkonsistent movement...

[aegisofrime]

Nvidia just open sourced Nvidia image scaling... Would this be a candidate for the next filter?

[Selur]

Has anyone tested https://github.com/HolyWu/vs-hinet ?

Here are a few screen shots: (not sure what to make of them and for what content this is really useful)

Mode: Deblur GoPro
https://i.ibb.co/0MjBKN3/deblur-Go-Pro.png (https://ibb.co/QNF8kBG)
Mode: Deblur REDS
https://i.ibb.co/WyWdJyG/deblur-REDS.png (https://ibb.co/xjScTjX)
Mode: denoise
https://i.ibb.co/2SP8LJ7/denoise.png (https://ibb.co/pQZWsqv)
Mode: derain
https://i.ibb.co/zP0j9sH/derain.png (https://ibb.co/wpx3Gyr)

[PatchWorKs]

According to your tests on that frame, the highest fidelity seems to be achieved by derain model, btw here are some questions:

how fast is it ?
how does it performs (in terms of both speed and fidelity) compared to xClean (https://forum.doom9.org/showthread.php?t=183245) ?
its description claims that it perform "restoring" (aka denoising ?) function, but since is BasicSR-based does it upscale too ?


Last but not least (even if OT): did you tried RIFE ?
https://github.com/HolyWu/vs-rife

[Selur]

speed: ~2-3fps for sd content, so not that slow
xClean: no clue about xClean, haven't played around with it too many options for my taste (+ would need to add znedi3 and nnedi3cl support to it)
upscale: at least the current interface offers no upscaling and the method does not upscale
rife: yes, I like it (with sceneChange added). Waiting for FrameRateConverter to properly support it

[lansing]

I don't understand, what was the filter supposed to do??

[Selur]

I'm not sure either which is why is simply took a clip and made some screen shots and asked whether someone has experience with it,..from what I have seen so far each of the models has a different effect.
So far I would say:
deblur GoPro: seems to mainly sharpen and 'add details' (seems to causes artifacts on cartoons)
deblur REDS: strop denosing and sharpening (seems to only suited for cartoon content)
denoise: also seems to be useful for really noise vhs sources or cartoons
derain: milder denoise/degrain, unsure where this could be useful, seems to remove less when used on more detailed content,...
-> so no real clue

[poisondeathray]

for hinet: the main purpose is single image deblurring . It's not a temporal algorithm. The type of blur it's most suited to - is mostly camera motion blur / shutter blur.

"derain" is for removing "rain" (it's not "degrain") . ie. short streaks of white, like rain drops. It actually works ok on random dark film scratches (not persistent , long scratches) if you invert the src.

Another semi popular deblurring algorithm is deblurGANv2 (also works best on camera motion blur)

[lansing]

for hinet: the main purpose is single image deblurring . It's not a temporal algorithm. The type of blur it's most suited to - is mostly camera motion blur / shutter blur.

"derain" is for removing "rain" (it's not "degrain") . ie. short streaks of white, like rain drops. It actually works ok on random dark film scratches (not persistent , long scratches) if you invert the src.

Another semi popular deblurring algorithm is deblurGANv2 (also works best on camera motion blur)

Ok thanks. So it's targeting high resolution photo images, not videos.

[poisondeathray]

Ok thanks. So it's targeting high resolution photo images, not videos.

Not necessarily; the gopro model is trained on gopro video . The resolution isn't "high", though

All the single image algorithms tend to have some temporal inconsistencies in the end result when used on video .

YMMV for all these research projects

[PatchWorKs]

Investigating Tradeoffs in Real-World Video Super-Resolution (aka RealBasicVSR) (https://github.com/ckkelvinchan/RealBasicVSR) has just been released:

https://user-images.githubusercontent.com/7676947/143370499-9fe4069b-46cc-4f12-b6ff-5595e8e5e0b8.mp4

https://user-images.githubusercontent.com/7676947/143370350-91f751f3-0f33-4ee4-9b1a-b9279bf41c18.mp4

https://user-images.githubusercontent.com/7676947/143370556-9e7019d4-e718-46af-859f-54d5576cd370.mp4

https://user-images.githubusercontent.com/7676947/143370859-e0293b97-f962-476f-acf8-14fad27cea77.mp4

[kedautinh12]

Wow, interesting results

[Selur]

Doesn't really look better than BasicVSR++ to me atm. but in case someone ports it to Vapoursynth we will see more. :)

[Gravitator]

At the end, stripes appear under the left arm in Titanic.

[PatchWorKs]

Doesn't really look better than BasicVSR++ to me atm. but in case someone ports it to Vapoursynth we will see more. :)
Some other "real world" comparisons done with Colab Demo (https://colab.research.google.com/drive/1JzWRUR34hpKvtCHm84IGx6nv35LCv20J?usp=sharing):
https://user-images.githubusercontent.com/7676947/144365675-5c558a89-8b7f-4af9-a696-228bc73ab56e.png

[Selur]

Here's a nice example what BasicVSR++ can do:
https://i.ibb.co/WHyMhX5/Basic-VSR-Model1.png (https://ibb.co/HKzbSy4)

Cu Selur

[aegisofrime]

Can I get a bit of help here? After finally setting up the dependencies (they are a bit convoluted for this plugin honestly), I keep getting the following error:

vpy [error]: CUDA out of memory. Tried to allocate 1.24 GiB (GPU 0; 8.00 GiB total capacity; 5.08 GiB already allocated; 0 bytes free; 5.21 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation occurred while getting frame 0
x265 [error]: unable to open input file <P:\ISO\MainMovie\ME_021\OME021_temp\OME021.vpy>

My Script is very simple,


clip = core.avs.DGSource(r"D:\ISO\Test.dgi")
#core.max_cache_size=16384
#core.num_threads = 8
import havsfunc as haf
core.max_cache_size=16384
clip = haf.QTGMC(clip, Preset='Placebo', TFF=True, Denoiser="KNLMeansCL")
from vsbasicvsrpp import BasicVSRPP
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
clip = BasicVSRPP(clip,model=1)
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P12, matrix_s="470bg", range_s="limited")
clip = core.resize.Spline36(clip, 1440,960)
clip.set_output()


Input is just a 720x480 DVD video, while my GPU is a RTX 3060Ti, so it only has got 8GB of RAM. It should still be enough right? How can I tweak the arguments so that this works at all?

[Selur]

adding:
,fp16=True
and/or
,tile_x=360,tile_y=480
to the BasicVSRPP code should help.

Cu Selur

[PatchWorKs]

For those - like me - who don't own powerful GPUs, I just asked kodxana (aka Mediator2011) (https://github.com/kodxana):

https://github.com/kodxana/VapourSynthColab/issues/1

[PatchWorKs]

Hi there, can someone make some (performances, particularry) tests of VapourSynth Single Image Super-Resolution Generative Adversarial Network (GAN) (https://github.com/rlaPHOENiX/VSGAN) ?
(note: I tried their colab (https://colab.research.google.com/github/rlaphoenix/VSGAN/blob/master/VSGAN.ipynb) without success)

Thanks in advance !

[Selur]

What do you have in mind?

[PatchWorKs]

What do you have in mind?
Oh, well, it would be great to have a colab "service" where upload (old) SD stuff and download (x265-10bit encoded) 1080p...

[Selur]

Ah okay, sounded like wanted some benachmarks. No clue about colab. :)

[PatchWorKs]

Ah okay, sounded like wanted some benachmarks. No clue about colab. :)
Oh, well, of course it would be great to have some tests of newest algos:

https://github.com/Weepingchestnut/Awesome-Video-Super-Resolution

[poisondeathray]

PP-MSVSR featured in PaddleGAN looks like the next improvement over basicvsr++ . I'm trying to get it to work, but no luck so far with modifying config files and paths

https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US/tutorials/video_super_resolution.md
https://arxiv.org/pdf/2112.02828.pdf

[PatchWorKs]

PP-MSVSR featured in PaddleGAN looks like the next improvement over basicvsr++ . I'm trying to get it to work, but no luck so far with modifying config files and paths
...to me could be VERY interesting too:

OverNet: Lightweight multi-scale super-resolution with overscaling network (https://openaccess.thecvf.com/content/WACV2021/papers/Behjati_OverNet_Lightweight_Multi-Scale_Super-Resolution_With_Overscaling_Network_WACV_2021_paper.pdf)

https://arxiv.org/abs/2008.02382

Unfortunally their GH seems empty:
https://github.com/pbehjatii/OverNet

[ReinerSchweinlin]

thanx for mentioning it,m looking good :)

there is hope, GITHUB says:

This repository is an official PyTorch implementation of the paper "OverNet: Lightweight multi-scale super-resolution with overscaling network" from WACV 2021. The code will be released soon.

[poisondeathray]

I requested VRT: A Video Restoration Transformer

https://github.com/JingyunLiang/VRT

https://github.com/HolyWu/vs-basicvsrpp/issues/21

[PatchWorKs]

Here's another kind of enhancer that claims to make HDR from SDR:
https://github.com/nothinglo/Deep-Photo-Enhancer

https://raw.githubusercontent.com/nothinglo/Deep-Photo-Enhancer/master/images/teaser.png

Results on video:
https://www.youtube.com/watch?v=d7OXb2sqoec

Pytorch implementation:
https://github.com/mtics/deep-photo-enhancer

[Selur]

HolyWu posted an update for vs-dpir. I'm not sure what provider uses what onxruntime.
Okay, got that cleared up, see: https://github.com/HolyWu/vs-dpir/issues/20

CUDA:
encoded 429 frames, 8.83 fps, 1117.06 kb/s
DIRECTML:
encoded 429 frames, 6.03 fps, 1117.74 kb/s
CPU:
encoded 429 frames, 0.49 fps, 1116.08 kb/s

(can't test CUDA TensorRT since I only got an old Geforce GTX 1070ti)
-> seems that onnxruntime-gpu is the way to go for CUDA cards.

[cubicibo]

Here's another kind of enhancer that claims to make HDR from SDR:
https://github.com/nothinglo/Deep-Photo-Enhancer

[Image]

I wouldn't say this is an improvement. Look at the halo around the "PUMA" on the shirt.

[Selur]

Did anyone get vs-dpir installed when using Vapoursynth R58-RC1? (https://github.com/HolyWu/vs-dpir/issues/22)
vs-basicvsrpp also doesn't work since mmcv_full is meant for Python 3.9 which isn't supported in Vapoursynth R58 (only 3.8 and 3.10) (https://github.com/HolyWu/vs-basicvsrpp/issues/23)
Similar for vs-rife and vs-realesrgan.

Cu Selur

[Selur]

Thanks to HolyWu I got vs-basicvsrpp working using:
python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11/index.html
python -m pip install tqdm
python -m pip install opencv-python
python -m pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp
:)

Cu Selur

[ReinerSchweinlin]

Thanks to HolyWu I got vs-basicvsrpp working using:
python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11/index.html
python -m pip install tqdm
python -m pip install opencv-python
python -m pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp
:)

Cu Selur

Nice :) Thanx to you both :)

[PatchWorKs]

Thanks to HolyWu I got vs-basicvsrpp working using:
python -m pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.11/index.html
python -m pip install tqdm
python -m pip install opencv-python
python -m pip install --upgrade vsbasicvsrpp
python -m vsbasicvsrpp
:)

Cu Selur

Holy & Slur contributions & testing tandem rockz !

[PatchWorKs]

How about (new ?) VRT - A Video Restoration Transformer ?

https://github.com/JingyunLiang/VRT

Tests would be great.

[Selur]

Maybe vs-mlrt (https://github.com/AmusementClub/vs-mlrt) will add support for it, based onhttps://docs.microsoft.com/en-us/windows/ai/windows-ml/tutorials/pytorch-convert-model conversion from pth to onnx models might be possible.

[WolframRhodium]

Sadly, ther non-supported operations (deformable convolution and others) in VRT prevent the conversion to the standard onnx format.

[Selur]

argh,.. good to know, thanks for looking into it. :)

[~ VEGETA ~]

is there any good use of this filter on high quality anime encode releases?

I mean if sharpness and detail preservation are wanted, then descaling is the way to go if can be done. Plus some high quality denoisers and careful masked debanding can yield best results.

what can this filter offer?

[Selur]

Assuming you are referring to BasicVSR++, it's meant for restoration so using it on a 'high quality' sounds like wanting to give a new car to a restaurator,...
As a side note: There are tons of Anime trained models (https://upscale.wiki/wiki/Model_Database) for VSGAN which might be more suited, detail preservation totally depends on the model used.
Using ml stuff on high resolution content is really slow with current hardware, so if high quality also means high source resolution be prepared to wait.

Cu Selur

[poisondeathray]

I've modified __init__.py in the vsbasicvsrpp folder to include the NTIRE 4x VSR model if anyone is interested

https://www.mediafire.com/file/k0bbai32yrk7y05/__init__(modified+bvsrpp+to+incl+NTIRE+VSR).zip/file

https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/basicvsr_plusplus

Extract __init__.py from the zip archive ,and download the model and place in same vsbasicvsrpp directory
https://download.openmmlab.com/mmediting/restorers/basicvsr_plusplus/basicvsr_plusplus_c128n25_ntire_vsr_20210311-1ff35292.pth


model: Model to use.
0 = REDS
1 = Vimeo-90K (BI)
2 = Vimeo-90K (BD)
3 = NTIRE 2021 Video Super-Resolution
4 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 1
5 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 2
6 = NTIRE 2021 Quality enhancement of heavily compressed videos Challenge - Track 3


I inserted it as model "3" because models 0 to 2 are 4x models, and NTIRE 2021 Video Super-Resolution is a 4x model too; and that bumps the NTIRE 1x models down

The model uses mid_channels 128 and num_blocks 25 (instead of 64 and 7), as written in the pth name, and in the py file, so I modified those entries and the other relevant ones
https://github.com/open-mmlab/mmediting/blob/master/configs/restorers/basicvsr_plusplus/basicvsr_plusplus_c128n25_600k_ntire_vsr.py

It seems to work ok, and results are similar to REDS model, maybe slightly better or on some sources. The py says it was trained on REDS data set with spynet_20210409-c6c1bd09.pth pretrain

[Selur]

Nice.Thanks!
(added a request in the vs-basicvsrpp bugtacker to add this)

[Selur]

btw. HolyWu just release vs-dpir-ncnn version :)

python -m pip install -U vsdpir_ncnn
python -m pip install --upgrade https://github.com/HolyWu/ncnn/releases/download/1.0.20220910/ncnn-1.0.20220910-cp310-cp310-win_amd64.whl
python -m vsdpir_ncnn

usage:
from vsdpir_ncnn import dpir

ret = dpir(clip)
options:
def dpir(
clip: vs.VideoNode,
strength: float | vs.VideoNode | None = None,
task: str = 'denoise',
tile_w: int = 0,
tile_h: int = 0,
tile_pad: int = 8,
gpu_id: int | None = None,
fp16: bool = True,
) -> vs.VideoNode:
"""
DPIR: Deep Plug-and-Play Image Restoration
Parameters:
clip: Clip to process. Only RGB and GRAY formats with float sample type of 32 bit depth are supported.
strength: Strength for deblocking/denoising. Defaults to 50.0 for 'deblock', 5.0 for 'denoise'.
Also accepts a GRAY8/GRAYS clip for varying strength.
task: Task to perform. Must be 'deblock' or 'denoise'.
tile_w, tile_h: Tile width and height, respectively. As too large images result in the out of GPU memory issue,
so this tile option will first crop input images into tiles, and then process each of them.
Finally, they will be merged into one image. 0 denotes for do not use tile.
tile_pad: The pad size for each tile, to remove border artifacts.
gpu_id: The GPU ID.
fp16: Enable FP16 mode.
"""

------------------------------


If someone compares this to https://github.com/HolyWu/vs-dpir let us know whether there are some speed differences between the two.
Thanks!

On my Geforce GTX 1070ti on 640x352 content I get:

vs-dpir:
clip = DPIR(clip=clip, strength=5.000, task="denoise", provider=1, device_id=0, dual=True)
Output 429 frames in 96.04 seconds (4.47 fps) (without dual=True, I get 4.29fps)

clip = DPIR(clip=clip, strength=5.000, task="denoise", provider=1, device_id=0)
Output 429 frames in 96.04 seconds (4.47 fps)

vs-dpir-ncnn:
clip = dpir(clip=clip, strength=5.000, task="denoise", gpu_id=0)
Output 429 frames in 117.23 seconds (3.66 fps)

So for me it's better to stick with normal vs-dpir, but I wonder whether this is true for folks with other cards.

Cu Selur

[Selur]

update to vs-dpir-ncnn:
v2.0.0
Repository: HolyWu/vs-dpir-ncnn · Tag: v2.0.0 · Commit: 0938c18 · Released by: HolyWu
Turn dpir function into a class so as to separate model initialization and inference. It's more memory friendly when you run the same task more than once (such as on different clips or with different strengths) since the same model will be initialized only once.

Cu Selur