vs-basicvsr

[PatchWorKs]

@Selur how you judge MIA-VSR ?

Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention
Recently, Vision Transformer has achieved great success in recovering missing details in low-resolution sequences, i.e., the video super-resolution (VSR) task. Despite its superiority in VSR accuracy, the heavy computational burden as well as the large memory footprint hinder the deployment of Transformer-based VSR models on constrained devices. In this paper, we address the above issue by proposing a novel feature-level masked processing framework: VSR with Masked Intra and inter-frame Attention (MIA-VSR). The core of MIA-VSR is leveraging featurelevel temporal continuity between adjacent frames to reduce redundant computations and make more rational use of previously enhanced SR features. Concretely, we propose an intra-frame and inter-frame attention block which takes the respective roles of past features and input features into consideration and only exploits previously enhanced features to provide supplementary information. In addition, an adaptive block-wise mask prediction module is developed to skip unimportant computations according to feature similarity between adjacent frames. We conduct detailed ablation studies to validate our contributions and compare the proposed method with recent state-of-the-art VSR approaches. The experimental results demonstrate that MIAVSR improves the memory and computation efffciency over state-of-the-art methods, without trading off PSNR accuracy.

https://raw.githubusercontent.com/LabShuHangGU/MIA-VSR/refs/heads/main/assets/Results.png


Git: https://github.com/LabShuHangGU/MIA-VSR
Paper: https://arxiv.org/abs/2401.06312

[Selur]

Haven't tried it. Since it's flownet based, maybe styler00dollar or HolyWu will write a Vapoursynth wrapper for it in the future.

[poisondeathray]

Maybe styler00dollar or HolyWu can port evtexture , currently the best 4x video superres by PSNR - REDS4 32.93, On Vid4 29.78

https://github.com/dachunkai/evtexture
https://arxiv.org/abs/2406.13457

I can't get it to run on your own video data set - I can't get the event voxel flow grid generation step modified correctly. The author is supposed to provide an inference script on user's own video data, but has not posted it yet

[ReinerSchweinlin]

This looks promising - at least from the example videos they show...

[PatchWorKs]

Some other upcoming interesting video-enhancing papers in CVPR-2024 (https://github.com/liuzhen03/awesome-video-enhancement?tab=readme-ov-file#cvpr-2024) and ECCV-2024 (https://github.com/liuzhen03/awesome-video-enhancement?tab=readme-ov-file#eccv-2024)...

[poisondeathray]

Some other upcoming interesting video-enhancing papers in CVPR-2024 (https://github.com/liuzhen03/awesome-video-enhancement?tab=readme-ov-file#cvpr-2024) and ECCV-2024 (https://github.com/liuzhen03/awesome-video-enhancement?tab=readme-ov-file#eccv-2024)...

Thx for the heads up

I got FMA-Net from CVPR-2024 to work . The only pretrained model provided was trained on Reds
https://github.com/KAIST-VICLab/FMA-Net

FWIW here is suzie in FFV1 in RGB (bgr0)
https://www.mediafire.com/file/f47ytaqkfx9cw6v/suzie_FMA-Net_Reds_ffv1.mkv/file

FMA-Net is signifcantly faster than something like basicvsr++ or vrt/rvrt - but more aliasing and temporal flickering (many of the metrics commonly used like PSNR/SSIM don't measure temporal characteristics like temporal consistency artifacts)

One quirk I can't figure out is you lose the first and last frames

I couldn't get the other ones that have code published to work on your own datasets

[PatchWorKs]

More VSR, more fun !

StableVSR (https://github.com/claudiom4sir/StableVSR#readme) - Enhancing Perceptual Quality in Video Super-Resolution through Temporally-Consistent Detail Synthesis using Diffusion Models (ECCV 2024)

[PatchWorKs]

[I've deleted my previous reply with some other links]

OK, since I don't wanna use this 3ad "as a notepad" more, I listed those VSR cited here (and some other) under VIDEO (https://github.com/FORARTfe/HyMPS#-1) \ AI-based page (https://github.com/FORARTfe/HyMPS/blob/main/Video/AI-based.md#--) \ Upscalers (https://github.com/FORARTfe/HyMPS/blob/main/Video/AI-based.md#upscalers-): feel free to add or fork it.

Some "real use" (video) shootout would be cool... :thanks:

[Selur]

If anyone tries Distance Ratio Based Adjuster for Animeinter (DRBA) (https://github.com/routineLife1/DRBA) through cvffi (https://github.com/TensoRaws/ccvfi?tab=readme-ov-file) let me know how it compares to newer RIFE models.
Thanks!

Cu Selur

[Z2697]

DRBA is "a control mechanism for Video Frame Interpolation (VFI) networks specifically tailored for anime", it's meant to be used with RIFE or other VFI nets.
Judging by the demo video on github, it's not only tailored for anime, but even a quite specific type of scenes in anime: the background is moving and the forgound character is "semi-static" (she's changing pose but not actually moving, and is the typical shots in anime that has duplicated "frames").

[Selur]

Thanks for clearing that up.

[PatchWorKs]

Better late than never: Happy New Year !

Just discovered that Intel has "own" (server-oriented ?) open source Video Super Resolution library:
Intel Library for Video Super Resolution consist of a few different algorithms including machine learning and deep learning implementations to offer a balance between quality and performance.

We have enhanced the public RAISR (Rapid and Accurate Image Super Resolution), an AI based Super Resolution algorithm https://arxiv.org/pdf/1606.01299.pdf, to achieve better visual quality and beyond real-time performance for 2x and 1.5x upscaling on Intel® Xeon® platforms and Intel® GPUs. Enhanced RAISR provides better quality results than standard (bicubic) algorithms and a good performance vs quality trade-off as compared to compute intensive DL-based algorithms.

Enhanced RAISR is provided as an FFmpeg plugin inside of a Docker container(Docker container only for CPU) to help ease testing and deployment burdens. This project is developed using C++ and takes advantage of Intel® Advanced Vector Extension 512 (Intel® AVX-512) on Intel® Xeon® Scalable Processor family and OpenCL support on Intel® GPUs.
https://private-user-images.githubusercontent.com/89970744/363394862-e28b52c2-67c7-44a9-a66f-df8b355735f9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDAwNDc5OTIsIm5iZiI6MTc0MDA0NzY5MiwicGF0aCI6Ii84OTk3MDc0NC8zNjMzOTQ4NjItZTI4YjUyYzItNjdjNy00NGE5LWE2NmYtZGY4YjM1NTczNWY5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMjAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjIwVDEwMzQ1MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPThiYTgxYWUwNjJhNTU3MTRkZDYyNTk3MmRjZWZjYmY2NDkyMzU4MThiMjUzOWU4MTFhNDk5MTZhMWU5ODExYTQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.-jWF59M1V5ygFWHag8XX-lT2PDj5swqWiCe6Quwq8DA

Check out this interesting doc about performance/usage (https://github.com/OpenVisualCloud/Video-Super-Resolution-Library/blob/main/docs/performance.md) too.

Git: https://github.com/OpenVisualCloud/Video-Super-Resolution-Library#readme

[PatchWorKs]

Hi everyone, just discovered (and tested on a single image) S3Diff that looks VERY promising:
Diffusion-based image super-resolution (SR) methods have achieved remarkable success by leveraging large pre-trained text-to-image diffusion models as priors. However, these methods still face two challenges: the requirement for dozens of sampling steps to achieve satisfactory results, which limits efficiency in real scenarios, and the neglect of degradation models, which are critical auxiliary information in solving the SR problem. In this work, we introduced a novel one-step SR model, which significantly addresses the efficiency issue of diffusion-based SR methods. Unlike existing fine-tuning strategies, we designed a degradation-guided Low-Rank Adaptation (LoRA) module specifically for SR, which corrects the model parameters based on the pre-estimated degradation information from low-resolution images. This module not only facilitates a powerful data-dependent or degradation-dependent SR model but also preserves the generative prior of the pre-trained diffusion model as much as possible. Furthermore, we tailor a novel training pipeline by introducing an online negative sample generation strategy. Combined with the classifier-free guidance strategy during inference, it largely improves the perceptual quality of the super-resolution results. Extensive experiments have demonstrated the superior efficiency and effectiveness of the proposed model compared to recent state-of-the-art methods.

Demo results: https://github.com/ArcticHare105/S3Diff#visual_comparison

Git: https://github.com/ArcticHare105/S3Diff#readme

[Selur]

Since it's aiming at images not video, maybe we will end up with a onnx mask. :)

[PatchWorKs]

Bump.

SeedVR2 (https://iceclear.github.io/projects/seedvr2/) anyone ?

https://www.ainvfx.com/blog/one-step-4k-video-upscaling-and-beyond-for-free-in-comfyui-with-seedvr2/

https://youtu.be/I0sl45GMqNg

[takla]

Demo results: https://github.com/ArcticHare105/S3Diff#visual_comparison
https://imgsli.com/MzAzNjQ1


Such massive bullshit. There is NO WAY you'd ever get that level of detail restoration on any image that wasn't part of the training set.

[ReinerSchweinlin]

Such massive bullshit. There is NO WAY you'd ever get that level of detail restoration on any image that wasn't part of the training set.

Correct.
to be fair - thats the case for EVERY Demo of any algo... No one publishes suboptimal results with footage that does not werk well on the specific trained model.

[Selur]

VS-DistilDRBA (https://github.com/routineLife1/VS-DistilDRBA) might be interesting for some. :)

[Selur]

Such massive bullshit. There is NO WAY you'd ever get that level of detail restoration on any image that wasn't part of the training set.
did a few short tests with a small 3b model:
test1: https://www.mediafire.com/folder/d2sysyer1xe0l/seedvr2_test1
test2: https://www.mediafire.com/folder/wcaihu4kacasl/seedvr2_test2
test3: https://www.mediafire.com/folder/4wq2qdngb25en/seedvr2_test3
each test contains the source I used and the output I got.

=> Nice, but with my current gpu (Geforce RTX 4080 16GB VRAM) this is way too slow to be usable
(second clip took 30min for 520 frames to process, so roughly 17 frames per minute, so 0.28fps for sd to hd)

Cu Selur

[ReinerSchweinlin]

did a few short tests with a small 3b model:
test1: https://www.mediafire.com/folder/d2sysyer1xe0l/seedvr2_test1
test2: https://www.mediafire.com/folder/wcaihu4kacasl/seedvr2_test2
test3: https://www.mediafire.com/folder/4wq2qdngb25en/seedvr2_test3
each test contains the source I used and the output I got.

=> Nice, but with my current gpu (Geforce RTX 4080 16GB VRAM) this is way too slow to be usable
(second clip took 30min for 520 frames to process, so roughly 17 frames per minute, so 0.28fps for sd to hd)

Cu Selur

Nice, thank you :)

Diffusion based Video Enhancers are nice :) The speed seems roughly comparable to what the commercial Topaz Video does in a similar test scenario...

Getting my hopes up... You did not - by any chance - fiddle SeedVR2 into hybrid ?

:)

CU

[Selur]

Getting my hopes up... You did not - by any chance - fiddle SeedVR2 into hybrid ?
No, don't really see a meaningful way to do it atm.
I used it through ComfyUI following https://www.youtube.com/watch?v=MBtWYXq_r60
Just wanted to show that it really produces nice results if you feed it a clean source. :)
Haven't played around with it much simply due to the resource hunger of the whole thing. :)

Cu Selur

[ReinerSchweinlin]

No, don't really see a meaningful way to do it atm.
I used it through ComfyUI following https://www.youtube.com/watch?v=MBtWYXq_r60
Just wanted to show that it really produces nice results if you feed it a clean source. :)
Haven't played around with it much simply due to the resource hunger of the whole thing. :)


:D
I did not really expect it :)

Yes, its very hungry... I tried to use multiple GPUs to speed it up - no luck at the moment...

The diffusion ability indeed is able to introduce new details, very fascinating to see.

[Selur]

The diffusion ability indeed is able to introduce new details, very fascinating to see.
Yup, it's impressive (https://www.mediafire.com/folder/pqr3kma7keqpg/seedvr2_test4) and I'm hoping someone will come up with a way to leverage this in Vapoursynth. :)

[Selur]

Just saw RVRT (Recurrent Video Restoration Transformer) (https://github.com/Lyra-Vhess/vs-rvrt), which atm. doesn't work with Python 3.12+, but I thought it might be interesting for someone here. :)
vs-rvrt is based on https://github.com/JingyunLiang/RVRT

Update 1:
might work with Python 3.12+ now, but fails on portable environment since it can't find cuda,...

Cu Selur

[Selur]

vs-vsrvrt is working now. :) (will probably add it to Hybrids dev next weekend)

[ReinerSchweinlin]

VERY COOL, thank you !

do you have a rough speed estimate from your tests? are 10 x 5090 enough for 1 hour / frame ?

[Selur]

Just tested with
clip = vsrvrt.SuperRes(clip, scale=4, model="reds",preview_mode=True)
and get:
F:\Hybrid\64bit\Vapoursynth>VSPipe.exe c:\Users\Selur\Desktop\test3.vpy -c y4m NUL --progress
RVRT: Using preview mode (lazy chunk processing)
Warning: F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\native\TensorShape.cpp:4316.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

Loading model from: C:\Users\Selur\AppData\Local\vsrvrt\models\001_RVRT_videosr_bi_REDS_30frames.pth
RVRT: Preview mode - 429 frames in 9 chunk(s)
RVRT: Chunk size=64, processing on-demand
Script evaluation done in 2.14 seconds
RVRT: Processing chunk 1/9 (frames 0-63) for preview
RVRT: Auto-tiling: (11, 256, 256)
RVRT: Available VRAM: 14.5 GB, Estimated: 14.5 GB
RVRT: Processing chunk 2/9 (frames 48-111) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 13.2 GB
RVRT: Processing chunk 3/9 (frames 96-159) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 4/9 (frames 144-207) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 5/9 (frames 192-255) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 6/9 (frames 240-303) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 7/9 (frames 288-351) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 8/9 (frames 336-399) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 9/9 (frames 384-428) for preview
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
Output 429 frames in 1034.04 seconds (0.41 fps)

using:
clip = vsrvrt.SuperRes(clip, scale=4, model="reds",preview_mode=False)
I get:
F:\Hybrid\64bit\Vapoursynth>VSPipe.exe c:\Users\Selur\Desktop\test3.vpy -c y4m NUL --progress
RVRT: Using chunked processing (video too long for single-pass)
Warning: F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\native\TensorShape.cpp:4316.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

Loading model from: C:\Users\Selur\AppData\Local\vsrvrt\models\001_RVRT_videosr_bi_REDS_30frames.pth
RVRT: Processing 429 frames in 9 chunk(s)
RVRT: Chunk size=64, overlap=16
RVRT: Processing chunk 1/9 (frames 0-63)
RVRT: Auto-tiling: (11, 256, 256)
RVRT: Available VRAM: 14.5 GB, Estimated: 14.5 GB
RVRT: Processing chunk 2/9 (frames 48-111)
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 13.2 GB
RVRT: Processing chunk 3/9 (frames 96-159)
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 4/9 (frames 144-207)
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 5/9 (frames 192-255)
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 6/9 (frames 240-303)
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 7/9 (frames 288-351)
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 8/9 (frames 336-399)
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Processing chunk 9/9 (frames 384-428)
RVRT: Padded chunk from 45 to 64 frames
RVRT: Auto-tiling: (10, 256, 256)
RVRT: Available VRAM: 14.3 GB, Estimated: 13.2 GB
RVRT: Final output shape: torch.Size([1, 429, 3, 1408, 2560])
Script evaluation done in 1050.36 seconds
Output 429 frames in 6.47 seconds (66.28 fps)

so 0.4fps seems to be the correct speed.

Also tested:
clip = vsrvrt.Denoise(clip)
and got:
F:\Hybrid\64bit\Vapoursynth>VSPipe.exe c:\Users\Selur\Desktop\test3.vpy -c y4m NUL --progress
Information: VideoSource track #0 index progress 7%
RVRT: Using chunked processing (video too long for single-pass)
Warning: F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\native\TensorShape.cpp:4316.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

Loading model from: C:\Users\Selur\AppData\Local\vsrvrt\models\006_RVRT_videodenoising_DAVIS_16frames.pth
RVRT: Processing 429 frames in 9 chunk(s)
RVRT: Chunk size=64, overlap=16
RVRT: Processing chunk 1/9 (frames 0-63)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.5 GB, Estimated: 8.7 GB
RVRT: Processing chunk 2/9 (frames 48-111)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 3/9 (frames 96-159)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 4/9 (frames 144-207)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 5/9 (frames 192-255)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 6/9 (frames 240-303)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 7/9 (frames 288-351)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 8/9 (frames 336-399)
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 9/9 (frames 384-428)
RVRT: Padded chunk from 45 to 64 frames
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Final output shape: torch.Size([1, 429, 3, 352, 640])
Script evaluation done in 63.90 seconds
Output 429 frames in 0.21 seconds (2090.79 fps)

clip = vsrvrt.Denoise(clip, preview_mode=True)
and got:
F:\Hybrid\64bit\Vapoursynth>VSPipe.exe c:\Users\Selur\Desktop\test3.vpy -c y4m NUL --progress
RVRT: Using preview mode (lazy chunk processing)
Warning: F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\native\TensorShape.cpp:4316.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

Loading model from: C:\Users\Selur\AppData\Local\vsrvrt\models\006_RVRT_videodenoising_DAVIS_16frames.pth
RVRT: Preview mode - 429 frames in 9 chunk(s)
RVRT: Chunk size=64, processing on-demand
Script evaluation done in 2.03 seconds
RVRT: Processing chunk 1/9 (frames 0-63) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.5 GB, Estimated: 8.7 GB
RVRT: Processing chunk 2/9 (frames 48-111) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 3/9 (frames 96-159) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 4/9 (frames 144-207) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 5/9 (frames 192-255) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 6/9 (frames 240-303) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 7/9 (frames 288-351) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 8/9 (frames 336-399) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 9/9 (frames 384-428) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
Output 429 frames in 61.37 seconds (6.99 fps)

for:
clip = vsrvrt.Deblur(clip, preview_mode=True)
I got:
F:\Hybrid\64bit\Vapoursynth>VSPipe.exe c:\Users\Selur\Desktop\test3.vpy -c y4m NUL --progress
RVRT: Using preview mode (lazy chunk processing)
Warning: F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\aten\src\ATen\native\TensorShape.cpp:4316.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

Loading model from: C:\Users\Selur\AppData\Local\vsrvrt\models\005_RVRT_videodeblurring_GoPro_16frames.pth
RVRT: Preview mode - 429 frames in 9 chunk(s)
RVRT: Chunk size=64, processing on-demand
Script evaluation done in 2.14 seconds
RVRT: Processing chunk 1/9 (frames 0-63) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.5 GB, Estimated: 8.7 GB
RVRT: Processing chunk 2/9 (frames 48-111) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.5 GB, Estimated: 8.7 GB
RVRT: Processing chunk 3/9 (frames 96-159) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.5 GB, Estimated: 8.7 GB
RVRT: Processing chunk 4/9 (frames 144-207) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 5/9 (frames 192-255) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 6/9 (frames 240-303) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 7/9 (frames 288-351) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 8/9 (frames 336-399) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
RVRT: Processing chunk 9/9 (frames 384-428) for preview
RVRT: Auto-tiling: (64, 256, 256)
RVRT: Available VRAM: 14.4 GB, Estimated: 8.7 GB
Output 429 frames in 60.19 seconds (7.13 fps)

Cu Selur

[ReinerSchweinlin]

thanx. 16gb vram? which gpu did you use?

[Selur]

my Geforce RTX 4080

[Selur]

Uploading a new dev&torch atm., which will be up in 1 1/2 hours which has basic support for RVRT.

Cu Selur

[ReinerSchweinlin]

Uploading a new dev&torch atm., which will be up in 1 1/2 hours which has basic support for RVRT.

Cu Selur
nice :)
thanx !

[Selur]

side note: you might want to enable 'Lazy Chunking'/'preview_mode' and lower the chunking values, if you use the filter in a preview. :)
Without 'Lazy Chunking'/'preview_mode' all frames get processed before outputting a frame,..

[ReinerSchweinlin]

side note: you might want to enable 'Lazy Chunking'/'preview_mode' and lower the chunking values, if you use the filter in a preview. :)
Without 'Lazy Chunking'/'preview_mode' all frames get processed before outputting a frame,..
good to know :) Thanx !

[ReinerSchweinlin]

tried it on my 4060TI 16GB ...

The deblur model works fine, but with the Super resoultion I get no visible difference (enabled DIFF in preview). Running a 1x "clean up pass"

Its a MPEG2 File as MKV, just deinterlaced prior to running RVRT.

I remember in the past that some of the resizers needed actual resize to be enabled in the resize tab - tried that, made no difference.

ah, another thing I noticed:

the tiled processing (deblur in this case) seems to produce a fine line sometimes visible

[Selur]

Resizing does give quite different results for me, depending on the input. Like results are not as aggresive as for example RealESRGAN, but they are there. To to me that seems to work fine.
Since the changes are more subtile, you might loose them depending on the additional resizing that is done to archive your target resolution.
About tiling: I'll probably also add the tile_overlap control to the gui, which should allow to lower / circumvent this effect.

Cu Selur

[Selur]

Added tile_overlap controls to Hybrid, which should help with those lines.

[ReinerSchweinlin]

Resizing does give quite different results for me, depending on the input.

Thanx for reminding - I do "resizing to the optical resolution" quite oten, in my tests here, I just threw the original DVD into hybrid and was wondering why "nothing happened"... After lowering the resolution of the input file, I get usefull resoults :) Thanx for the nice addition :)

Normaly, I prepare files externaly (use some VD stuff I am used to for decades now) and also do the resizing - then throw this prepared file into whatever I want to use next for Upscaling / AI processing..

Is there a way to pre-downscale inside hybrid? Id would be nice if all the filtering, deinterlacing etc.. could be done in hybrid, then lower the resolution and then let the Scalers do their thing (I bet there is one, I am just not bright enough to see it :) )

[Selur]

Is there a way to pre-downscale inside hybrid? Id would be nice if all the filtering, deinterlacing etc.. could be done in hybrid, then lower the resolution and then let the Scalers do their thing (I bet there is one, I am just not bright enough to see it )
Yes.
For deinterlacing and resizing there are special controls under "Filtering->Vapoursynth->Misc->Script->Lower res. before resize / Lower res. before deinterlace", for general filtering you can enable "Filtering->Vapoursynth->Misc->UI->Show 'Gimmick'-controls" which aside from other options, also adds an option to resize before the filter (and undo the resizing if wanted)
https://i.ibb.co/gFT5qm41/grafik.png (https://ibb.co/0VMg4msN)

Cu Selur

Ps.: "Filtering->Vapoursynth->Misc->UI" allows to add quite a few additional controls.

[ReinerSchweinlin]

Great, missed that one so far, will check it out, very cool !!