vs-basicvsr

[PatchWorKs]

@Selur how you judge MIA-VSR ?

Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention

Recently, Vision Transformer has achieved great success in recovering missing details in low-resolution sequences, i.e., the video super-resolution (VSR) task. Despite its superiority in VSR accuracy, the heavy computational burden as well as the large memory footprint hinder the deployment of Transformer-based VSR models on constrained devices. In this paper, we address the above issue by proposing a novel feature-level masked processing framework: VSR with Masked Intra and inter-frame Attention (MIA-VSR). The core of MIA-VSR is leveraging featurelevel temporal continuity between adjacent frames to reduce redundant computations and make more rational use of previously enhanced SR features. Concretely, we propose an intra-frame and inter-frame attention block which takes the respective roles of past features and input features into consideration and only exploits previously enhanced features to provide supplementary information. In addition, an adaptive block-wise mask prediction module is developed to skip unimportant computations according to feature similarity between adjacent frames. We conduct detailed ablation studies to validate our contributions and compare the proposed method with recent state-of-the-art VSR approaches. The experimental results demonstrate that MIAVSR improves the memory and computation efffciency over state-of-the-art methods, without trading off PSNR accuracy.

Git: https://github.com/LabShuHangGU/MIA-VSR
Paper: https://arxiv.org/abs/2401.06312

[Selur]

Haven't tried it. Since it's flownet based, maybe styler00dollar or HolyWu will write a Vapoursynth wrapper for it in the future.

[poisondeathray]

Maybe styler00dollar or HolyWu can port evtexture , currently the best 4x video superres by PSNR - REDS4 32.93, On Vid4 29.78

https://github.com/dachunkai/evtexture
https://arxiv.org/abs/2406.13457

I can't get it to run on your own video data set - I can't get the event voxel flow grid generation step modified correctly. The author is supposed to provide an inference script on user's own video data, but has not posted it yet

[ReinerSchweinlin]

This looks promising - at least from the example videos they show...

[PatchWorKs]

Some other upcoming interesting video-enhancing papers in CVPR-2024 and ECCV-2024...

[poisondeathray]

Originally Posted by PatchWorKs

Some other upcoming interesting video-enhancing papers in CVPR-2024 and ECCV-2024...

Thx for the heads up

I got FMA-Net from CVPR-2024 to work . The only pretrained model provided was trained on Reds
https://github.com/KAIST-VICLab/FMA-Net

FWIW here is suzie in FFV1 in RGB (bgr0)
https://www.mediafire.com/file/f47yt..._ffv1.mkv/file

FMA-Net is signifcantly faster than something like basicvsr++ or vrt/rvrt - but more aliasing and temporal flickering (many of the metrics commonly used like PSNR/SSIM don't measure temporal characteristics like temporal consistency artifacts)

One quirk I can't figure out is you lose the first and last frames

I couldn't get the other ones that have code published to work on your own datasets

[PatchWorKs]

More VSR, more fun !

StableVSR - Enhancing Perceptual Quality in Video Super-Resolution through Temporally-Consistent Detail Synthesis using Diffusion Models (ECCV 2024)

[PatchWorKs]

[I've deleted my previous reply with some other links]

OK, since I don't wanna use this 3ad "as a notepad" more, I listed those VSR cited here (and some other) under VIDEO \ AI-based page \ Upscalers: feel free to add or fork it.

Some "real use" (video) shootout would be cool...

[Selur]

If anyone tries Distance Ratio Based Adjuster for Animeinter (DRBA) through cvffi let me know how it compares to newer RIFE models.
Thanks!

Cu Selur

[Z2697]

DRBA is "a control mechanism for Video Frame Interpolation (VFI) networks specifically tailored for anime", it's meant to be used with RIFE or other VFI nets.
Judging by the demo video on github, it's not only tailored for anime, but even a quite specific type of scenes in anime: the background is moving and the forgound character is "semi-static" (she's changing pose but not actually moving, and is the typical shots in anime that has duplicated "frames").

[Selur]

Thanks for clearing that up.