VS Port of TOFLOW possible?

[ReinerSchweinlin]

Hello,

I am twisting my head around AI Upscaling lately, trying to get a grasp of what is developed at the moment.. Some Companies pick up the topic and release Software (Topaz AI Video Enhancer for example) and some stuff already has found it´s way into open source / free tools (Waifu2x in Selur's hybrid as vapoursynth variants using Vulkan or Cuda, Video2x has waifu2x and lately incorporated realsr)... A lot going on... Playing around with ESRGAN Resizers, Topaz Beta Software, etc. is fun and some results are really astonishing compared to traditional filters we all know.

Would someone have the skill and have fun building a vapoursynth variant of one of the more mature models out there developed for video? I know of an VS - ESRGAN Port, but it is only "single image" for now...

As far as I know, Task Oriented Flow seems to be a candidate which is especially for video and is able to recover details, denoise and upscale - So my thinking is that it could be a good candidate..

https://github.com/anchen1011/toflow
pytorch version:
https://github.com/Coldog2333/pytoflow

I am aware that some of the dependencies probably aren´t "portable", so some stuff like CUDA or Python might have to be installed seperately, but that's just my uneducated guess... And wouldn´t hurt..

Actually, I really don`t care if TOFLOW would be ported or something similar or even better... I am just dreaming of a vapoursynth variant of some AI-Upscale/Denoise/whatsoever :) But I am lacking the skills to do so - maybe someone with more knowlegde than me wants to give it a shot?

[ChaosKing]

Have you seen this? https://github.com/WolframRhodium/Super-Resolution-Zoo (mostly upscaler denoiser models)
I didn't like any of the denoiser models bcs you can't really tweak them with paramteres. So they are often too strong or too weak or just look bad. I think the ByNet upscaler looked good and had some decent fps.

Maybe it is possible to converted these models to "SuperResZoo models"

The x2 toflow fps looked not bad. I wonder how fast it is.

[ReinerSchweinlin]

I stumpled across the Wholfram Rhodium one - Since it said "models", I always thought these simply where the trained models for each framework and let it be - looking at it again it seems the frameworks are also included (ported to mxnet).. Am I getting it right?

Up till now, I only managed to install some of the windows GUIs for ESRGAN or waifu2x like:
http://forums.qhimm.com/index.php?topic=18798.50
https://github.com/ptrsuder/IEU.Winforms

and have been browsing through various sites like
https://awesomeopensource.com/projects/super-resolution

to get a grasp of whats out there and compare stuff (only by looking at the demos mostly - ESRGAN with different models and waifu2x I managed to get running and have been playing around with them on my RTX2060 in windows using the above mentioned GUIs. So far, my skills are not refined enough to compile stuff myself, I am only getting up to the point where someone else has done work and managed to put it into a GUI..)

Since the Game-upscale-community is very active, I found a guide for training own models for Esrgan (https://www.reddit.com/r/GameUpscale/comments/dz8j8n/mmsr_esrgan_training_and_setup_guide/) which I thought to be a good idea (making a training set out of bluray frames and record them onto VHS - then train ESRGAN on it) - but my last attempt to Linux has been a few years ago and my skills are very basic. If I have time, I manage to find out stuff, but I am no programmer or "daily Linux user"... fiddling a Linux on yn ancient G3 Power Mac I manage, installing different distros on a PC and configuring some basic stuff - ok... But that´s about it for the moment.

Shortly after taking many close looks at ESRGAN, I learned that it is only single frame capable and that there are some newer techniques (like TOFLOW) out there which are able to recover some details out of the motion of a video, taking several frames into account.

one thing I encountered which seems to use vapoursynth is:
https://github.com/AlphaAtlas/VapourSynth-Super-Resolution-Helper
But somehow its not running correctly on my system. I must say that I grew up with Avisynth and still have to get my head wrapped around Vapoursynth more.

So I'm afraid I can´t contribute much on the actual programming here, I am simply pointing out a suggestion and giving my amateur thoughts on the matter.

I´d love to learn to write stuff like this - but this takes a lot of time and sadly, a noob google "write your own vapoursynth filter" brings up stuff over my head :)

[ChaosKing]

New upscaler / denoiser https://github.com/Kiyamou/VapourSynth-SRMD-ncnn-Vulkan

vsrepo install srmd

[ReinerSchweinlin]

hm, nice :) Thanks :)

[Selur]

Is there a portable version of this available?
-> scratch that missed the release on github