Porting Image Upscaling Algorithms

[MysteryX]

Some time ago I saw someone post a link to an upscaling competition where various people implemented various algorithms for upscaling, and some of them produced really great results.

Anyone has the link of that?

Is anyone interested in implementing those algorithms for Avisynth/VapourSynth?

[WolframRhodium]

upscaling competition

NTIRE2017 (http://www.vision.ee.ethz.ch/ntire17/)
NTIRE2018 (http://www.vision.ee.ethz.ch/ntire18/)
PIRM2018 (https://www.pirm2018.org/)

The competitions above mainly focus on single image super-resolution, and there are also some video super-resolution algorithms but I've never seen related competitions.

Collections of SISR algos: https://github.com/YapengTian/Single-Image-Super-Resolution

[MysteryX]

Would these winners apply well for video, is the code available, and would it be easy to port? (or perhaps they are too slow because they are designed for single frames?)

[WolframRhodium]

The algorithm behind the well-known waifu2x is simply an early approach to upscale image using convolutional neural network.

To the best of my knowledge, various CNN-based SISR algorithms not limited to waifu2x have been utilized into DVD or even BD upscaling without too many serious problems. They should work at least not bad if they have been properly trained. The speed mainly depends on the GPU(s), for medium algorithms the speed is acceptable.

The codes and trained models are usually publicly available, but the codes are often written in Python or MATLAB, which should be ported to C/C++ for use in avs/vs. MXNet with C++ interface is a good candidate for such algorithms under Windows IMHO.

[MysteryX]

How does the training work? Is it a generic training done once by the dev, or could it go through a specific video, upscale, compared with original to analyze differences, and then fine-tune for that specific video?

[StainlessS]

I think it can go through millions of iterations, if code changed then, do it all again.
Perhaps wrong, not sure.

[wonkey_monkey]

or could it go through a specific video, upscale, compared with original to analyze differences, and then fine-tune for that specific video?

That would seem to be rather redundant if you already have the original. Maybe you can finetune for a particular type of video, e.g. beach shots, city shots, nature shots, and so on.

[MysteryX]

Still wondering how the training works.

[wonkey_monkey]

1. Feed degraded image(s) into neural net.
2. Change the neural net a little bit.
3. Feed degraded image(s) into neural net. Compare this result, and the previous result, and if the previous result was better, roll back the changes you just made.
4. Goto step 2.

That's my super-simplified understanding of it anyway. It's done by the dev, then once he's happy with it the packaged neural net is hardcoded into the program. I seem to recall someone saying NNEDI3 was trained for days, so it's not practical to do it for every video (and you need an undegraded original to do the training anyway, which you wouldn't have if you're looking to use this process for real).

[kgrabs]

Some of the image doublers for mpv (https://github.com/bjin/mpv-prescalers/tree/master) might be worth porting. The Ravu ones look like a great, much faster alternative to nnedi3. The link has a benchmark (https://github.com/bjin/mpv-prescalers/wiki/Performance) and I put the image tests into a diff.pics comparison (https://diff.pics/1HY4MMiTc5st/1)

[MysteryX]

I'm still waiting for Madshi to port his NGU over to Avisynth (but he has said that the free version would support YV16 but not YV24).

There are lots of more advanced algorithms now available, it would be good to have access to some of it.

[`Orum]

I totally concur that more resizers are needed. Porting them to AviSynth, assuming the license permits, usually isn't too hard, depending on how platform-specific you're willing to be and what hardware requirements you're willing to stomach. * promptly runs and hides under a rock *

That said, there's a wealth of other filters aside from just resizers that, to my knowledge, aren't available yet in AviSynth, and they're not even that new. For example, I think it's still not possible to use the various PPAA filters that many games feature, e.g. FXAA, SMAA, and CMAA, as no one has taken the time to port them yet. FXAA is 9 years old already, and SMAA not far behind. But, I suppose instead of complaining I should just make them happen...

Edit: Perhaps one day we can have a team dedicated to porting? :D

[Stereodude]

I'm still waiting for Madshi to port his NGU over to Avisynth (but he has said that the free version would support YV16 but not YV24).
Has he committed to do that? I would love to be able to use NGU as a resizer in AVIsynth+.