Mathematically Evaluating mpv's Upscaling Algorithms

[PatchWorKs]

Just discovered this - updated - study of performance and quality by João Vitor Chrisóstomo:
https://artoriuz.github.io/blog/mpv_upscaling.html

[butterw2]

You probably should mention that the test results you posted are for an Anime test image.

While the linked study is Anime focused it also tests a live action photo, with different results. It's worth a look for those who haven't done so already.

[FranceBB]

I'm really surprised to see Lanczos above Spline64 in both tests...

[huhn]

having better psnr doesn't mean the scaler is better.
both are very similar anyway.
lanczos is sharper but rings more.

[Alexkral]

The ArtCNN models have been trained with the Manga109 dataset, this is a huge advantage when testing with anime content, in short, it is what makes the biggest difference, so their good results are not surprising at all. Also, not everyone knows that FSRCNNX has been trained with increased distortion, which makes it able to reduce compression artifacts, but also makes it a bit less sharp with clean content.

I'm not really into this anymore to make a more scientific comparison, I'll leave it to whoever may be interested, but looking at the network architecture, my bet is that the results of these models should be very similar to the AiUpscale LineArt models, which can be used also in mpv.

These models work really well for 2x (or maybe even 3x) upscaling, the real challenge is for 4x upscaling, where GAN models have shown great superiority, but are too resource intensive to be used in real time.

[huhn]

ArtCNN is trained like this:
The shaders are trained on the Manga109 dataset using the Adam optimiser with a learning rate of 1e-4 and the L1/MAE loss function. The high-resolution images are downsampled with a box filter, and they're also split into small 64x64 patches for performance and memory reasons.

and what is used in that test:
magick convert aoko.png -colorspace rgb -filter box -resize 50% -colorspace srgb downsampled.png
shocker. it has to be trained on something so fine.

i'm not going into the sRGB part...

let's take and anime image and look at it.
gpu-next has been avoided because of a current bug that's been around for awhile.

FSRCNNX_x2_16 ArtCNN_C4F32 and NGU sharp very high
i'm not going to tell which is which because they all perform terrible to whatever on this image:
https://slow.pics/c/8jsbuzNR
please guess what is what.
edit: more scaler this time with names
https://slow.pics/c/UoS7hM07

this is one image so no need to jump to a general conclusion here.
ArtCNN is quite impressive compared to NGU sharp and fsrcnnx because they both need to cease and desist here.
the problem on this image is ArtCNN looks the same as super xbr or well it looks the same but rings...
super xbr biggest problem is it is quite unremarkable and well rings.
what so ever ArtCNN not going completely crazy yet is very good achievement

anime 4k was added for comedic effect.