Waifu2x comparison

[ryrynz]

Waifu2x (http://waifu2x.udp.jp/) is an Image Super-Resolution for anime/fan-art using Deep Convolutional Neural Networks. It's produces images similar to that of NNEDI3.

It looks quite promising, please post your test images, image evaluations and discussions here.

[huhn]

not sure if software player is the best place.
it sounds like real time usage is not an option.

[The 8472]

not sure if software player is the best place.
it sounds like real time usage is not an option.

It might be interesting for offline upscaling applications if someone has massive amounts of GPU time to burn. Although mostly as some sort of reference point, given <1fps performance.

Additionally I think it could be possible to tweak madvr to achieve better results for animation content. Having a goal/reference to strive for may help there. See what it does better/where NNEDI3 performs worse and what can be tweaked

My original post, will add some more samples:

Source image (http://i.imgur.com/uWMu9zt.png)
new upscaler w/ noise reduction (http://i.imgur.com/WnHLPE4.png)
new upscaler w/o noise reduction (http://i.imgur.com/QG8aVtr.png)
NNEDI3 32 w/ SuperRes (http://a.pomf.se/ywwfia.png)
NNEDI3 64 w/o SuperRes (http://a.pomf.se/smpgig.png)


Another thing of interest, that algorithm seems to focus on luma only and use some regular upscaling filter for chroma. We should consider that in comparisons.


As far as I can tell they're not using better "guesswork" they're just using far far more neurons, they also look at a larger region of the image to calculate a single pixel (hence why it's better at removing aliasing). I estimate it's at least 20x slower than NNEDI3 with 256 neurons, but even that is a very generous estimate, it could easily be 100x.

My question is still what data is NNEDI3 trained on? I very very vaguely recall that the coefficients have been obtained by a mixed set of real life and animation content.
Would it be possible to retrain two separate NNEDI networks specialized for different content types?

And as you can see in the samples above NNEDI+SuperRes exacerbates the shortcomings I see in nnedi (the amplification of stair effects). Maybe better parameters can be found to reduce that effect? I've tried playing with anti-aliasing but it didn't seem to do all that much.

What is the reasonable domain for those parameters anyway? 0.0-1.0?

[Shiandow]

My question is still what data is NNEDI3 trained on? I very very vaguely recall that the coefficients have been obtained by a mixed set of real life and animation content.
Would it be possible to retrain two separate NNEDI networks specialized for different content types?


That would be possible, of course whether it is easy is a whole different issue. I did once wonder what would happen if you made a neural network retrain online, that is, train it on the same material it is currently displaying. Doing so for NNEDI3 would require quite a bit of work, so I'm not sure if it's worth it. Might be better to build a neural network from scratch.


And as you can see in the samples above NNEDI+SuperRes exacerbates the shortcomings I see in nnedi (the amplification of stair effects). Maybe better parameters can be found to reduce that effect? I've tried playing with anti-aliasing but it didn't seem to do all that much.

What is the reasonable domain for those parameters anyway? 0.0-1.0?

Usually somewhere between 0.0 to 1.0 is reasonable. You may also need to set "error upscaling quality" to "high" to get a better result.

[Garteal]

I was expecting something else when I read the title. :p

That filter is really nice though. I'm doing something similar using a script chain in Avisynth to upscale some art to a higher resolution trying to keep it good looking.
Just did a quick test here and my parameters aren't 1:1, but it should be good enough.
Source (http://abload.de/img/2625857f7bc8ba0524eb9frq5u.jpg)
Waifu2x (http://abload.de/img/kurisu-waifu04on4.png)
AviSynth scripts (http://abload.de/img/kurisu-avisynth8hpn1.png)

There are some cases (and places) where Waifu2x does a bit better such as her tie, but atm I still prefer what my script chain outputs, but that could change, and I'm hoping for it.

[The 8472]

First attempt to compare various upscale combinations: archive (http://a.pomf.se/hrphqw.7z)

At both 2x and 4x they seem to be trading fairly even blows, which is kind of impressive given the difference in neurons.

I tried tweaking the superres parameters at 4x and they do make some difference, but there are too many knobs to turn to find one that's subjectively better.

The problem both seem to be having is that they're mostly amplifying artifacts with superres on or noise reduction off respectively.

waifu2x seems to perform fairly well on perfectly artifact-free PNG images, so I guess I'll try a BDMV next. Or can anyone suggest lossless reference material which I could use?

[ryrynz]

not sure if software player is the best place.


Wondered that myself but seems there's a lot of other discussions about sharpeners and such going on here so..

I was expecting something else when I read the title. :p


You still used it to post your waifu though.. :)


That filter is really nice though. I'm doing something similar using a script chain in Avisynth to upscale some art to a higher resolution trying to keep it good looking.


Ahh warpsharp.. yeah I might run that through my script and see what you think. BTW tone down or change your denoising, you're losing a fair amount of detail there (legs being the obvious spot)

[mandarinka]

Random try I gave it: http://screenshotcomparison.com/comparison/127997
It cleans up artifacts and sharpens, but the effect is unnatural, with wobbly and weird looking lines. Basically what you get on many upscaled blurays that don't look very good. On cleaner source than this it would be quite horrible :/
Neutral scalers are IMHO better than these messful filters.

Edit:
Some more experiments on women from anime (given its name...):
http://screenshotcomparison.com/comparison/127998
http://screenshotcomparison.com/comparison/128000

These four pics are without the denoising option.

Basically I would say that it sharpens but doesn't really improve quality. Au contraire, the lineart that was balanced, contiguous and neutral before is getting garbled/artifacted...

[heiseikiseki]

Someone Rewrite the waifu2x in c++ & opencv
https://github.com/WL-Amigo/waifu2x-converter-cpp/releases

And here is another person write a GUI tool

http://inatsuka.com/extra/koroshell/


So now we can use it in our pc.

But it's very slow and memory usage is really high.

Still waiting someone modify it to use GPU. I think it's possible to use in realtime.

[madshi]

I was expecting something else when I read the title. :p

That filter is really nice though. I'm doing something similar using a script chain in Avisynth to upscale some art to a higher resolution trying to keep it good looking.
Just did a quick test here and my parameters aren't 1:1, but it should be good enough.
Source (http://abload.de/img/2625857f7bc8ba0524eb9frq5u.jpg)
Waifu2x (http://abload.de/img/kurisu-waifu04on4.png)
AviSynth scripts (http://abload.de/img/kurisu-avisynth8hpn1.png)

There are some cases (and places) where Waifu2x does a bit better such as her tie, but atm I still prefer what my script chain outputs, but that could change, and I'm hoping for it.
May I ask which Avisynth script you've used for that image?

[heiseikiseki]

Another person Re-write Waifu2x converter in caffe which can use GPU。
It's really fast compare to waifu2x converter.

https://github.com/lltcggie/waifu2x-caffe