new super resolution toy for yall to mess with

[feisty2]

if you optimize your neural net for bilinear, which is rarely used anyways, it will over-sharpen images resampled by any other algorithm, and sharpening is shit
sinc is an ideal low pass filter, and it makes perfect sense to use sinc for a neural net designed to reconstruct high frequency components, a truncated sinc makes ringings but the neural net will take care of that

[Groucho2004]

subjective quality measurement
What's the measurement unit for subjective quality?

[feisty2]

What's the measurement unit for subjective quality?

https://en.wikipedia.org/wiki/Mean_opinion_score
I assume that's an appropriate formal measurement

but I do believe that I have well trained eyes for image quality, so I can judge the quality myself if I'm not planning to list subjective quality scores in the paper

[poisondeathray]

No, everyone has to struggle to see the differences. Some, like myself, Manolito, and Riffraff, just don't squint long enough to develop strong opinions about photos of Joe Biden eating sandwiches.

Not "everyone ." Or do you mean the entire "every" 3 of you ? :D

For the bridge/stream - Compare the nnedi3 result with the last result. The difference is astonomically massive from an image processing perspective . (And I'm not just talking about the black borders :devil: :D ) . Even "Joe Public" / layman should be able to see differences as-is without even flipping back and forth

[Groucho2004]

For the bridge/stream - Compare the nnedi3 result with the last result. The difference is astonomically massive from an image processing perspective .Indeed, even with my currently fucked up eye sight (my specs urgently need an update) it's quite obvious.

[feisty2]

Some mod plz delete Katie's trolling comments?

[tebasuna51]

Ok, only let their initial opinion, after that the other comments are useless.

[madshi]

Hmmmm... Interesting stuff!

Would be nice if you could add the low-res images to your image comparisons, and describe how you created them from the groundtruth. That's important to know because nnedi3, waifu2x etc behave quite differently depending on how the image was downscaled. E.g. nnedi3 works best for nearest neighbor downscaled images, while waifu2x works best for Catmull-Rom downscales.

Generally, "copying" the low frequency stuff and then just adding high frequency detail sounds cool, but what does it mean practically? I suppose a bloated fat line becomes thinner? But waifu2x already does that. What about texture detail? Is your neural network going to try inventing texture detail that isn't there in the low res image?

IMHO a lot of the cartoonized look comes from reproducing sharp edges with total lack of texture detail. And there isn't so much we can do about that if the low-res image doesn't contain any texture detail to start with. So a cartoon image is sometimes the best we can do. Unless we try to hallucinate texture detail. I've seen some scientific PDFs that try to use neural networks to hallucinate texture detail, but I don't like the results. Sometimes it looks surprisingly great, only to create truly scary horror images in other situations.

The real prove of your approach of course is to test the network on images you *didn't* train with. Testing a neural network on the training images is kinda cheating... :p Yeah, yeah, I know, it was only meant as a test. Still, are any of the image upscaling results in this thread from a neural network of yours that wasn't trained on that particular image?

[feisty2]

Generally, "copying" the low frequency stuff and then just adding high frequency detail sounds cool, but what does it mean practically? I suppose a bloated fat line becomes thinner? But waifu2x already does that. What about texture detail? Is your neural network going to try inventing texture detail that isn't there in the low res image?


it's kinda hard to explain, I got the idea from ResNet and Plum
speaking from the data science aspect, assuming low frequencies were not affected by downscaling, it should be an irrelevant component to what we really care about here, so removing it for the net gives a much sparser representation of things we do care about, a sparser representation generally helps the net "learn" things better, think of it as some kind of "data whitening"
another way of seeing this is, it's sort of like we're doing hacks to the loss function, the loss function is simply an MSE function so it don't know shit about the frequency properties of the image, and removing low frequencies is like doing MSE in a transformed domain that makes the net frequency-aware, it optimizes a transformed frequency loss instead of a pixel-wise loss, about adding imaginary textures....uhmm, not sure about that, but it will probably work sometimes, considering the weight binary for waifu2x is ~5MB and it's ~500MB for mine, so it apparently "memorizes" a lot more stuff than waifu2x

[feisty2]

IMHO a lot of the cartoonized look comes from reproducing sharp edges with total lack of texture detail. And there isn't so much we can do about that if the low-res image doesn't contain any texture detail to start with. So a cartoon image is sometimes the best we can do. Unless we try to hallucinate texture detail. I've seen some scientific PDFs that try to use neural networks to hallucinate texture detail, but I don't like the results. Sometimes it looks surprisingly great, only to create truly scary horror images in other situations.


not sure why exactly, but being frequency-aware does help with reducing cartoonization more or less, test it for urself, it's an easy thing to do, 2 nets with the same architecture, one takes the regular input and one takes the frequency-aware input, getting trained and then,,, just check their outputs and the difference should be obvious

[pandy]

Great work feisty! - personally like Architecture 3 results - they may be for some people a bit too oversharpened but i like them.
:thanks:

[feisty2]

The real prove of your approach of course is to test the network on images you *didn't* train with. Testing a neural network on the training images is kinda cheating... :p Yeah, yeah, I know, it was only meant as a test. Still, are any of the image upscaling results in this thread from a neural network of yours that wasn't trained on that particular image?

wip wip wip
a mirror symmetrical 2x VGG-19 combo is like...

[madshi]

not sure why exactly, but being frequency-aware does help with reducing cartoonization more or less, test it for urself, it's an easy thing to do, 2 nets with the same architecture, one takes the regular input and one takes the frequency-aware input, getting trained and then,,, just check their outputs and the difference should be obvious
Fair enough, it's worth a try. So if I understand correctly, you're doing the following to make the input "frequency aware"?

> the initial guess of the high frequencies is done by further
> downscaling the low res image then extracting the difference
> between the low res image and the further downscaled very
> low res image

This way you have 3 different resolutions: 1) high-res image, 2) low-res image, 3) very-low-res image. You're describing that you take the difference between 2) and 3), but how do you do that if both have a different resolution? And then you say you add the result of the neural network to 2), but the output of that operation should still have the resolution of 2), so how do turn this to 1) resolution?

[feisty2]

Fair enough, it's worth a try. So if I understand correctly, you're doing the following to make the input "frequency aware"?

> the initial guess of the high frequencies is done by further
> downscaling the low res image then extracting the difference
> between the low res image and the further downscaled very
> low res image

This way you have 3 different resolutions: 1) high-res image, 2) low-res image, 3) very-low-res image. You're describing that you take the difference between 2) and 3), but how do you do that if both have a different resolution? And then you say you add the result of the neural network to 2), but the output of that operation should still have the resolution of 2), so how do turn this to 1) resolution?

say 1) is 512x512
2): 512x512->256x256->512x512 (use a regular resampling filter here, spline64 maybe)
3): 512x512->128x128->512x512 (likewise)

[feisty2]

that's the training part for sure, and for predictions, it goes like
1) ??? to be reconstructed
2) 256x256 -> 512x512
3) 256x256 -> 128x128 -> 512x512

[madshi]

Ah ok, I see, thanks. So basically you're saying that upscaling the input image to output resolution (256x256 -> 512x512) with spline64 more or less means "copying the low frequency components"?

[feisty2]

Ah ok, I see, thanks. So basically you're saying that upscaling the input image to output resolution (256x256 -> 512x512) with spline64 more or less means "copying the low frequency components"?

It is exactly "copying the low frequency components" if u pick sinc as the resampling filter,,, and sinc was the proposed resampling filter in my earlier posts, but I found spline works much better in practice than sinc so I'm using spline now

[Sparktank]

i forgot about this...

note to self:
(must learn to VS)