new super resolution toy for yall to mess with

[feisty2]

I just did a super resolution neural net, it's a pretty simple architecture but it's one alternative piece of shit, unlike most super resolution architectures out there tryna reconstruct the entire image, my architecture is frequency-aware, it merely cares about reconstructing high frequency components (which are the lost components after downscaling) and no other bullshit
the neural net itself doesn't know shit about the fact that downscaling only fucks high frequencies and low frequencies stay virgin, the net will probably waste a lot of capacity to first mess with low frequencies then realize it's no good, then waste another bunch of capacity tryna restore low frequencies if you don't tell it "don't fuck with low frequencies" explicitly.
so I designed a ResNet-like architecture and informed it "no bullshit for low frequencies"
concretely
http://i.imgur.com/9dkqSyM.png

the initial guess of the high frequencies is done by further downscaling the low res image then extracting the difference between the low res image and the further downscaled very low res image

"sinc" is the scaling algorithm for all resampling stuff, simply because it's the best possible filter frequency-wise, I'll let the neural net worry about the ringing crap

the architecture's based on SRCNN
(http://mmlab.ie.cuhk.edu.hk/projects/SRCNN.html)
test result:
lena
ground truth
http://i.imgur.com/CTUapHp.png
sinc (also the input of my architecture)
http://i.imgur.com/abRzkom.png
nnedi3
http://i.imgur.com/JPtHWdF.png
waifu2x (VGG?)
http://i.imgur.com/TBy0MZW.png
my architecture
http://i.imgur.com/iY2B4Q8.png

source code (keras) (https://github.com/IFeelBloated/SuperResolution/tree/master/SelfDesignedArchitecture)

[feisty2]

the major difference between the result of waifu2x and my architecture is that, mine doesn't turn photos into cartoonized posters, the underlying reason might be because waifu2x is not frequency-aware, so it incorrectly turned some low frequencies to high frequencies

[Phanton_13]

The reason for what waifu2x do is that it was it was designed and optimized to be used with anime/manga stile drawings, because of that it undeperforms (cartonize) with realworld images.

[kolak]

Looks good, so now we want this as vs and avisynth crazy optimised plugin :)

[feisty2]

The reason for what waifu2x do is that it was it was designed and optimized to be used with anime/manga stile drawings, because of that it undeperforms (cartonize) with realworld images.

no, waifu2x has a "photo" mode, trained specifically for photos, the result I posted was waifu2x with photo mode on, and the image still got cartoonized anyways

besides, the original SRCNN architecture was designed and trained for photos from the scratch, and it also cartoonizes photos like waifu2x

[poisondeathray]

Looks promising f2, keep it coming

Is it WIP, or finished?

Are there plans for avs/vs ?

Why are there border overlays ? Is there an issue with border pixels ? or the algorithm decided pretty girl looks better with a "picture frame" ? :D

[feisty2]

Looks promising f2, keep it coming

Is it WIP, or finished?

wip, this one has an average error of 3e-4, I'm working on another more complex architecture like, a combo of this one + VGG, it's very deep and has a lot of well engineered skip connections, yet has cut the average error down to somewhere around 6e-7, but it's a large net and fairly hard to train, u aint gonna see it anytime soon
feel free to play with this little one for now, it's written in keras so super user-friendly


Are there plans for avs/vs ?

no idea about vs, def no plan for avs

Why are there border overlays ? Is there an issue with border pixels ? or the algorithm decided pretty girl looks better with a "picture frame" ? :D

it's a prototype program, I didn't pad the input so pixels that don't have enough neighbor pixels around are black

[johnmeyer]

Nice job, expecially on the top back of the hat (the part that is brightly lit). The fine patterns do not get totally destroyed.

[feisty2]

another lame test, to see the potential max capacity of various architectures
we gonna train the architectures with just one image sample, and then feed them with the low resolution version of that sample, and see how well the architectures could (over)fit the sample they just learned, the sample has a lot of fragile high frequency details and not easy to learn, so we gonna analyze the output image and see how the architectures do their tradeoffs if they can't handle everything about the sample image

ground truth
http://i.imgur.com/l5ZkJy9.png

sinc
http://i.imgur.com/PEgs0lR.png

nnedi3
http://i.imgur.com/alYZmkw.png

SRCNN after 2000 iterations of backprop
http://i.imgur.com/N3ILYOf.png

waifu2x
http://i.imgur.com/ZnrNBWA.png

frequency-aware SRCNN after 2000 iterations of backprop (architecture proposed at #1)
http://i.imgur.com/N4a4IGp.png

new architecture wip after merely 300 iterations of backprop
http://i.imgur.com/pvkHY5Z.png

it's not exactly a fair test to nnedi3 and waifu2x cuz they are not trained specifically to overfit this test sample, but you see the tendencies of how they corrupt the sample

so sinc got a predictable result, ringing + lack of "delicateness"
nnedi3, waifu2x and srcnn all cartoonized the sample more or less, and nnedi3 got this weird "whirling" thing going in its output
frequency-aware srcnn didn't cartoonize the sample, but it also looks weird, I don't really know how to describe
the last wip architecture (https://github.com/IFeelBloated/SuperResolution/blob/master/SelfDesignedArchitecture%20(Advanced)/Architecture.png)has a huge capacity, it basically learned everything from the sample like absolutely no sweat, it has a pretty much picture perfect reconstruction...

[poisondeathray]

That last one is a very nice result, very clear demonstration of the "coarseness" vs. "fine" details in the various approaches

Approx. how long does it take for this per frame, for this dimension input/output ? Rough ballpark

When you eventually get a chance, can you run some test charts ?

Just thinking a bit ahead here for video, but how stable are results temporally ? (I'm thinking in terms of aliasing/flicker issues)

[feisty2]

Approx. how long does it take for this per frame, for this dimension input/output ? Rough ballpark

like the last wip architecture?
training: eons!! 300 backprop iterations on that single sample image took me ~10hrs with GTX1080, now you want maybe 3 billion iterations and 1 million samples to make that architecture practical in general... just do the math

reconstruction: I don't know, seconds, maybe?

When you eventually get a chance, can you run some test charts ?

I'm guessing that "when" will probably never happen, if u clicked the link up above, you woulda seen how big this architecture is and I simply got no appropriate hardware to run the training, I could probably feed it 1 or 2 samples occasionally, I just can't..... unless someone's willing to train it for me on a cluster of super computers



Just thinking a bit ahead here for video, but how stable are results temporally ? (I'm thinking in terms of aliasing/flicker issues)


my bet's on it should be temporally stable, the thing is, like I said earlier it's frequency aware so, the net will not reconstruct the entire image like other architectures and it directly takes the low res input as low frequency components, do some guess work on high frequency components, and merge them as the output, now because the low frequency components are simply copied from the input, they're already temporally stable by nature, and the high frequency components produced by the net gotta match the temporally stable low frequency components, I got good reasons to believe the high frequency guess work should be temporally stable as well, it's gotta match! and all that adds up to, the entire output sequence should be temporally stable

[kolak]

Hmm...quite computing intensive, but end result is crazy good (at least for me).

[Atak_Snajpera]

my bet's on it should be temporally stable, the thing is, like I said earlier it's frequency aware so, the net will not reconstruct the entire image like other architectures and it directly takes the low res input as low frequency components, do some guess work on high frequency components, and merge them as the output, now because the low frequency components are simply copied from the input, they're already temporally stable by nature, and the high frequency components produced by the net gotta match the temporally stable low frequency components, I got good reasons to believe the high frequency guess work should be temporally stable as well, it's gotta match! and all that adds up to, the entire output sequence should be temporally stable
Reconstruction high frequencies from low frequencies reminds me of HE-AAC where High Frequencies (typically from 24KHz) are "magically" reconstructed from lower frequencies on fly. Final result is also pretty good.

Ps. Yeah I know naive comparison but that was my first thought ;)

[LoRd_MuldeR]

Reconstruction high frequencies from low frequencies reminds me of HE-AAC where High Frequencies (typically from 24KHz) are "magically" reconstructed from lower frequencies on fly. Final result is also pretty good.

...only that SBR (the technology behind HE-AAC) is not blind.

With SBR, the down-sampling is part of the encoding/pre-processing process. So the original "high-frequency" spectrum is always known and can be analyzed by the SBR encoder before it gets "removed" by the down-sampling. This information about the original "high-frequency" spectrum is passed on the SBR decoder (at a very low bitrate) and allows for a more accurate reconstruction of the "lost" high frequencies.

The reconstruction of "high-frequency" details when all you have is the "low-pass" filtered version, i.e. no additional "magic" (out-of-band information about the original) is available, is a whole different challenge...

[StainlessS]

Again OT, but, is that also the way that Fraunhofer MP3Pro worked (had though to be decodable via standard mp3 decoder).
(also stored at ~ 24KHz I think).

EDIT: Nice one Feisty, lovely job, you deserve a pat on the back.
(No Grouchy, not a cow pat).

[Katie Boundary]

They all look the same to me.

[manolito]

They all look the same to me.
I wanted to say the same thing, but I kept quiet out of fear to be called "visually challenged" (or probably something less P.C.)... :devil:

[LoRd_MuldeR]

I wanted to say the same thing, but I kept quiet out of fear to be called "visually challenged" (or probably something less P.C.)... :devil:

If you can't see a difference between "ground truth" and "my architecture" it would indicate that the reconstruction filter indeed does an excellent job ;)

The difference between "ground truth" (original) and "sinc" (low resolution), and then again between "sinc" (input) and "my architecture" (output) should be pretty obvious, at least in this particular example.

Also the difference between, e.g., "nnedi3" and "my architecture" is quite apparent. The former introduces that "expressionism" look, the latter is sharper and less "artificial".

[StainlessS]

I knew that there was good reason that we' all, put up with master Feisty, this is it. I say again, lubely jubely young master Feisty, job well done sir.
You be a star :)

[kolak]

They all look the same to me.

Your glasses are on your desk :)

[raffriff42]

Just looking at them on the page yes, they look the same. I had to save all the images and flip between them in IrfanView; the differences then became clear.

Another good way to compare images is to paste them into a suitable image editor (Photoshop, PSP, Gimp) as layers and then toggle the layers.

feisty2's result looks dam good BTW.

[hello_hello]

I wanted to say the same thing, but I kept quiet out of fear to be called "visually challenged" (or probably something less P.C.)... :devil:

That was my initial reaction too, but I'm still using a Trinitron CRT for general perusal of the internet so I excused myself. :)

Downloading each image and switching between them displaying on the TV shows some obvious differences, and comparing them that way also doubles as a memory test given the files have names such as pvkHY5Z.png and ZnrNBWA.png.

Something I somehow managed to notice....
According to Irfanview each image has a similar number of unique colours. Around 265 for the latest examples.... except for the waifu2x screenshots, where the first one has around 4000 unique colours and for the second it's 8000. If I reduce the saturation to zero the number of unique colours drops back to match the rest.

It doesn't make any difference to anything, but I'm a little curious as to why that is.

[feisty2]

R u guys browsing web on cell? Images are already displayed at full size on the web page and you don't really need to download them

[raffriff42]

"memory test" lol, I gave them mnemonic names right away as I knew I would fail that test.

Since the images are grayscale, any color count over 256 must be due to compression artifacts.

[feisty2]

Those r png images, no compression artifacts

[feisty2]

Ur software fucked the source images up if ur seeing more than 256 unique colors

[poisondeathray]

Ur software fucked the source images up if ur seeing more than 256 unique colors

No - he's correct . irfanview reports >256

The reason is the lower black border has some colored pixels, not 0,0,0. Use a color picker .e.g some are 6,4,3 , etc.. on the waifu2x

Interestingly, if you run a color picker on the main image, there are some channel discrepancies too e.g. 105,105,106 etc...


BTW - How does your "toy" do with color images , or are you testing that yet ?

[feisty2]

No - he's correct . irfanview reports >256

The reason is the lower black border has some colored pixels, not 0,0,0. Use a color picker .e.g some are 6,4,3 , etc.. on the waifu2x

Interestingly, if you run a color picker on the main image, there are some channel discrepancies too e.g. 105,105,106 etc...


BTW - How does your "toy" do with color images , or are you testing that yet ?

Waifu2x only accepts RGB input when "photo" mode is on, and I faked one by having R, G and B all copied from the grayscale image and I got an RGB representation, apparently waifu2x screwed the hell up and generated some imaginary colors

I don't need to test it for color images, an RGB image is a combo of 3 grayscale images, u do it for all 3 channels and get things done

[poisondeathray]

I don't need to test it for color images, an RGB image is a combo of 3 grayscale images, u do it for all 3 channels and get things done


You can get quite messed up results treating R,G,B separately , depending on the source image and what operations are done exactly.

Fine contrast and higher frequencies (or low frequencies for that matter) in a given channel doesn't mean that relationship is the the same in a sister channel

In real life images, taken with real cameras, often 1 channel is dramtically different. Or sometimes is "worse" , maybe more noisy. Microcontrast and high frequencies will already be different in a given channel to begin with. Or sometimes you have clipping 0 or 255 in only 1 channel in some parts. ie. R,G,B from the same image could be very different to begin with from a frequency point of view. In some of those scenarios when you look at the RGB composite, I suspect you're going to introduce some form of color noise problems. Feel free to shoot this down when you have time

[feisty2]

Then u convert the color image to LAB colorspace and do it on L, and scale AB with regular resizers like bicubic

[poisondeathray]

ok I'll get right on it after I finish building my supercomputer farm :D

meanwhile, a bit off topic, but why would waifu2x do that ? Copy channel to R,G,B should still be valid input. Which waifu2x implementation was it ?

[feisty2]

ok I'll get right on it after I finish building my supercomputer farm :D

meanwhile, a bit off topic, but why would waifu2x do that ? Copy channel to R,G,B should still be valid input. Which waifu2x implementation was it ?

Because there's only RGB model for photographic images, it rejects anything else and it simply doesn't bother to do the fake RGB trick for grayscale images

[poisondeathray]

So I think there is a bug. Are you saying waifu2x can't handle "greyscale" RGB image input ?

I get similar results so far with one of the waifu2x vpy implementations, where black is messed up. The small channel discrepancy observation like +/- 1 in a channel could possibly be explained by dithering during bitdepth conversions, but I made sure to disable and I didn't get any so far on quick tests. Anyways it's a bit off topic for this thread

[feisty2]

ok I'll get right on it after I finish building my supercomputer farm :D

meanwhile, a bit off topic, but why would waifu2x do that ? Copy channel to R,G,B should still be valid input. Which waifu2x implementation was it ?

And I got something off the topic as well, I remember seeing u on the videohelp forum and it says ur Canadian, how tolerant r u guys about spelling, will it be okay if I keep doing "archeology" and "dialog" and "maneuver" and not weird shit like "archaeology" or "dialogue" or "manoeuvre" and without being recognized as typos?
I'm thinking of doing my masters in uoft

[poisondeathray]

And I got something off the topic as well, I remember seeing u on the videohelp forum and it says ur Canadian, how tolerant r u guys about spelling, will it be okay if I keep doing "archeology" and "dialog" and "maneuver" and not weird shit like "archaeology" or "dialogue" or "manoeuvre" and without being recognized as typos?
I'm thinking of doing my masters in uoft

itt doesnt bother mee dood



But there are proper British ways of spelling like "colour" vs "color". They invented "English" so I suppose they should be right

No we're not a colony any more, neither are you guyz dood, so do whateva you feel like

[feisty2]

itt doesnt bother mee dood



But there are proper British ways of spelling like "colour" vs "color". They invented "English" so I suppose they should be right

No we're not a colony any more, neither are you guyz dood, so do whateva you feel like

Actually it's just my personal preference to keep things a bit more "futuristic", most people in states do "archaeology" and "dialogue" and "technique" but I'm like, screw them, I'm doing "archeology" "dialog" and "technic", bite me if they can, a more unified and coherent and less etymologically influenced spelling fits the futuristic concept better, one Aussie bitched about my spelling once cuz my spelling's considered alternative but still correct in states, and apparently plain incorrect outside us, I was just wondering if I could extend "outside us" to "outside North America"

And now back on topic :), the RGB model was presumably mostly trained with color images, and there's nothing detecting if the RGB input is genuine or grayscale in waifu2x, so I think it's perfectly normal (and predictable) that waifu2x does something fishy to grayscale RGB images

[feisty2]

so much for inter-architecture comparisons, one more intra-architecture comparison before I mail my keras code to the supercomputer lab in my college
so 3 variants of the lite version of my wip architecture, 3 share the same number and kind of layers and hyperparameters, the inputs and connections are different, all trained with 4 samples and 300 backprop iterations

Ground Truth
http://i.imgur.com/gJfEBF2.png
http://i.imgur.com/uNGetea.png
http://i.imgur.com/d1U7XlA.png
http://i.imgur.com/rdnMiNP.png

Architecture 1: straight forward connections, no frequency-aware, VGG-like, similar to waifu2x but a bit more complex
average error: around 6e-4
topology:
http://i.imgur.com/V6DA43b.png
results:
http://i.imgur.com/8rB3JOV.png
http://i.imgur.com/37xNPwy.png
http://i.imgur.com/tJk5AnF.png
http://i.imgur.com/nALw780.png

Architecture 2: forward and skip connections, no frequency-aware, combo of VGG and ResNet
average error: around 4e-4
topology:
http://i.imgur.com/j1bkNyc.png
results:
http://i.imgur.com/fLHmkFX.png
http://i.imgur.com/opW8PBU.png
http://i.imgur.com/nCXgtAd.png
http://i.imgur.com/kVBQs0M.png

Architecture 3: forward and skip connections, frequency-aware, lite version of my wip architecture
average error: around 8e-5
topology:
http://i.imgur.com/gkKwEss.png
results:
http://i.imgur.com/AH77GDL.png
http://i.imgur.com/H5qubQd.png
http://i.imgur.com/TIdkH2h.png
http://i.imgur.com/vQGuQJ9.png

[feisty2]

extras for poisondeathray: how to make it work for color images
say u got a low res RGB image
http://i.imgur.com/5Hmqlr5.png
first extract the luminance from it and u got
http://i.imgur.com/AAmh3Yi.png
and reconstruct the luminance with super resolution neural net and u got
http://i.imgur.com/kJaHE8a.png
then extract the difference (reconstructed high frequencies) between the reconstructed luminance and the low res luminance and apply the difference to R,G and B
concretely

import vapoursynth as vs
core = vs.get_core()
LowResY = core.raws.Source('LowRes.bin', 512, 512, src_fmt="GRAYS")
ReconstructedY = core.raws.Source('Reconstructed.bin', 512, 512, src_fmt="GRAYS")
LowResRGB = core.raws.Source('LowResRGB.bin', 512, 512, src_fmt="GRAYS")
HiFreq = core.std.MakeDiff(ReconstructedY, LowResY)
LowResR = core.std.SelectEvery(LowResRGB, 3, 0)
LowResG = core.std.SelectEvery(LowResRGB, 3, 1)
LowResB = core.std.SelectEvery(LowResRGB, 3, 2)
ReconstructedR = core.std.MergeDiff(LowResR, HiFreq)
ReconstructedG = core.std.MergeDiff(LowResG, HiFreq)
ReconstructedB = core.std.MergeDiff(LowResB, HiFreq)
clp = core.std.ShufflePlanes([ReconstructedR,ReconstructedG,ReconstructedB], 0, vs.RGB)
clp.set_output()

result:
http://i.imgur.com/nYMa2cG.png

ground truth
http://i.imgur.com/aHTxPOh.png

no color noise, no bullshit, no nothing

[poisondeathray]

It's not necessarily the best example , because "color" Lena ground truth doesn't really have much real "color". R,G,B, is severely quantized to begin with (255 unique colors, not even 256)

But I understand the color workaround , it should work

Also do you know off hand what weighting formula is used to extract "greys" in vs from RGB?


How long before you package it into a 1 click solution ? :D:D

[feisty2]

Also do you know off hand what weighting formula is used to extract "greys" in vs from RGB?


I went with rec709 coefficients, u could also have a more accurate luminance by converting ur image to CIE LAB and picking the L channel

given by the performance of the supercomputer in my college, I'll probably train a color model for CIE LAB images, 3 weight files, one for L (grayscale), one for A and one for B, or maybe 2 files, one for L and one for AB

How long before you package it into a 1 click solution ?

http://i.imgur.com/CZQ0giR.png

[luquinhas0021]

the initial guess of the high frequencies is done by further downscaling the low res image then extracting the difference between the low res image and the further downscaled very low res image
What about, instead this method for the initial guess of high frequency, you downscale the ground truth image, upsclae the downscaled image and do the difference map between them?
The Google's Super Resolution algorithm performs more or less under this way, and do it!

[feisty2]

the initial guess of the high frequencies is done by further downscaling the low res image then extracting the difference between the low res image and the further downscaled very low res image
What about, instead this method for the initial guess of high frequency, you downscale the ground truth image, upsclae the downscaled image and do the difference map between them?
The Google's Super Resolution algorithm performs more or less under this way, and do it!

how the fuck are you gonna find the "ground truth" when all you have is the low resolution image? Use your brain

[luquinhas0021]

how the fuck are you gonna find the "ground truth" when all you have is the low resolution image? Use your brain
You who gotta use a little more your brain instead try to offence me. Anybody who is a little bit wise realize that "ground truth", in my post, means any image that you didn't applied the downscale operator in.
Ok! Let's replace "ground truth" by "high resolution".

[feisty2]

You who gotta use a little more your brain instead try to offence me. Anybody who is a little bit wise realize that "ground truth", in my post, means any image that you didn't applied the downscale operator in.
Ok! Let's replace "ground truth" by "high resolution".

the net must take some low res input, and feed back some hi res output, the initial guess is an input, concretely, in the training process,
input: difference between the low res and the very low res (initial guess)
output: difference between the ground truth and the low res

now you literally make the output as the input, you mr know it all tell me what would the output be?

[feisty2]

also, the initial guess being the input of the net, must stay practical and valid after training when you're applying it to a new image that simply has no ground truth reference (what's the point of reconstruction if it already has what you want!?)
and how are you gonna make your version of "initial guess", which requires the ground truth, work on that new image that does not have a ground truth reference?

[luquinhas0021]

the net must take some low res input, and feed back some hi res output.

Coming back to suggestion: be any image - let's call it of "HR". The HR image is downscaled - let's call the downscaled HR image of "D HR". The D HR image is upscaled - let's call the upscaled D HR image of "U D HR" and assume that the resolution of U D HR image is the same resolution of HR image.
Certainly, the U D HR image has more blur and, possibly, less details than HR image.
Create a difference map between the HR image and the U D HR one, in your case, focusing on high frequencies, and feed the NN with this difference map.
Do that with many HR images
Done! Instad upscale any D HR image, all the NN has to do is minimize the difference between the U D HR image and the HR image; in other word, deblur it.

[feisty2]

shit, it's like talking to a fucking 3-year-old kid, just stop embarrassing yourself for your own sake, what you just said is the standard procedure for pretty much every super resolution neural net out there, and it sucks cuz it's not frequency-aware, the major point of my architecture is being frequency-aware, that is, reconstruct the high frequency components instead of the entire image, the point I have exhaustively emphasized in my previous posts, you really should stop telling other people "gotta use a little more of their brain" when you're not capable of understanding basic English

[luquinhas0021]

I'm not capable of understand basic english...
I won't reply you, because I know it isn't true.
But I'm very curious to know: once you say that your NN is so good and peculiar, if I try upscale a image that is a result of a downscaling done by Bilinear, for example, the result even will be good? Or your NN only performs well under the condition it was trained? If yes, it's not so peculiar as you wanna say.

[feisty2]

if I try upscale a image that is a result of a downscaling done by Bilinear, for example, the result even will be good?
that's called regularization, the answer is yes if trained with enough samples just like any neural net architecture

[luquinhas0021]

Why did you use sinc by default downscaler of your algorithm? There are downscalers that degrades more an image, in sense of lost more high frequencies, like Bilinear.