or maybe we dont need GANs to generate photo realistic image upscales..

[feisty2]

I went on and now experimented on 1/9 of my super resolution neural net, 5 million parameters roughly trained out of total 45 million parameters, I only trained it with 3 actual images (augmented to 18 images via flipping, mirroring and stuff) for 20 epochs of backprop, then tested the neural net on 4 images that it had never seen before

test results
small:
https://i.imgur.com/3gmdTF1.png
upscaled:
https://i.imgur.com/UI6uxM3.png

small:
https://i.imgur.com/SZJH9DC.png
upscaled:
https://i.imgur.com/gMQckBb.png

small:
https://i.imgur.com/6SYq3xy.png
upscaled:
https://i.imgur.com/q5jWfZr.png

small:
https://i.imgur.com/BH6V7cK.png
upscaled:
https://i.imgur.com/NPtvo28.png

the results already look pretty photo realistic to me, I aint got no GAN in my neural net, no perceptual loss either, I simply picked L2 (aka MSE) as the loss function (PSNR friendly) and I honestly wasn't expecting something this good, the results are almost as photo realistic as the results of GANs, and the chance of shit going south should be much smaller than GANs since the neural net was optimized for L2, which is very loyal to the ground truth,,,

so maybe we don't need GANs to have photo realistic upscales, all we need is a better neural net structure (the structural design of my neural net serves a special physical significance, it's not about brainlessly stacking layers and blocks like some other models)

[poisondeathray]

the textures don't quite look right , keep on training it f2 . 5M not enuf

It's progress and it looks good, but there is a "coarseness" to the fine details that is reminiscent of nnedi2/3 upscaling artifacts . The fine details are what differentiates this approach

There are some issues with the bridge /railing that weren't there on your earlier example (look almost like field/interlace artifacts)

[feisty2]

the textures don't quite look right , keep on training it f2 . 5M not enuf

any way I can make GPUs 100x faster..? I got tesla p100 in school lab but it aint fast enough for my neural net and training set,,, I'm desperate for better hardware

well, it doesn't look quite rite for mainly 2 reasons, and the number of parameters is actually not one of them, first, training set of 3 effective images is too small, also, 20 epochs of backprop aint the shit for a large neural net to converge, I'd say probably at least 300 epochs

[poisondeathray]

I don't know the details of what it takes to train... Can you pause and restart for example ? Does it take 1 contiguous session ? Can you train it overnight at school for example for a few weeks ?



It looks good, I just think you can do better :D

Any thoughts as to why would this version generate those bridge horizontal artifacts ?

[feisty2]

I don't know the details of what it takes to train... Can you pause and restart for example ? Does it take 1 contiguous session ? Can you train it overnight at school for example for a few weeks ?

yes, but I did the math, with tesla p100 it's gonna take approx 8 yrs if I wanna train the full model with the full training set for 300 epochs of backprop


It looks good, I just think you can do better :D

its gonna be better for sure, I'm just surprised that for now it already looks closer to the results of GANs in general (yes there're shitty details and textures) rather than "traditional" stuff like nnedi3 or waifu2x


Any thoughts as to why would this version generate those bridge horizontal artifacts ?

simple, training set too small and 20 epochs of backprop too few

[feisty2]

It's progress and it looks good, but there is a "coarseness" to the fine details that is reminiscent of nnedi2/3 upscaling artifacts . The fine details are what differentiates this approach


speaking of nnedi3

https://i.imgur.com/Q8gYkuW.png
https://i.imgur.com/V9KzJIC.png
https://i.imgur.com/Lp8xtBF.png
https://i.imgur.com/b3xUTWi.png

no, this is not what I would call, photo realistic

[Snap]

Always enjoy seeing NN image scaler examples. Looking great.

speaking of nnedi3

no, this is not what I would call, photo realistic

It's interesting comparing them at similar sharpness. Lacks the same finer details which make up most of the photo realism but does have cleaner edges in places (the vertical bridge posts/sign for ex).

https://i.imgur.com/kfHsGQa.png

https://i.imgur.com/8cKNMLR.png

[poisondeathray]

speaking of nnedi3

no, this is not what I would call, photo realistic

Exactly! That's the main differentiating factor, the very 1st thing I said in the other thread that impressed me. The "coarseness" result common to nnedi3 type approaches vs. the fine details when you introduced this in the other thread

This version has some coarseness features too (definitely better than nnedi3), but going back now I can see why - the other pics in the other thread were at lower dimensions

So I'm thinking you can improve on it :D

[raffriff42]

The bridge has minor artifacts (oversharpening and/or residual interlacing) that are exaggerated (or at least, not smoothed) in f2's images

[CkJ]

Could you test on these images?
https://images2.imgbox.com/4b/70/PaaABeLQ_o.png

https://images2.imgbox.com/4b/c7/SJdyZmpN_o.png

https://images2.imgbox.com/5c/62/CWq3oLBq_o.png

I'm curious to see your approach compares with NGU Sharp.
https://images2.imgbox.com/84/40/VXfOlcoP_o.png

https://images2.imgbox.com/8a/53/TZ2LPixt_o.png

https://images2.imgbox.com/92/b0/XhytigJK_o.png

[feisty2]

there's zero point to compare this with NGU just yet, cuz it's doomed to introduce hell lot more artifacts than NGU for now, cuz the neural net is not yet fully trained, actually make that, yet just "barely trained"
anyways, the castle image, the source image is way too soft and doesn't have a lot of details,, so the result is just mediocre quality and suffers from mild level of artifacts
https://i.imgur.com/L653Xpr.png

the cartoon image is 100% pointless, it's not photo realistic to begin with
the result of lighthouse image suffers from crappy artifacts..

[CkJ]

Well, it looks promising. How about the speed? Is it possible to use on videos or just available for images? And will you release an avisynth plugin or it will be a commerical software? Thanks.

[feisty2]

Well, it looks promising. How about the speed? Is it possible to use on videos or just available for images? And will you release an avisynth plugin or it will be a commerical software? Thanks.

it's free and open source, but I will only release the source code and the exact design of the neural net after I submitted my paper to CVPR, sounds a bit snobbish but... trust me I'm really not snobbish like that, I just need this paper to get me thru my applications for graduate schools, if I release the source code rite now someone mite silently steal my idea and I'm fucked for good (the neural net mentioned in this thread is very different from the one in my super resolution github repo, so the code is not yet public)

it works for videos (temporally stable), but it's slow

[wonkey_monkey]

feisty2, can you make anything of this?

https://github.com/DmitryUlyanov/deep-image-prior

Seems to be about using untrained neural nets for noise reduction, inpainting, and upscaling. One of the test images from this thread is in the paper.

[poisondeathray]

feisty2, can you make anything of this?

https://github.com/DmitryUlyanov/deep-image-prior

Seems to be about using untrained neural nets for noise reduction, inpainting, and upscaling. One of the test images from this thread is in the paper.

Yummy!

Looks promising too

Does it lend itself more easily to vapoursynth since both use python ?

[feisty2]

feisty2, can you make anything of this?

https://github.com/DmitryUlyanov/deep-image-prior

Seems to be about using untrained neural nets for noise reduction, inpainting, and upscaling. One of the test images from this thread is in the paper.

I just read the abstract of this paper and I wouldn't say it's... entirely untrained, it does seem like some kind of unsupervised learning, anyways it's an intriguing idea and I could try it when I got time
there're also other multi purpose neural nets based on supervised learning
https://arxiv.org/pdf/1608.03981.pdf
https://arxiv.org/pdf/1606.08921.pdf

[feisty2]

Yummy!

Looks promising too

Does it lend itself more easily to vapoursynth since both use python ?

no, cuz there's no numpySource, also u gonna need GPU versions of TensorFlow or PyTorch or stuff like that, all extremely hard to install, u mite as well wanna shoot urself after 1000 error msgs pop outta nowhere and ur installation of any of these things is still failure, CPU versions r easy to install but they r useless, too damn fucking slow, I'm talking about 100x slower
and amd gpus r fucked too, all require nvidia gpus

python does rule the machine learning field, so basically implementations for all recent papers r coded in python

[StainlessS]

Respect master Feisty, for something so early in development, is looking really good.

[poisondeathray]

no, cuz there's no numpySource, also u gonna need GPU versions of TensorFlow or PyTorch or stuff like that, all extremely hard to install, u mite as well wanna shoot urself after 1000 error msgs pop outta nowhere and ur installation of any of these things is still failure, CPU versions r easy to install but they r useless, too damn fucking slow, I'm talking about 100x slower
and amd gpus r fucked too, all require nvidia gpus

python does rule the machine learning field, so basically implementations for all recent papers r coded in python

Ok thanks for explaining

No 1 clicky, or even semi-easy solution for a script :(

Back to training then...

https://s9.postimg.org/bs8r0967j/f2_back_to_training.gif (https://postimages.org/)

[cwk]

any way I can make GPUs 100x faster..? I got tesla p100 in school lab but it aint fast enough for my neural net and training set,,, I'm desperate for better hardware


How about a kickstarter campaign to rent a GPU-heavy instance in AWS for a couple of days? I'd contribute if it got you closer to a fully-trained instance.

[madshi]

Looks promising. It definitely has a more "high-res" look to it than any other non-GAN algorithms I've seen so far. Of course a key question will be how much the artifacts will go down when fully trained.

As much as I like comparisons to NGU, it's probably not very fair, because NGU was created to run in real time. @feisty, how long does it take to upscale one 1080p video frame (using the Tesla), just to get a first impression about speed?

(And if I may suggest, try using L1 loss instead of L2 loss.)

[feisty2]

@cwk
thx but my school is now funding me 10 tesla p100 gpus for ntire2018

@madshi
I tried L1 loss before and it somehow failed to converge for my neural net, I tried to keep L1 loss and experimented tons of weird shit like "residual scaling", "warm up very low learning rate", "gradient clipping", ..., all failed to make L1 loss converge, things went back normal as I switched to L2 loss later, it just converged without any other extra bullshit
it takes a few secs to upscale 512x512 to 1024x1024 on tesla p100

[madshi]

Hmmm... So about a minute for upscaling one 1080p frame?

Strange thing, never had any trouble with L1 yet. Of course L2 helps achieve better PSNR, which might be beneficial for your paper. ;) L1 usually looks better to my eyes, though. You could try Huber or Charbonnier loss as alternatives. Or you could try something like 10% L2 and 90% L1. Maybe those 10% L2 could help "guiding" L1 with your net?

[feisty2]

results after 150 epochs of backprop
https://i.imgur.com/s0xjkwI.png
https://i.imgur.com/6qOL76F.png

@madshi
I always knew charbonnier loss from LapSRN (CVPR2017) but never got to try that, maybe I'll give it a shot :D

[madshi]

Looks even "finer structured" than before now. It does look increasingly similar to GAN upscaled images. Are you still training with only 3 images? I wonder if training with such a small dataset might make your NN behave somewhat more GAN like? My understanding of GAN is that (very simplified) the NN learns a tendency to pick exactly one most likely solution, instead of averaging a number of probable solutions. So I wonder if your NN has such a large capacity and such a small dataset, maybe it ends up having exactly one solution for each problem that is far more likely than the others, resulting in something which looks more like GAN? Maybe if you train with a larger dataset, the typical look of averaged NNs comes back to some extent, because the NN then has more possibly matching solutions, leading to a higher use of averaging?

Just wildly speculating, of course.

Would also like to see how your algo does on this image:

http://madshi.net/clown.png

I like the clown and castle images because they have a good mixture of high contrast geometric features and nature (trees, bushes etc). A good algo should do well on both. E.g. adding weird dot-crawl artifacts etc can make random textures (like trees, bushes etc) look even more realistic, but can harm elsewhere...

[feisty2]

https://i.imgur.com/3CZ7k2K.png
well, the training set contains 3 actual images but augmented to total 18 images, the uncompressed binary of the training set is around 1GB (after augmentation and overlapped slicing) and the file size of the weights is around 20MB, so no way it coulda memorized all training samples, but yeah it's been overfitting cuz the training set is still way too small, I'll expand the training set to a much larger one for ntire 2018 and then I'll see how it goes, maybe ur rite :)

[madshi]

One thing the clown and castle images are also good for is to check how consistent the upscaling algo is in keeping line thinning roughly constant. E.g. if you look at the white van in the clown image, it has a stripe in the middle. Ideally an image upscaling algo should thin this stripe the same way from left to right. You can see in your upscaling result that the stripe is partially thinned very much and partially not thinned at all. For comparison, here's NGU:

NGU Sharp Low Quality: http://madVR.com/doom9/clown/clownGrayNguSharpLow.png
NGU Sharp Very High Quality: http://madVR.com/doom9/clown/clownGrayNguSharpVeryHigh.png

In Low Quality, the stripe is not thinned equally, either, and not thinned much at all. In Very High Quality, the result is not perfect, but quite ok.

Do you have good training images? If not, I can send you some nice ones.

[feisty2]

Do you have good training images? If not, I can send you some nice ones.

yeah, I got DIV2K training set from ntire 2018, it has 900 extremely high quality 2k images and 100 low res test samples, the reason I used only 3 of them was that I didn't have enough GPUs to go for a larger training set


One thing the clown and castle images are also good for is to check how consistent the upscaling algo is in keeping line thinning roughly constant. E.g. if you look at the white van in the clown image, it has a stripe in the middle. Ideally an image upscaling algo should thin this stripe the same way from left to right. You can see in your upscaling result that the stripe is partially thinned very much and partially not thinned at all. For comparison, here's NGU:

well, cuz I only trained it on 3 actual images...

[madshi]

Yes, just saying it's a good test image to check things like that... :)

[feisty2]

Strange thing, never had any trouble with L1 yet. Of course L2 helps achieve better PSNR, which might be beneficial for your paper. ;) L1 usually looks better to my eyes, though. You could try Huber or Charbonnier loss as alternatives. Or you could try something like 10% L2 and 90% L1. Maybe those 10% L2 could help "guiding" L1 with your net?

I found logcosh loss (defined as "ln(cosh(error))") generally works the best for my neural net after a few experiments, it has the following mathematical properties:
https://i.imgur.com/c1flNIv.png
https://i.imgur.com/3UrmYAq.png
so it shares similar behaviors with Huber loss but it's a bit better cuz it's analytic, and any order of its derivative is also analytic, it's prettier cuz it got no non-differentiable singularities and Huber loss does (edit: the Huber loss itself is differentiable but its derivate is not, it's not differentiable for higher orders)

[madshi]

Interesting! Might give that a try, too.

[feisty2]

the ground truth HD version of the castle image, extracted from DIV2k, if anyone is interested
https://s10.postimg.org/x9nhffb3r/0887.png

[madshi]

It's not exactly the same image (e.g. the clouds are in a different position), but I suppose it doesn't matter much. Generally, I think it's a really good test image. So we could create a low-res version of this groundtruth and then use that instead of the other image. Of course a good question would be which downscaling algo should be used for that.

[feisty2]

It's not exactly the same image (e.g. the clouds are in a different position), but I suppose it doesn't matter much. Generally, I think it's a really good test image. So we could create a low-res version of this groundtruth and then use that instead of the other image. Of course a good question would be which downscaling algo should be used for that.

I would personally very much prefer spline64, seems like a balanced choice to me, but currently my neural net is optimized for bicubic (b=c=1/3) cuz that's what ntire2018 asked for..

[madshi]

Yeah, almost all papers test with catrom. Which makes sense because it makes PSNR/SSIM results more comparable. But for real life use it would be interesting to test the final NN with different downscaling methods, too.

[edcrfv94]

Can NGU restoration or line thinning with out upscale?

[madshi]

Wrong thread, edcrfv94. Anyway, short answer: That's not what it was made for. However, there are special cases where it might still do what you want. E.g. if the video you're playing was upscaled by the studio with a soft/bad upscaler (e.g. many UHD Blu-Rays were upscaled with Catrom), then you could try downscaling it to half resolution first, then upscale it again with NGU. That might then remove some aliasing and produce thinner lines. Follow-up please in the madVR thread.

[madshi]

One month later. Good progress on your network training, feisty2? :)

[Socia]

Wrong thread, edcrfv94. Anyway, short answer: That's not what TestoPrime review (https://www.globenewswire.com/news-release/2025/05/06/3074688/0/en/TestoPrime-Review-and-Results-A-30-Day-Test-By-WhatBest-Reveals-Results-Of-Using-TestoPrime-For-1-Month.html) were made for. However, there are special cases where it might still do what you want. E.g. if the video you're playing was upscaled by the studio with a soft/bad upscaler (e.g. many UHD Blu-Rays were upscaled with Catrom), then you could try downscaling it to half resolution first, then upscale it again with NGU. That might then remove some aliasing and produce thinner lines. Follow-up please in the madVR thread.

That's an awesome idea imo. How much would something like that cost?

[cwk]

That's an awesome idea imo. How much would something like that cost?

Last I checked, Google has the best GPU offerings, including the P100 nvidia chips. The P100s are roughly $750/month as an add-on to any other Google Compute Engine instance. So, roughly $900-$1200/month for a single GPU depending on your instance type. You could add additional GPUs to an instance depending on your need.

Pricing for the GPUs are here: https://cloud.google.com/compute/pricing#gpus

[Asmodian]

At those prices buying a Titian V would seem a better investment.

[Selur]

may be buying a Titan V wouldn't be such a good idea,... http://www.theregister.co.uk/2018/03/21/nvidia_titan_v_reproducibility/

[SSH4]

Sorry, may be not correct to post in this thread.
But found fresh commercial release of Topaz A.I. Megascale.
And looks like it finally that can be named modern A.I. upscale, that ready for consumers.

SRC
https://i.imgur.com/gMsPwyw.png

200% no enchance
https://i.imgur.com/Ftly6zJ.jpg

400% with enchance
https://i.imgur.com/PoEIzan.jpg


SRC
https://i.imgur.com/FDHoTBu.png

200% no enchance
https://i.imgur.com/3tdJR1Y.jpg

400% with enchance
https://i.imgur.com/Zd6EMVa.jpg


SRC
https://i.imgur.com/vjzcJT1.png

200% no enchance
https://i.imgur.com/T6MCEJM.jpg

400% with enchance
https://i.imgur.com/gVJa87X.jpg


SRC
https://i.imgur.com/9Zk9ape.png

200% no enchance
https://i.imgur.com/mIj8TTW.jpg

400% with enchance
https://i.imgur.com/3I2A99w.jpg


Good that it quite fast and have batch support.
Bad (may be) that this it work only with RGB.

[madshi]

It looks very nicely detailed and has a high-res look, but whether or not it has the same problems (e.g. create "ghost" images where the neural network paints distorted faces where they shouldn't be etc) like other SRGAN neural networks we don't know for sure. Furthermore, in the baby face, look at the eye lashes of the left eye. *Heavy* ringing artifacts there!

[wonkey_monkey]

And the real question is: how well does it work with video?

[johnmeyer]

It looks very nicely detailed and has a high-res look, but whether or not it has the same problems (e.g. create "ghost" images where the neural network paints distorted faces where they shouldn't be etc) like other SRGAN neural networks we don't know for sure. Furthermore, in the baby face, look at the eye lashes of the left eye. *Heavy* ringing artifacts there!I see ringing on the 200% pic in his right eye (his right, our left), but do NOT see those artifacts in the 400% version posted on Imgur. The 400% has "enhancement" and the 200% does not. However, I have no idea what enhancement is being applied and whether that would reduce ringing.

[wonkey_monkey]

For what it's worth, here is the baby's eye from the 200% version (left) compared with the 400% version scaled down:

https://i.imgur.com/5OG7Oam.png

400% version is a lot cleaner and has a lot fewer JPEG artefacts (over the whole image).

[SSH4]

I guess that enhance mean some presharpen and some denoise.

With videos... well it have batch mode. just drag and drop and start.

More question about temporal stability for videos. And not least colors (as i see it work only with RGB)

[SSH4]

Comparing to other upscalers. I think this one can be used for some video works. But it will required prefiltering (especially for DVD sources) and post filterings (like dehalo,"smart" sharpen, etc.) to rich best results.
So this is not "one click" Solution. but it looks more ready than most of "proof of concepts" on web. Because already have prebuild binaries (with not a bad UI) already optimized for memory (you can upscale 4K to 8K) etc.

[foxyshadis]

Katie-derailment has been moved out to its own thread.