How nnedi3 works?

[luquinhas0021]

How this algorithm work?

[raffriff42]

It's very simple, really :eek:The predictor networks in nnedi3 take in an area around the pixel in question, and, based on those pixel values, produce a guess for the missing pixel value. Those predictor networks are just mathematical models with sets of parameters (weights). To learn the parameters I take lots of training examples (grabbed from a 1000 frame video sequence containing images from many different sources), start with a random initial guess for the weights, and then perform online gradient descent training. Which simply means that I repeatedly grab a single training pattern, run the pattern through the model, compute the error in guessing the output value, compute the partial derivative of the error with respect to each weight (term) in the model, and then update the weights. Keep repeating until a minimum in the error surface is reached (progressive stops).

L1 (absolute error) vs L2 (squared error) simply define how I compute the error term. L2 will weight large errors more - an error of 20 will count 16 times as much as an error of 5 - whereas L1 will weight errors proportionally - an error of 20 will count 4 times as much as an error of 5. So with the L2 error measure the model will spend more 'effort' trying to fit the hardest cases since it will make more sense to fit the hard cases just a little better at the cost of some accuracy in a lot of the easy cases (for example, drop a single error from 100 to 20 while increasing many errors of 5 to 8, etc...). L1 will tip the balance more toward fitting the easier cases better. For deinterlacing, I think L1 makes more sense than L2 because after a certain point (say a miss of 20 or whatever) it doesn't matter if you miss the true value by 50 or 100 or 150 they will all look bad. However, the model will spend a lot of effort trying to fit the 150 case just a little better. On the other hand, if you are close to the true value a small difference can make a big visual improvement (for example: miss by 2 vs 10). The results of nnedi3_abs seem to back this up. In fact, I think going even further, using L0.5 - sqrt(abs(error)) - produces better results based on my initial tests.

Speaking of which, does anyone here think there is room for significant improvement by selecting specialized training samples? Is that even possible for a non-developer? It seems to me this could only work with sources having a visual character that is both consistent and very distinct from the default training sources. I have heard waifu2x does something like this, specializing in Anime - perhaps it is worth training the neural net on a particular video title, or a particular video game, etc. How much processing time would this take?

[feisty2]

https://github.com/IFeelBloated/NNEDI3SF/blob/master/NNEDI3SF.cpp
see for yourself
I'm younger than you and I can do this, sure you got no problem with it

[luquinhas0021]

How old are you, Feisty2?

[feisty2]

18 currently, will be 19 in Dec

[luquinhas0021]

You aren't so younger than me! lol!
Ah, taking the oportunity: I'm developing a resizer algorithm, from zero, from new image function class. If you want help me, mainly with code, you will be welcome. I don't know language codes, but, if you take, I will send you, when I finish, the detailed algorithm.

[feisty2]

guess I'll pass, good luck with your magic "algorithm"

[luquinhas0021]

"Magic algorithm" lollollol
This makes me remember of Magic kernel!

[Groucho2004]

I don't know language codesFrom the context I assume that translates into programming languages?

[feisty2]

lollollol


what's so funny?
how about a web search for "I failed to get the punch line"

[Groucho2004]

what's so funny?
how about a web search for "I failed to get the punch line"You have to cut him some slack, some things don't translate easily into Portuguese.

[luquinhas0021]

Groucho, yes, I don't know translate an algorithm to computer language.
Feisty, "lollollol" is the same of "lol lol lol". I wrote it because I'm happy and very excited. Can't I stay like this?! In Brazil, the people has the habit of write "rsrsrs", means "riso, riso, riso"; in english, "laugh, laugh, laugh".

[feisty2]

You're certainly entitled to that opinion

[luquinhas0021]

Raffrif42, even tough nnedi3 is a good algorithm, I am not a neural network fan, in resampling, because exist n LR images types, n HR image types, and, considering that a LR image is a downsampled version of HR image, exist n types of downscaling algorithms, including the camera processing. Hence, try to train a neural network for this purpose is very non-universal. :)
Answering your questions: you would have to have a LR and HR patches of the images types that you want train nnedi3 for; considering nnedi3's source code is almost out of acess possibility, unless you record it on a CD, it's a little hard you implement something on that.
Maybe implement more neurons or more weights help improve the nnedi3 detail retention amount. Ah, what about L0.5 error implementation?

[raffriff42]

Since Nnedi seems based on Edi (edge directed interpolation), I would guess that the neural network is trained to recognize edges of various types. From Tritical's description quoted above, it sounds like the neural network is perhaps "helping" Edi by tuning the algorithm to optimally handle various recognizable patterns in the source image - e.g. stairstepped edges, soft edges, etc.

For more on Edi, here is an early paper (http://web.archive.org/web/20120102123444/http://neuron2.net/library/nedi.pdf), linked from Tritical's website (http://web.archive.org/web/20070429205747/http://bengal.missouri.edu/~kes25c/).

As described in the technical but readable Image Scaling using Deep Convolutional Neural Networks (http://engineering.flipboard.com/2015/05/scaling-convnets/), the more sophisticated neural networks are trained on a hierarchy of features:The kernels inform us of the presence and location of directional edges. When used in a Neural Network we can learn the appropriate weight settings of the kernels to extract out basic features such as edges, gradients and blurs. If the network is deep with enough convolutional layers it will start learning feature combinations off of the previous layers. The simple building blocks of edges, gradients, and blurs become things like eyes, noses, and hair in later layers!I don't suppose Nnedi3 is quite this sophisticated though, yet.

[luquinhas0021]

Raffrif, looks like the algorithm you send to me is better than nnedi3. Is it implemented somewhere?

[raffriff42]

Since the article I linked to (http://engineering.flipboard.com/2015/05/scaling-convnets/) is hosted at flipboard.com, we can guess that they have an implementation; but as the article mentions, many organizations are working on the problem.Neural Networks are an amazing type model that is able to learn from given data. They have a large breadth of applications and have enjoyed a recent resurgence in popularity in many domains such as computer vision, audio recognition, and natural language processing. Some recent feats include captioning images, playing Atari, aiding self-driving cars, and language translation.
Some of their code is open source (https://www.google.com/search?q=neural+network+image+processing+open+source).

[luquinhas0021]

In image/video super resolution, there's data "dictionary" approach too, which isn't more than put n LR patterns, n HR patterns (Of same frame), "discover" the mathematical models and try to extrapolate them to patterns that looks like with those that are in "dictionary". nnedi3 must works seemly, but only nnedi3, until what I know, TRY to minimize error.

[luquinhas0021]

Originally posted by Tritical

"Which simply means that I repeatedly grab a single training pattern,run the pattern through the model, compute the error in guessing the output value, compute the partial derivative of the error with respect to each weight (term) in the model, and then update the weights.Keep repeating until a minimum in the error surface is reached (progressive stops)."

Compute any type of error requires the error, and error is some relation between original value (HR pixel) and guessed value (Guessed HR pixel). But the most of images that are interpolated by nnedi3 didn't pass through nnedi3 neural network training. So how can nnedi3 compute error?

[wonkey_monkey]

What you've quoted is, I think, tritical talking about training the neural net, not the workings of nnedi3 itself. He ran it against test images, and could test the neural net's predictions against the correct answer. Once the error was minimised, the filter was ready to build and release and be put to work on real images.

[luquinhas0021]

I did understand, however I keep not totally convinced about the neural net efficiency for upscaling. Even nnedi3 can detect all types of edges, generally images edges aren't perfects, because it is affected by noise, halo, ringing and aliasing, that is, there is a lot of mathematical models in order to represent each edge type, each one with your interaction, and a few in order to represent perfect edges. So, I think 1000 frames isn't enough to train neural net. If have enough number...

[foxyshadis]

You keep saying you don't trust neural nets, no matter how it gets explained. Do you care about how it works or not? If not, don't bother everyone with pointless threads.

[luquinhas0021]

I wrote that in previous reply, but didn't appear, no lie:

"Although of this (I don't trust in neural net efficiency for upscaling), I think neural net, if trained, is good for recognize artifacts, because it has less variations than imperfect edges."

[feisty2]

https://en.wikipedia.org/wiki/Artificial_general_intelligence
here, work it out, and you'll have a one for all, or universal solution to everything. (including perfect image upscaling)

edit: wrong link

[luquinhas0021]

Wow, you are well-futuristic! Anyway, thank you so much for link you send me.
I will keep developing my own image function class and define the steps for upscaling (My preference is upscaling by vectorization, vectorization by calculate the ortogonal angle to gradient angle).
If won't work well, I will try strong A.I.

[feisty2]

I will try strong A.I.
I don't know how to respond to that, seriously.
but please tell him when you got him covered.
"hey, Skynet, I'd like to upsize some small and blurry photos, I love peace, please don't nuke the entire human race on this planet"

[luquinhas0021]

You gave me the way!
Strong A.I can be used for add details in image, because it can know the difference between, for example, a nose and a flower, and, based on that, don't add details which, actually, never would be in this things.
Hmmmm, right, weak A.I. is already enough, but is good go to go further!