Are there any mathematical papers abut NNEDI3?

[FranceBB]

Hi there,
I just begun studying Soft Computing Systems at university as part of my Computer Science Engineering degree and there's a whole part about the different kind of neural networks with the difference between RBFN (Radial Basis Function Networks) and MPL (Multi-Layer Perceptron) and how they work, their training sets, how one should be trained, what to do, common issues like when using the gradient descend you can be stuck on a local minimum and never reach a global minimum, the different activation functions like ReLU (Rectified Linear Unit), Bell Curves etc.
It's very interesting.
Anyway, we have several assignments before the final exam and I'm completing them all, but the very last one is about bringing a real life neural network and write everything like a RAD (Requirement Analysis Document) about it and code it, so that we can be admitted to the final exam.
Long story short: I'm now much more interested in NNEDI3 'cause I've been using it for years but now I have the mathematical knowledge to understand how it works. So... is there any official document depicting how it was made, which kind of function it uses, which kind of training set and training method have been used etc from a mathematical point of view? It would be enlightening and definitely interesting to play around. I know that it's a kinda old Neural Network and that it has probably been coded and trained in a way that is not used anymore, but I'd like to find out the inner workings of it. :)

So... do you have a place where I can look?
And if nothing is documented, do you know any other well documented neural network I can look at?

(Feisty? Suggestions on your side? We're about the same age, so I'm pretty sure you've been taking this kind of exam as well)

[feisty2]

there's no documentation or whatsoever for the model of nnedi3, the training set and various hyper parameters and the adopted augmentation methods are also completely unknown, only the binary file for the converged model is ever released.

and other things that you've mentioned (RBF, fully connected aka MLP) are also rarely used nowadays. it is also extremely unlikely that you land on a local minimum in a very high dimensional space like the parameter space for modern neural nets. you're much more likely to hit a saddle point or a tip of the extremely rugged loss surface of a deep network (https://papers.nips.cc/paper/2018/file/a41b3bb3e6b050b6c9067c67f663b915-Paper.pdf). anyways, all this optimization nonsense doesn't really matter to modern neural nets, they are so large (we're talking about 100+ billion parameters) that it is guaranteed that they will always converge to the global minimum (zero loss). a sufficiently wide overparameterized network always converges to zero loss (http://proceedings.mlr.press/v97/du19c/du19c.pdf).

then again you're probably not that interested in all that theoretical stuff, if you simply want neural nets for super resolution reconstruction, EDSR (https://openaccess.thecvf.com/content_cvpr_2017_workshops/w12/papers/Lim_Enhanced_Deep_Residual_CVPR_2017_paper.pdf) would be a good start, the baseline EDSR model already gives you way superior reconstruction quality than nnedi3.

now I have the mathematical knowledge to understand how it works.
I doubt it.

I'm pretty sure you've been taking this kind of exam as well
we PhD students don't take this kind of exam.

[FranceBB]

there's no documentation or whatsoever for the model of nnedi3, the training set and various hyper parameters and the adopted augmentation methods are also completely unknown, only the binary file for the converged model is ever released.


I see... that's a shame...



and other things that you've mentioned are also rarely used nowadays.


Ah really? I think our books are quite outdated then... :(



it is also extremely unlikely that you land on a local minimum in a very high dimensional space like the parameter space for modern neural nets. you're much more likely to hit a saddle point or a tip of the extremely rugged loss surface of a deep network (https://papers.nips.cc/paper/2018/file/a41b3bb3e6b050b6c9067c67f663b915-Paper.pdf).


Well actually yes, I see what you mean now: I'm reading the paper you linked, thanks for the reference. ;)



anyways, all this optimization nonsense doesn't really matter to modern neural nets, they are so large (we're talking about 100+ billion parameters) that it is guaranteed that they will always converge to the global minimum (zero loss). a sufficiently wide overparameterized network always converges to zero loss (http://proceedings.mlr.press/v97/du19c/du19c.pdf).


This is unexpected as I didn't really think they were always going to converge to the global minimum...
I mean, that's quite something!


EDSR (https://openaccess.thecvf.com/content_cvpr_2017_workshops/w12/papers/Lim_Enhanced_Deep_Residual_CVPR_2017_paper.pdf) would be a good start


Thanks a lot for the PDF, that's pretty much the first step of what I was looking for.



I doubt it.


Subtle as always, but I didn't pass the exam yet so I guess I deserved it as I said it too early ehehehe


we PhD students don't take this kind of exam.

Really? That's unexpected...


Anyway, thanks a lot for the links.
It's gonna be an interesting evening! :D