A.I. Denoising?!

[asarian]

I have a distinct feeling I may have to ask this question again, some time in the future. :) But are 'we' anywhere like this yet? NVIDIA's Image Restoration AI: Almost Perfect (https://www.youtube.com/watch?v=P0fMwA3X5KI) Imagine the wicked denoising one could do with that!

So, commercially, it would seem ppl are already making huge progress with this. Hope this technique will hit this community ere long too!

[amichaelt]

The results in that video look terrible. Any fine detail is smudged out and people look like their skin is made of wax. I can only see that be used by studios to create even worse looking home video releases than the DNRfests we already get from time-to-time.

[asarian]

The results in that video look terrible. Any fine detail is smudged out and people look like their skin is made of wax. I can only see that be used by studios to create even worse looking home video releases than the DNRfests we already get from time-to-time.

Ha! I'm sure it was just a proof-of-concept: you know, extreme added noise, and then extremely denoised -- just to see how far they could take it. Like with any other denoiser, there's no law says you need to crank it up all the way. I believe this is going to be the future, though.

Also, they claim to be able to remove tons of added text from a vid too; kinda like a logo-remover, only done with A.I., and done exceptionally well.

[zorr]

I read the paper (https://arxiv.org/abs/1803.04189) and it's interesting because it doesn't need any "reference" noise-free data. It only needs two or more noisy variants of the same image to learn. That would work pretty well with video if you use MVTools to recreate the current frame from the neighbour frames. So for best results you could feed it your whole video and it would learn the noise from that. The learning took 13 hours using 5000 256x256 images on Tesla P100 so it's within possibility, at least it will be in the near future as the GPUs get even faster.

[EDIT] the code is released at GitHub (https://github.com/NVlabs/noise2noise) and it's using Python and TensorFlow. WolframRhodium, are you listening? ;)

[WolframRhodium]

[EDIT] the code is released at GitHub (https://github.com/NVlabs/noise2noise) and it's using Python and TensorFlow. WolframRhodium, are you listening? ;)

I am maintaining a repo named Super-Resolution-Zoo (https://github.com/WolframRhodium/Super-Resolution-Zoo) which includes mainly super-resolution models including waifu2x but also some denoising models like official DnCNN, IRCNN, CBDNet and an unofficial noise2noise model.

I have not included any TensorFlow models since the framework is too complicated. But if people really want to run TF models, I can provide some help.

Anyway, recent advances on CNN-based image denoising techniques includes CBDNet (https://arxiv.org/abs/1807.04686), Unprocessing Images for Learned Raw Denoising (https://arxiv.org/abs/1811.11127), Self-Supervised Deep Image Denoising (https://arxiv.org/abs/1901.10277), Noise2Self (https://arxiv.org/abs/1901.11365), etc. But for normal video processing tasks, I still prefer KNLMeansCL and (V-)BM3D over them.

[asarian]

Thanks for the feedback, guys.

[zorr]

Anyway, recent advances on CNN-based image denoising techniques includes CBDNet (https://arxiv.org/abs/1807.04686), Unprocessing Images for Learned Raw Denoising (https://arxiv.org/abs/1811.11127), Self-Supervised Deep Image Denoising (https://arxiv.org/abs/1901.10277), Noise2Self (https://arxiv.org/abs/1901.11365), etc. But for normal video processing tasks, I still prefer KNLMeansCL and (V-)BM3D over them.

Thanks, interesting links. Can you elaborate on why you prefer KNLMeansCL and (V-)BM3D? Is it because they are more reliable/consistent or have better quality/performance ratio? Based on the papers those neural network based denoisers should have better quality.

Do you have a tutorial on how to run these NN denoisers? I'm interested in trying them on my VHS videos which also have jpeg artifacts (they were captured with M-JPEG codec in the 90's with quite high compression ratio). I tried Oyster and it seems to do wonders, maybe these NN denoisers can perform as well.

[ChaosKing]

You should give https://sites.google.com/site/amdctfilter/ also a try. Should be much faster than Oyster.

[WolframRhodium]

Can you elaborate on why you prefer KNLMeansCL and (V-)BM3D? Is it because they are more reliable/consistent or have better quality/performance ratio? Based on the papers those neural network based denoisers should have better quality.

Yes, I think KNLMeansCL and (V-)BM3D are superior in terms of texture preservation and computation speed. NN-based models usually remove details and they are not stable enough.

Do you have a tutorial on how to run these NN denoisers? I'm interested in trying them on my VHS videos which also have jpeg artifacts (they were captured with M-JPEG codec in the 90's with quite high compression ratio). I tried Oyster and it seems to do wonders, maybe these NN denoisers can perform as well.

You should first install MXNet (https://mxnet.apache.org/versions/master/install/index.html).

There are two ways to run these models in VapourSynth.
One way is through vs_mxnet@kice (https://github.com/kice/vs_mxnet). You can simply use the plugin, and I also have written a wrapper function named muvsfunc.super_resolution() (https://github.com/WolframRhodium/muvsfunc/blob/116aa4a8db0403d44450e9115504e585453f7e3d/muvsfunc.py#L4198)[/URL] for it. This method always runs faster than the other method, which runs these models in Python directly using [URL="https://github.com/WolframRhodium/muvsfunc/blob/116aa4a8db0403d44450e9115504e585453f7e3d/Collections/muvsfunc_numpy.py#L1507"]muvsfunc_numpy.super_resolution() (https://github.com/WolframRhodium/muvsfunc/blob/116aa4a8db0403d44450e9115504e585453f7e3d/muvsfunc.py#L4198). You can check an example (https://github.com/WolframRhodium/muvsfunc/blob/master/Collections/examples/super_resolution_mxnet.vpy) for detailed usage. (Note that although automatic color space conversion exists, it may be incorrect in some cases and you may do it yourself. The output of these filters are always in 32-bit float format.)

You may also find vs_mxnet_helper@AlphaAtlas (https://github.com/AlphaAtlas/vs_mxnet_helper) useful, which provides additional instructions and scripts.