Deep Denoise Temporal

[anton_foy]

Trying to find nn denoisers in the temporal domain.
Here is one that seems interesting:
https://github.com/clausmichele/ViDeNN
Has anyone here tried it or has any other suggestions of temporal nn denoisers?

Edit: more links
https://github.com/ZhouYiiFeng/TDMS-Net

[poisondeathray]

Originally Posted by anton_foy

Trying to find nn denoisers in the temporal domain.
Here is one that seems interesting:
https://github.com/clausmichele/ViDeNN
Has anyone here tried it or has any other suggestions of temporal nn denoisers?

Never tried this one, but it runs ok in Windows

Ok results so far on the test sets , but nothing spectacular

No strength adjustment

Sometimes there is a loss of a frame , I think based on the way it works if there is a scene change, such as in the table tennis set. You have to be careful if you're using PSNR or metrics to align the original vs. results

Has CPU mode fallback , but no tiling options

Older smaller neural net, relatively faster than larger NN. eg
VRT deniosing model about 5-6x larger and more than that many times slower
https://github.com/JingyunLiang/VRT

Edit: more links
https://github.com/ZhouYiiFeng/TDMS-Net

Haven't tried this, but you have to install some baidu application for downloading the model. Not going to do that

[anton_foy]

Thanks for that!

Ok results so far on the test sets , but nothing spectacular

No strength adjustment

Sometimes there is a loss of a frame , I think based on the way it works if there is a scene change, such as in the table tennis set. You have to be careful if you're using PSNR or metrics to align the original vs. results

Has CPU mode fallback , but no tiling options

Ok and no adjustment for temporal radius either I guess. Speed and quality compared to mdegrainN is not close then for either ViDeNN or VRT would you say?

[PatchWorKs]

We strongly suggest to check the User-friendly, Fast, Self-Supervised Image Denoising for All project:
https://github.com/royerlab/aydin/

[poisondeathray]

Originally Posted by anton_foy

Thanks for that!


Ok and no adjustment for temporal radius either I guess. Speed and quality compared to mdegrainN is not close then for either ViDeNN or VRT would you say?

No adjustments for ViDeNN (other than model choice) , it's slower than mdegrain derivatives.

VRT has adjustments for temporal frame : width: height tiling, otherwise you'd need massive amounts of GPU memory, and strength 0-50 . Speed is very, very, very, very slow. It's got to be the one of slowest NN processing algorithms (not just for denoising)

The resulting quality depends on the noise type for the pretrained models. Both tend to do better on gaussian type noise, not on stuff like large grain, because that's what they were trained on. If you look at the table tennis example in ViDeNN, mdegrain derivatives typically won't do very well on that type of noise, but ViDeNN will do ok, and scunet will do great (even though it's spatial, there will be no temporal inconsistencies in that particular case)

[anton_foy]

@PatchWorks
Thanks looks interesting, it says also that it is spatiotemporal. Must look into, but it only seems to take still sequencies.

@Poisondeathray
Interesting that scunet doesn't leave those temporal inconsistencies yet probably because high ISO and low light footage has too little information especially in the darker parts while the tennis clip has added noise that does not distort in that way.

Edit:
Using Davinci Resolve's NR alot at work and while it smears (motionblurs) alot it has a great ability at low settings to remove lighter temporal fluctuations and blotches from bad compression.

[poisondeathray]

Originally Posted by anton_foy

@Poisondeathray
Interesting that scunet doesn't leave those temporal inconsistencies yet probably because high ISO and low light footage has too little information especially in the darker parts while the tennis clip has added noise that does not distort in that way.

Scunet can leave temporal inconsistencies - in general. All spatial only algorithms can.

But on that specific table tennis sequence it does not, and you don't even need to add other smoothing afterwards. ViDNN OTOH does leave some temporal fluctuations on that sequence even though it's temporal. Scunet retains more detail and is cleaner, more temporally stable than ViDNN on that sequence . I mention the table tennis sequence because that's the highlight presented for ViDNN

[anton_foy]

Originally Posted by poisondeathray

Scunet can leave temporal inconsistencies - in general. All spatial only algorithms can.

But on that specific table tennis sequence it does not, and you don't even need to add other smoothing afterwards. ViDNN OTOH does leave some temporal fluctuations on that sequence even though it's temporal. Scunet retains more detail and is cleaner, more temporally stable than ViDNN on that sequence . I mention the table tennis sequence because that's the highlight presented for ViDNN

Scunet is also in the slower category you would say?

[poisondeathray]

Originally Posted by anton_foy

Scunet is also in the slower category you would say?

Yes - All the NN algorithms are several orders slower than mvtools2 derivatives or typical avs denoisers. The temporal ones like VRT can be exponentially slower unless you have a quadro with 48GB GPU memory

Scunet is only good for specific types of noise with the pretrained models, but it falls in the "remarkable" category. Very useful tool, sometimes as a prefilter for some avs/vpy filters . It makes you wonder how it would behave with custom training sets

[anton_foy]

I found this clip where they are using what they call "Overlap Loss" for temporal consistency:

https://youtu.be/LA747HTukTQ

[poisondeathray]

Originally Posted by anton_foy

I found this clip where they are using what they call "Overlap Loss" for temporal consistency:

https://youtu.be/LA747HTukTQ

Looks interesting .

Only paper link , no test code published that I could find .

Might have to wait a few months, sometimes people eventually publish an implementation of a paper on github

[anton_foy]

Originally Posted by poisondeathray

Looks interesting .

Only paper link , no test code published that I could find .

Might have to wait a few months, sometimes people eventually publish an implementation of a paper on github

Yes I could only find this

and test code

but it is only for temporal consistencies and I have no idea how useful it is for real footage. Edit: the kinetics videos used there are broken.

Edit: and this and this
but it is for superres.

[StainlessS]

I know little to nothing of A.I. generative techniques, but thought that this link to a Tom'sHARDWARE article,
posted via CodeProject might be of interest to some, might even give hints to potential problems in other A.I. disciplines.
Generative AI Goes 'MAD' When Trained on AI-Created Data Over Five Times:- https://www.tomshardware.com/news/ge...ver-five-times

Generative AI goes "MAD" after five training iterations on artificial outputs.

Model Autophagy Disorder isn't confirmed to affect all AI models, but the researchers did verify it against autoencoders, Gaussian mixture models, and large language models.

It just so happens that all of these types of models that can "go MAD" have been widespread and operating for a while now: autoencoders can handle things such as popularity prediction (in things such as a social media app's algorithm), image compression, image denoising, and image generation; and gaussian mixture models are used for density estimation, clustering, and image segmentation purposes, which makes them particularly useful for statistical and data sciences.

This research provides a way to peer into the black box of AI development. And it shreds any hope that we'd found an endless fountain of data from making a hamster wheel out of certain AI models: feeding it data, and then feeding its own data back into it, in order to generate more data that's then fed back again.

This could be an issue for currently-existing models and applications of these models: if a model that's achieved commercial use has, in fact, been trained on its own outputs, then that model has likely regressed toward its mean (remember it takes around five cycles of input-output for that to manifest). And if that model has regressed towards its mean, then it's been biased in some way, shape, or form, as it doesn't consider the data that would naturally be in the minority. Algorithmic bigotry, if you will.

[ClickMe]


(a sort of 'A.I. ringing')
Perhaps pertinent for eg, Super Resolution generative techniques.

[anton_foy]

Originally Posted by StainlessS

I know little to nothing of A.I. generative techniques, but thought that this link to a Tom'sHARDWARE article,
posted via CodeProject might be of interest to some, might even give hints to potential problems in other A.I. disciplines.
Generative AI Goes 'MAD' When Trained on AI-Created Data Over Five Times:- https://www.tomshardware.com/news/ge...ver-five-times





[ClickMe]


(a sort of 'A.I. ringing')
Perhaps pertinent for eg, Super Resolution generative techniques.

Yes I have seen this over at the Vapoursynth section with some super res esrgan stuff. But my main interest is the temporal consitency with denoising. I got Chat GPT to write a simple avs+ script that works ok actually stabilizing temporally and spatial denoise. Just for fun that is besides the point of this thread

[PatchWorKs]

Sorry for the delay in reply, we're quite busy.

Originally Posted by anton_foy

@PatchWorks
Thanks looks interesting, it says also that it is spatiotemporal. Must look into, but it only seems to take still sequencies.

Here's an interesting reply about by aydin's author @ my GitHub "issue":

Originally Posted by Ahmet Can Solak

I want to clarify that Aydin works on spatio-temporal data(TZYX) when all four dimensions are available and can do TYX as well. Aydin doesn't read video file formats directly though, if you can save it as a file format that is supported then you can denoise movies too. With biological live imaging datasets, we use Aydin for denoising movies on daily basis.

Hope that helps.

[anton_foy]

Originally Posted by PatchWorKs

Sorry for the delay in reply, we're quite busy.



Here's an interesting reply about by aydin's author @ my GitHub "issue":

Hope that helps.

Thanks I will try on my mac I have problems with python on my PC.

[anton_foy]

Ok this is a shader but it seems interesting and I wonder if it can be used to reduce noise instead of just AA. Only previous frame for motion vectors though but it looks quite stable and the nvidia paper linked in the dropboxlink mentions reduced noise and reduced ghosting. Maybe possible to tweak or replace the AA part with spatial denoising (since AA is spatial in it self)?

Link: https://www.shadertoy.com/view/MscSD7

[DTL]

It looks current cameras have enough effective internal denoisers (and/or pro shooting conditions provide enough light) so development of good denoisers for processing after shooting close to stopped.
So developers of 'neural networks' make very little progress in this way. In previous decades when we had a big market for consumer video cameras of medium and poor (cheap) quality - there was higher demand on post-shooting denoise (but PCs were slow). So it looks the era of best development of denoise solutions is spreaded in some years of ending of consumer video cameras and beginning of not very slow PCs at home.

Currently even if some business will invest in development of denoise solution - there is no market for it to return the investment (like was in the old times of Neat Video products for consumer market). And pro denoise solutions for video cameras are closed in the production of video cameras for both pro and smartphone video cameras. The RAW shooting 'video modes' of consumer photo cameras may expect good lighting conditions and not shipped with good RAW denoise post-processing software. And typical large-sensor consumer photo-cameras do not have enough processing hardware resources onboard to perform good temporal denoise.

[anton_foy]

Another interesting thing perhaps:
https://sreyas-mohan.github.io/udvd/
https://github.com/sreyas-mohan/udvd

Up to 5 frames temporal radius.

[poisondeathray]

Originally Posted by anton_foy

Another interesting thing perhaps:
https://sreyas-mohan.github.io/udvd/
https://github.com/sreyas-mohan/udvd

Up to 5 frames temporal radius.

There are PSNR rankings for many of these on commonly used test datasets (DAVIS , and set8) on the paperswithcode website

udvd gets about 2-3db lower than VRT/RVRT and ReVid . PSNR isn't the best metric, and certainly doesn't measure any temporal component (unlike VMAF), but that delta is large enough that any metric will show similar trend. ReVid is the top on every dataset there, but there is no public code or model available yet .

https://paperswithcode.com/task/video-denoising

Some of the older papers have other comparisons like Neat Video for the dB included in the results chart, but it's not clear how Neat Video was used, or if it was used properly. Neat does surprisingly poorly, I suspect not used correctly. Might be fun to run current Neat video or some avs/vpy denoisers on one of the known datasets to compare
https://github.com/m-tassano/dvdnet#psnrs-set8-testset

I think currently VRT/RVRT are probably the best "working" ones, with configurable settings, ... but soooooooo slow . And you really need to train your own models for the best results, unless your sources are similar to DAVIS