Deep Denoise Temporal

[anton_foy]

Trying to find nn denoisers in the temporal domain.
Here is one that seems interesting:
https://github.com/clausmichele/ViDeNN
Has anyone here tried it or has any other suggestions of temporal nn denoisers?

Edit: more links
https://github.com/ZhouYiiFeng/TDMS-Net

[poisondeathray]

Trying to find nn denoisers in the temporal domain.
Here is one that seems interesting:
https://github.com/clausmichele/ViDeNN
Has anyone here tried it or has any other suggestions of temporal nn denoisers?

Never tried this one, but it runs ok in Windows

Ok results so far on the test sets , but nothing spectacular

No strength adjustment

Sometimes there is a loss of a frame , I think based on the way it works if there is a scene change, such as in the table tennis set. You have to be careful if you're using PSNR or metrics to align the original vs. results

Has CPU mode fallback , but no tiling options

Older smaller neural net, relatively faster than larger NN. eg
VRT deniosing model about 5-6x larger and more than that many times slower
https://github.com/JingyunLiang/VRT





Edit: more links
https://github.com/ZhouYiiFeng/TDMS-Net

Haven't tried this, but you have to install some baidu application for downloading the model. Not going to do that

[anton_foy]

Thanks for that!
Ok results so far on the test sets , but nothing spectacular

No strength adjustment

Sometimes there is a loss of a frame , I think based on the way it works if there is a scene change, such as in the table tennis set. You have to be careful if you're using PSNR or metrics to align the original vs. results

Has CPU mode fallback , but no tiling options


Ok and no adjustment for temporal radius either I guess. Speed and quality compared to mdegrainN is not close then for either ViDeNN or VRT would you say?

[PatchWorKs]

We strongly suggest to check the User-friendly, Fast, Self-Supervised Image Denoising for All project:
https://github.com/royerlab/aydin/

[poisondeathray]

Thanks for that!


Ok and no adjustment for temporal radius either I guess. Speed and quality compared to mdegrainN is not close then for either ViDeNN or VRT would you say?


No adjustments for ViDeNN (other than model choice) , it's slower than mdegrain derivatives.

VRT has adjustments for temporal frame : width: height tiling, otherwise you'd need massive amounts of GPU memory, and strength 0-50 . Speed is very, very, very, very slow. It's got to be the one of slowest NN processing algorithms (not just for denoising)

The resulting quality depends on the noise type for the pretrained models. Both tend to do better on gaussian type noise, not on stuff like large grain, because that's what they were trained on. If you look at the table tennis example in ViDeNN, mdegrain derivatives typically won't do very well on that type of noise, but ViDeNN will do ok, and scunet will do great (even though it's spatial, there will be no temporal inconsistencies in that particular case)

[anton_foy]

@PatchWorks
Thanks looks interesting, it says also that it is spatiotemporal. Must look into, but it only seems to take still sequencies.

@Poisondeathray
Interesting that scunet doesn't leave those temporal inconsistencies yet probably because high ISO and low light footage has too little information especially in the darker parts while the tennis clip has added noise that does not distort in that way.

Edit:
Using Davinci Resolve's NR alot at work and while it smears (motionblurs) alot it has a great ability at low settings to remove lighter temporal fluctuations and blotches from bad compression.

[poisondeathray]

@Poisondeathray
Interesting that scunet doesn't leave those temporal inconsistencies yet probably because high ISO and low light footage has too little information especially in the darker parts while the tennis clip has added noise that does not distort in that way.

Scunet can leave temporal inconsistencies - in general. All spatial only algorithms can.

But on that specific table tennis sequence it does not, and you don't even need to add other smoothing afterwards. ViDNN OTOH does leave some temporal fluctuations on that sequence even though it's temporal. Scunet retains more detail and is cleaner, more temporally stable than ViDNN on that sequence . I mention the table tennis sequence because that's the highlight presented for ViDNN

[anton_foy]

Scunet can leave temporal inconsistencies - in general. All spatial only algorithms can.

But on that specific table tennis sequence it does not, and you don't even need to add other smoothing afterwards. ViDNN OTOH does leave some temporal fluctuations on that sequence even though it's temporal. Scunet retains more detail and is cleaner, more temporally stable than ViDNN on that sequence . I mention the table tennis sequence because that's the highlight presented for ViDNN

Scunet is also in the slower category you would say?

[poisondeathray]

Scunet is also in the slower category you would say?

Yes - All the NN algorithms are several orders slower than mvtools2 derivatives or typical avs denoisers. The temporal ones like VRT can be exponentially slower unless you have a quadro with 48GB GPU memory

Scunet is only good for specific types of noise with the pretrained models, but it falls in the "remarkable" category. Very useful tool, sometimes as a prefilter for some avs/vpy filters . It makes you wonder how it would behave with custom training sets

[anton_foy]

I found this clip where they are using what they call "Overlap Loss" for temporal consistency:

https://youtu.be/LA747HTukTQ

[poisondeathray]

I found this clip where they are using what they call "Overlap Loss" for temporal consistency:

https://youtu.be/LA747HTukTQ

Looks interesting .

Only paper link , no test code published that I could find .

Might have to wait a few months, sometimes people eventually publish an implementation of a paper on github

[anton_foy]

Looks interesting .

Only paper link , no test code published that I could find .

Might have to wait a few months, sometimes people eventually publish an implementation of a paper on github

Yes I could only find this (https://wilson1yan.github.io/teco/)

and test code (https://github.com/wilson1yan/teco)

but it is only for temporal consistencies and I have no idea how useful it is for real footage. Edit: the kinetics videos used there are broken.

Edit: and this (https://github.com/skycrapers/TecoGAN-PyTorch) and this (https://github.com/thunil/TecoGAN)
but it is for superres.

[StainlessS]

I know little to nothing of A.I. generative techniques, but thought that this link to a Tom'sHARDWARE article,
posted via CodeProject might be of interest to some, might even give hints to potential problems in other A.I. disciplines.
Generative AI Goes 'MAD' When Trained on AI-Created Data Over Five Times:- https://www.tomshardware.com/news/generative-ai-goes-mad-when-trained-on-artificial-data-over-five-times
Generative AI goes "MAD" after five training iterations on artificial outputs.

Model Autophagy Disorder isn't confirmed to affect all AI models, but the researchers did verify it against autoencoders, Gaussian mixture models, and large language models.

It just so happens that all of these types of models that can "go MAD" have been widespread and operating for a while now: autoencoders can handle things such as popularity prediction (in things such as a social media app's algorithm), image compression, image denoising, and image generation; and gaussian mixture models are used for density estimation, clustering, and image segmentation purposes, which makes them particularly useful for statistical and data sciences.


This research provides a way to peer into the black box of AI development. And it shreds any hope that we'd found an endless fountain of data from making a hamster wheel out of certain AI models: feeding it data, and then feeding its own data back into it, in order to generate more data that's then fed back again.

This could be an issue for currently-existing models and applications of these models: if a model that's achieved commercial use has, in fact, been trained on its own outputs, then that model has likely regressed toward its mean (remember it takes around five cycles of input-output for that to manifest). And if that model has regressed towards its mean, then it's been biased in some way, shape, or form, as it doesn't consider the data that would naturally be in the minority. Algorithmic bigotry, if you will.

[ClickMe]
https://i.postimg.cc/r0vtdmk9/AI-Gen-Flaw.jpg (https://postimg.cc/r0vtdmk9)

(a sort of 'A.I. ringing')
Perhaps pertinent for eg, Super Resolution generative techniques.

[anton_foy]

I know little to nothing of A.I. generative techniques, but thought that this link to a Tom'sHARDWARE article,
posted via CodeProject might be of interest to some, might even give hints to potential problems in other A.I. disciplines.
Generative AI Goes 'MAD' When Trained on AI-Created Data Over Five Times:- https://www.tomshardware.com/news/generative-ai-goes-mad-when-trained-on-artificial-data-over-five-times





[ClickMe]
https://i.postimg.cc/r0vtdmk9/AI-Gen-Flaw.jpg (https://postimg.cc/r0vtdmk9)

(a sort of 'A.I. ringing')
Perhaps pertinent for eg, Super Resolution generative techniques.

Yes I have seen this over at the Vapoursynth section with some super res esrgan stuff. But my main interest is the temporal consitency with denoising. I got Chat GPT to write a simple avs+ script that works ok actually stabilizing temporally and spatial denoise. Just for fun that is besides the point of this thread :)

[PatchWorKs]

Sorry for the delay in reply, we're quite busy.

@PatchWorks
Thanks looks interesting, it says also that it is spatiotemporal. Must look into, but it only seems to take still sequencies.

Here's an interesting reply about by aydin's author (https://github.com/AhmetCanSolak) @ my GitHub "issue" (https://github.com/royerlab/aydin/issues/292):
I want to clarify that Aydin works on spatio-temporal data(TZYX) when all four dimensions are available and can do TYX as well. Aydin doesn't read video file formats directly though, if you can save it as a file format that is supported then you can denoise movies too. With biological live imaging datasets, we use Aydin for denoising movies on daily basis.

Hope that helps.

[anton_foy]

Sorry for the delay in reply, we're quite busy.



Here's an interesting reply about by aydin's author (https://github.com/AhmetCanSolak) @ my GitHub "issue" (https://github.com/royerlab/aydin/issues/292):

Hope that helps.

Thanks I will try on my mac I have problems with python on my PC.

[anton_foy]

Ok this is a shader but it seems interesting and I wonder if it can be used to reduce noise instead of just AA. Only previous frame for motion vectors though but it looks quite stable and the nvidia paper linked in the dropboxlink mentions reduced noise and reduced ghosting. Maybe possible to tweak or replace the AA part with spatial denoising (since AA is spatial in it self)?

Link: https://www.shadertoy.com/view/MscSD7

[DTL]

It looks current cameras have enough effective internal denoisers (and/or pro shooting conditions provide enough light) so development of good denoisers for processing after shooting close to stopped.
So developers of 'neural networks' make very little progress in this way. In previous decades when we had a big market for consumer video cameras of medium and poor (cheap) quality - there was higher demand on post-shooting denoise (but PCs were slow). So it looks the era of best development of denoise solutions is spreaded in some years of ending of consumer video cameras and beginning of not very slow PCs at home.

Currently even if some business will invest in development of denoise solution - there is no market for it to return the investment (like was in the old times of Neat Video products for consumer market). And pro denoise solutions for video cameras are closed in the production of video cameras for both pro and smartphone video cameras. The RAW shooting 'video modes' of consumer photo cameras may expect good lighting conditions and not shipped with good RAW denoise post-processing software. And typical large-sensor consumer photo-cameras do not have enough processing hardware resources onboard to perform good temporal denoise.

[anton_foy]

Another interesting thing perhaps:
https://sreyas-mohan.github.io/udvd/
https://github.com/sreyas-mohan/udvd

Up to 5 frames temporal radius.

[poisondeathray]

Another interesting thing perhaps:
https://sreyas-mohan.github.io/udvd/
https://github.com/sreyas-mohan/udvd

Up to 5 frames temporal radius.

There are PSNR rankings for many of these on commonly used test datasets (DAVIS , and set8) on the paperswithcode website

udvd gets about 2-3db lower than VRT/RVRT and ReVid . PSNR isn't the best metric, and certainly doesn't measure any temporal component (unlike VMAF), but that delta is large enough that any metric will show similar trend. ReVid is the top on every dataset there, but there is no public code or model available yet .

https://paperswithcode.com/task/video-denoising

Some of the older papers have other comparisons like Neat Video for the dB included in the results chart, but it's not clear how Neat Video was used, or if it was used properly. Neat does surprisingly poorly, I suspect not used correctly. Might be fun to run current Neat video or some avs/vpy denoisers on one of the known datasets to compare
https://github.com/m-tassano/dvdnet#psnrs-set8-testset

I think currently VRT/RVRT are probably the best "working" ones, with configurable settings, ... but soooooooo slow . And you really need to train your own models for the best results, unless your sources are similar to DAVIS

[anton_foy]

Thanks yeah I just threw it out there since it was temporal but still hoping for the TempForm with overlap loss to be available soon. For now I have tried vs-Scunet and Dpir (only in HD) on noisy uhd but I was disappointed and its just too slow even with my new setup. Mdegrain became a real joy though.

[PatchWorKs]

For those who are interested in, we've collected many "AI" (= GANs-based) denoisers here: https://github.com/forart/HyMPS/blob/main/AIvideo.md#denoisers-

[DTL]

At first it is nice to see such active new ideas production for denoising. At least after 2010 year. But after next look I see the marks 'last commits 2..4+ years ago' after all listed projects. All ideas do not go into finally working implementations for current public motion pictures data processing environments like Avisynth/Vapoursynth or ffmpeg.

It looks like most of these 'tech demos' are in the 'waiting for investor' state ? Or were developed only to make publication at IEEE/ICIP and put these tech demo sources at github for hosting ?

Unfortunately sometimes a project only designed to make publication and rating increase to authors at the graduation but not to make a practically useful product. It is typically started with 'based on the IEEE publication...' and ended in the 'working math demo in matlab'. We need some more steps to make 'engineers build' as compiled and optimized for performance software for today's end-users CPUs and GPUs to test and use in real encodings. These steps require practical programmers resources and looks like original developers of the new ideas do not have any profit from finishing some practically useful software.

And you really need to train your own models for the best results

It may be good for 'offline' moving pictures production. So the production team can make shots of talents with good lighting conditions and low camera noise to 'train' models and next apply such processing to noised shots in bad lighting and high camera noise.

But for 'blind' denoising without previously known clean and perfect image data it may be not applicable.

This processing workflow is close to 2 steps:
1. Create library of clean image blocks
2. Detects the most likely looking image block in the current noised frame and replaces it with a clean view from the library.

For 'blind' denoise of long movies (of standard runtime 1 hr 30 min or more) it is possible to 'train' model on the all presented views of talent (any subject) in all frames and make 'library of clean views'.
And at the second pass replace bad noised image blocks with transformed clean blocks (applying all required and supported transforms of lighting and geometry). Implementation of such 2 pass denoising can be also 'very slow'.

It means finally expanding 'temporal size' of denoiser to the total available movie runtime (all frames). It is possible to model this denoiser using current mvtools project sources too. But for temporal 'cleaned blocks' storage may require lots of HDD space (max expected is RGB24 uncompressed frames size). Also running MAnalyse with tr about total_num_frames/2 may be too slow. So this approach requires reduction of block search steps. Currently to make implementation of MAnalyse+MDegrain more simple we still have lots of redundant search operations (this makes quality better but takes more computing resources). Also the quality of noise reduction will be more variable over the cutscenes and each subject - the more time it appears in the total movie the better its views are cleaned. Rare subjects like 1..2 frames views for total movie can not be denoised in such a way.