spectral adaptive denoising with dfttest

raffriff42 · 2nd January 2016, 17:45

Quote:

Originally Posted by geometer

Is there a spectral adaptive denoiser for video, as they have in audio engineering? This would also use a second of (almost) black and another of white signal (lead-in, camera blend up or something) as a noise footprint.

Maybe. I've recently been experimenting with dfttest, and there is support for doing what you describe. I have not seen a working example online, so I will post my way of doing it, which includes a couple of hacks to work around some strange error messages.

Here is the wiki page; use it look up anything I don't explain well enough.

* Find a "test area" of your video:
-- it must be flat (no detail except gray level),
-- at least 12x12 pixels (assuming default sbsize or spatial window),
-- and at least 5(?) frames long (assuming default tbsize or temporal window)
(not sure about that last point, as my temporal window was 1)

* Crop the video so this test area is at 0, 0 (top left).
(haha! I see I am not the first to use this hack)

Code:

Crop(<top>, <left>, 0, 0) ## area under test should be at 0, 0 (top left)

* Convert video to YV24 and copy Y plane to U
-- I get the error "GetPlaneHeightSubsampling called with unsupported plane" if I try to sample plane 0 (Y) directly (and no, I'm not off by one)

Code:

ConvertToYV24
Y = Last.ConvertToY8
YToUV(Y, Y, Y) ## copy plane 0 (Y) to plane 1 (U)

* Create a new text file like this;
-- the only variable elements are the frame numbers you select; keep ",1,0,0" (Plane 1, top left) as is.
-- I am not sure if sequential numbers as shown are needed, or even desirable. Experiment.

Code:

#(comments allowed in this file)
#syntax: frame number ',' plane ',' top ',' left (in some scale that is not pixels)
191,1,0,0
192,1,0,0
193,1,0,0
194,1,0,0
195,1,0,0
196,1,0,0

* Call dfttest with nfile="<name of file created above>".

Code:

dfttest(tbsize=3, sbsize=8, sosize=6, nfile="<filename>")

* A file will be written in in your working directory called "noise_spectrum-date_string.txt";
it will look something like this (if your video is very bad)

(the numbers in blue are explained below)

Code:

     0.000     61.629     24.064      3.206      0.737 
    59.529     39.625     33.536     11.348      0.856 
    82.657     64.773     73.869     20.538      1.386 
     2.859      8.859     16.259      3.859      0.477 
     0.208      0.123      0.116      0.114      0.053 
     2.859      4.099     11.165      3.629      0.477 
    82.657     59.948     47.890     16.551      1.386 
    59.529     92.776     43.089      8.525      0.856 


# avg power = 24.259323

* I was working with a temporal window of 1 and spatial window of 5; if you use the default values you will see a lot more numbers (aka frequency coefficients) in this file, tbsize*sbsize*(sbsize/2+1) of them to be precise, where tbsize=temporal window and sbsize=spatial window)

* Now call dfttest again with sfile="<name of noise spectrum file>".

Code:

dfttest(tbsize=3, sbsize=8, sosize=6, sfile="<filename>")

* That's how it's supposed to work. In my experiments, for maximum strength, I scaled all the numbers using a spreadsheet, so the largest value came to about 9000.
-- (dfttest noise reduction strength sigma is log-scale-ish: 10 is mild, 100 is stronger, 1000 pretty strong, and >9000 is very strong indeed)

* If the result is hideously blurry, try setting the numbers shown in blue above (representing the lowest spatial frequencies) to a lower number.

* As a dfttest newbie, I'm not sure if there is a way to control overall strength when using sfile= without scaling the numbers in a spreadsheet as I did. f0beta might do it to some extent.

I'm not sure how to do chroma noise reduction without repeating this process for each color plane.

Apologies for any factual errors here; I hope someone will come along and correct them for me.

geometer · 2nd January 2016, 18:15

From what I see, it's doing most of the job already by itself.
Also, http://avisynth.org.ru/fft3dfilter/fft3dfilter.html is mentioned, very full implementation too.

What remains, is manual adjustment of the noise floor and spectral correction on the timeline, also for distinct blocks. This goes very far, but perhaps can salvage some parts of a video, e.g. with allowing more noise, and the human eye will still filter out some more information.

What I know from audio, there is hard operation (block by block, still with overlapping) and smooth operation (it builds an average of the sigma to use, with an adjustable time constant).
When hard operation is good, so we go for it, but smooth operation needs often a slower time constant, to avoid ugly artifacts.
With video, I guess this becomes 3-dimensional though, plus 3 planes to compute, with some intercorrection using a luminance check.
Ah and let's not forget physics. What is the source of the noise? If it is the three color planes in the camera array, which may work independently of each other, then probably it is the best solution to operate on RGB space with individual processing of all three. If it is the video recorder, then perhaps it is Y, U and V that have to be processed.

raffriff42 · 2nd January 2016, 18:55

Good points. Yes, fft3dfilter does work in a similar way, and has some other interesting features, but dfttest allows manual adjustment of noise reduction strength for each X & Y (and T?) frequency band through the sfile argument, which is why I looked at dfttest first.

EDIT I forgot to mention, the automatic noise reduction happens in the first nfile= pass, and that may be good enough for some cases.

geometer · 2nd January 2016, 19:36

I have another idea: dynamic cross-correlation.
When all 3 data planes show the same relative pattern, then sigma can be locally reduced. When moves (or delta) go fully different, then sigma can grow.
That might change content a little bit, but for heavy noise it might improve looks.

2nd January 2016, 18:15	#2 \| Link
geometer Registered User Join Date: Dec 2015 Posts: 59	From what I see, it's doing most of the job already by itself. Also, http://avisynth.org.ru/fft3dfilter/fft3dfilter.html is mentioned, very full implementation too. What remains, is manual adjustment of the noise floor and spectral correction on the timeline, also for distinct blocks. This goes very far, but perhaps can salvage some parts of a video, e.g. with allowing more noise, and the human eye will still filter out some more information. What I know from audio, there is hard operation (block by block, still with overlapping) and smooth operation (it builds an average of the sigma to use, with an adjustable time constant). When hard operation is good, so we go for it, but smooth operation needs often a slower time constant, to avoid ugly artifacts. With video, I guess this becomes 3-dimensional though, plus 3 planes to compute, with some intercorrection using a luminance check. Ah and let's not forget physics. What is the source of the noise? If it is the three color planes in the camera array, which may work independently of each other, then probably it is the best solution to operate on RGB space with individual processing of all three. If it is the video recorder, then perhaps it is Y, U and V that have to be processed. Last edited by geometer; 2nd January 2016 at 18:26.

2nd January 2016, 18:55	#3 \| Link
raffriff42 Retried Guesser Join Date: Jun 2012 Posts: 1,373	Good points. Yes, fft3dfilter does work in a similar way, and has some other interesting features, but dfttest allows manual adjustment of noise reduction strength for each X & Y (and T?) frequency band through the sfile argument, which is why I looked at dfttest first. EDIT I forgot to mention, the automatic noise reduction happens in the first nfile= pass, and that may be good enough for some cases. Last edited by raffriff42; 2nd January 2016 at 20:03.

2nd January 2016, 19:36	#4 \| Link
geometer Registered User Join Date: Dec 2015 Posts: 59	I have another idea: dynamic cross-correlation. When all 3 data planes show the same relative pattern, then sigma can be locally reduced. When moves (or delta) go fully different, then sigma can grow. That might change content a little bit, but for heavy noise it might improve looks.