New very slow FFT denoiser

[DarkFoon]

fizick, I have a question.
A few months back you said that you were going to open source the next release of this filter. The next release has come and gone, but no open source. Have I just not found it? Or did you change your mind?

[Fizick]

Firstly i did not want to open the source.
So, I still did not change this my mind about it.

Lets return from policy to technik.
Version 1.4 - July 23, 2005 - corrected neutral level for chroma processing,
changed weighting window type with new wintype parameter.

wintype - weighting windows type (default=1 since v1.4):
0 - same analysis and synthesis half-cosine window, used in all versions before 1.4;
1 - intermediate between 0 and 2;
2 - flat analysis window, rised cosine (Hanning) synthesis window.

Weighting window wintype=0 can produce the worst grid artifactes, window type wintype=2 do not produce grid artifactes, but can produce some ringing, wintype=1 is intermediate case.

Probably it is my last attempt in grid artefactes preventing.
It is still not perfect solution, but some advance.

What next?
I think about noise pattern measure and using for filtration.
But why i want to do it? I want remove not noise, but grain with some regular spectrum.
First alpha version is almost ready, but is not very good.
Here is method (partially implemented):
User must preview clip and select some region (block) as or x-y coordinate (block x-y numbers in current alpha version) in some one arbitrary frame (by frame number). This region must contain noise only, without any image detail.
Filter make FFT of this block. This spectrum is pattern of noise spectrum.
Then we use this noise pattern spectrum in place of (previously constant) sigma for all blocks filtration (for all frames).

This generally works, but not very good.
Firstly, it is spatial only, not temporal noise pattern.
Spatial filter is not so powerful as 3D.
Probaly we can take 2 (or 3) frames to get 3D noise.

Secondly, there are windowing problem again (grid formation).
Thirdly, the manual pure noise region selection is not so easy. And its spectrum using for all regions and frames is not very reasonable too.

But some success is quite posiible. At last we can use this spectrum to get real estimation of constant sigma.
May be some sigmas-map? :)

[708145]

Firstly i did not want to open the source.
So, I still did not change this my mind about it.


Hi Fizick!

Is it just that you don't want others to interfere with your coding? Or do you feel it's badly documented?
Whatever your reasons, I hope you will release it once you stop developing it so that others can at least continue and/or port it to new versions of avisynth. :D

About your "noise shaping": go ahead :) The filter is already great and any improvement will bring it closer to "perfection".

Maybe you could team up with prunedtree (meet him on #xvid on freenode) since he is working on (noise) filters as well.

bis besser,
Tobias

[Soulhunter]

Maybe you could team up with prunedtree...
Yeah, Fizick n' Pruned...

Would be THE dream-team!


Bye

[cwk]

Howdy Fizick,

Thank you for the new version. I am able to safely use Sigmas above 1 on VHS captures without Gridding. The interlaced flag also allows me to skip separatefields or TDeint, which saves some time and typing :) .

I have an enhancement request for the sharpener. As interlaced fields have 1/2 vertical resolution, how about a parameter to super-sample the image vertically by a factor of 2? (or a variable) A user could use this type of feature with interlaced material to sharpen without aliasing. Would this be difficult to do?

cwk

[Fizick]

cwk,
did you try svr parameter to control vertical sharpening?
I do not want upsampling.

[cwk]

I have tried the svr option, but most of my sources are old VHS tapes of sports events with alot of built-in aliasing. I was looking for a way to remove SangNom from my filter chain.

[Fizick]

cwk,
it is probably line jitter?

[Fizick]

New Version 1.5 - July 26, 2005 - added noise pattern method
Since v1.5 it is possible to get noise pattern (shape) by spectrum analyisis of some empty block (without any objects), and then to reduce the noise with the same pattern in whole frame and in whole clip. It may be useful for removal of film (especialy amateur 8 mm) grain and analog TV capture interference.
New parameters:
pframe - noise pattern frame number (default=false)
px - noise pattern block horizontal X position (default=0)
py - noise pattern block vertical Y position (default=0)
If px=px=0, then the pattern block is defined automatically with minimal power spectral density.
pshow - show noise pattern block and its properties (default=false)
pcutoff - noise pattern cutoff frequency (relative to max) (default=0.1)
pfactor - noise pattern denoise strength (0 to 1.0, default=0, this method disabled)

In order to use noise pattern method in place of ordinary (blind) method you must:
Firstly switch show mode pshow=true, and set some non-zero value pfactor=1.0.
Then select frame number and some block position, thus the block shown must not contain any objects beside typical noise pattern.
The switch off show mode pshow=false,
and set noise reduction strength, recomended value pfactor= 0.5 to 1.0.
The best windows type for this method is wintype=2.
The sigma and beta parameters are not used in this denoising method, but you can use this method in show mode pshow=true to estimate sigma value for ordinary denoising method (it is not strictly the same, but similar value).

The noise pattern method is very experimental.

[Soulhunter]

Hi, Ive already played around with the new noise-pattern stuff... One thing Ive noticed is that it removes much less noise than the usual processing. Even with very high denoising values only a small amount of noise gets removed! Ok, you wrote its still very experimental, and maybe Ive just used the wrong sample or the wrong settings or so, heh? Anyway, nice concept! Another thing Ive noticed is that "bt=0" produces some weird ghosting! Is this a known problem? Coz, I cant really remember that Ive seen this behavior before... :\

Source
http://img62.imageshack.us/img62/6492/81raw2ek.png (http://imageshack.us)

FFT3DFilter(Sigma=2,BT=0)
http://img62.imageshack.us/img62/1938/81fft3dfiltersigma2bt05qe.png (http://imageshack.us)

Bye

[DarkFoon]

@Soulhunter
bt=0 always causes ghosting with high sigmas, and (I can only guess) high pfactors.
also, check your kratio.
basically, it averages the frames out to remove the noise (if used with high values) so you get wierd ghosting when there is motion. Personally, I don't like to use that mode because it causes artifacts with motion. I use the weiner modes because, although less powerful than kalman at noise removal, they don't cause ghosting (nearly as bad).

[Soulhunter]

@ DarkFoon

Thanks for the confirmation... ^^

I mainly used bt=3 n' low sigma values for my encodes...

So, probably thats the reason Ive never noticed this ghosting... ^^;


Bye

[Soulhunter]

Ok, the source was the last chapter of "SAW" (PAL version) Its a very noisy/grainy flick, but to make it even harder I amplified the noise with some slight sharpening. Finally I cropped/resized the filtered result to 720x384 and encoded it with XviD @ Q2 / Soulhunters v3 / 1-1-1 B-VOPs. Well, here are some short samples of the results...

The "non-denoised" version: 27.250.688 Bytes / Sample (http://rapidshare.de/files/3436632/RAW_02.avi.html)

FFT3DFilter(Sigma=2,BT=1) on all planes: 18.161.664 Bytes / Sample (http://rapidshare.de/files/3437081/2x1x3_02.avi.html)

FFT3DFilter(Sigma=2,BT=3) on all planes: 17.246.208 Bytes / Sample (http://rapidshare.de/files/3437675/2x3x3_02.avi.html)

EDIT: FFT3DFilter(PFactor=1,BT=3) on all planes: 19.668.992 Bytes / Sample (http://rapidshare.de/files/3443697/P1.avi.html)

Observation 1: A compression gain of ~30% without killing the real details! Nothing new, but... ^^

Observation 2: Bt=1 seems to keep the fluctuation/flickering of the noise while BT=3 calms its down !?!


Bye

[DarkFoon]

@soulhunter
thanks for the stats! those are very interesting.
do you think you could run the script again using the new noise pattern removal for comparison?
thanks!! ^^

[Soulhunter]

@ DarkFoon

Ok, Ive added a additional "PFactor=1" sample... ^^


@ Fizick

If I use the "px/py" params + "pshow=true" VDubMod n' MPlayer crashes... :\


Bye

[acrespo]

I didn't understand the difference between noise pattern and old method.

With my analog anime TV captures I am using sigma=8 in old method. When I change pfactor=1.0 it's seem that old method is better to me, because more noise was removed.

Can anybody explain the difference between methods?

[DarkFoon]

the noise pattern method works by you telling the filter what is noise by selecting a certain block in a certain frame. Then the filter tries to find the same noise when it filters the whole movie, and remove it. the problem with this method, is that it is very difficult to remove large ammounts of noise (in my opinion, I perfer "blind" noise removal, over this "selective").
the normal mode just looks for the noise frequencies, and then removes them. It's much better.
I'm not a mathematician, nor a programmer, so I may be wrong on these, but this is how I understand how the filter works.
Fizick, please correct me if I am wrong.

[Fizick]

Soulhunter,
yes, it was a bug with pshow. I fixed it in v1.5.1 today.

About your tests: I almost do not see any noise in your sources. :)
If my sources would be same as your, i never create any denoiser.
A do not see ghosting (almost)
My denoisers was developed for really poor sources.
You may use fft3d for any sources, but at your own risk :)
It is especially true for pattern mode.
Pattern mode is developed for really big gran, or some noise structure (TV interference, like Defreq).
For "white" noise, the old blind method is optimal, as Darkfoon notes.
May be, pattern mode will be improved later.

Darkfoon,
your info is correct, thanks. I very like your comments. :)

[Soulhunter]

I almost do not see any noise in your sources. :)

Ouch, for me this sample is veeeery noisy (at least for a DVD source) Well, with the same settings my usual DVD->1024x576 encodes eat 4000-5000kbps, but this source eats more than 7000kbps at a framesize of just 720x384... :\



If my sources would be same as your, i never create any denoiser.
You may use fft3d for any sources, but at your own risk :)

Hey, a compression gain of ~30% without loosing "real" details is fantastic! Actually I consider FFT3D as one of the best denoising filters for DVD sources, especially for high bitrate encodes where you would notice even the smallest detail loss... ^^


Thanks n' Bye

[DarkFoon]

Actually I consider FFT3D as one of the best denoising filters, especially for high bitrate encodes where you would notice even the smallest detail loss... ^^

Amen!
Actually, right now, I am using FFT3D in a little experiment: I am taking a TV capture and cleaning it up and then upscaling it to 1280x720 (HD ^^) and re-compressing it.
In order to make the image as clean as possible and to make it sharp, I must use FFT3D as it is the only filter best suited to the job.

OT: anybody have any reccomendations for a codec that would handle that resolution? (I was going to use WMV9-VCM)
and I can't get x264 to playback on my computer(I have FFDshow and WMP still crashes) not to mention, stupid quicktime wants to play all my mp4 files.
sorry for the tangent ^^;

[Soulhunter]

OT: anybody have any reccomendations for a codec that would handle that resolution? (I was going to use WMV9-VCM)
and I can't get x264 to playback on my computer(I have FFDshow and WMP still crashes) not to mention, stupid quicktime wants to play all my mp4 files.
sorry for the tangent ^^;

Well, as always... it depends on the bitrate... ^^

For higher bitrates I still prefer XviD (faster n' less smoothing than WMV9/AVC)

But for lower bitrates AVC/WMV9 should give better results (less blocking etc.)


Bye

[DarkFoon]

@Fizick
As I understand, BT=3 or 2 means that the filter takes the current frame and averages it with the future frame and the past frame (depending on the mode). Then, the next consecutive frame, it filters by using the original frame preceding it, not the processed version of the previous frame.
... Am I right?

well, the feature I am requesting is where the filter uses the filtered version of the previous frame, instead of the un-filtered version. As I understand this, this would be like a hybrid between weiner and kalman modes.

I think this would remove much more noise than the current situation. I think it should be a parameter, so the user could choose between straight weiner mode, and this "hybrid" mode.

Unless I'm wrong, and weiner already uses the processed version of the frame previous to the one currently being processed. This also doesn't sound like it would be much work to implement.

I would like your feedback, and correction on my theories, if I am wrong.

[Fizick]

New Version 1.5.2 - July 31, 2005 - fixed bug with Kalman mode (bt=0) for Athlon (introduced in v1.5)

Soulhunter,
if you uses Athlon, please re-test ghosting in Kalman mode now.

Darkfoon,
Your understanding is correct.
As i noted somehow, there are many possible different denoise method for ordinary (not-frequency) approach in other filters existed .
We could use almost any of them for frequency approach in FFT filter.
But now I prefer use scientific approved methods only :)
Wiener and Kaman is well-known optimal methods.
IMHO, your "hybrid" mode is quite similar to Kalman.
To minimize ghosting in Kalman mode, try decrease sigma.
However, i will try implement bt=4 (two prev frames) and bt=5 (two prev and next frames) Wiener modes in next version.

[Soulhunter]

Soulhunter,
if you uses Athlon, please re-test ghosting in Kalman mode now.

Yup, Im a Athlon user... ^^

And it seems ya bugfix solved the problem!


Original:
http://img20.imageshack.us/img20/3853/63orig0zd.png

Old version
http://img20.imageshack.us/img20/322/63old2hf.png

New version
http://img20.imageshack.us/img20/6181/63new2wc.png


Thanks n' Bye

[Fizick]

Version 1.6 - August 03, 2005 - added mode bt=4; added optimized SSE version for bt=2,3; added sigma2,sigma3,sigma4 parameters for combined method.

Ñombined method: when you can directly set different noise values sigma for highest and sigma2, sigma3, sigma4 for lower frequencies. The pattern coefficients will be created internally from these sigmas values by interpolation. Set pfactor=0 for this method (internally it will be =1).

SSE version is nof faster then 3Dnow for my Athlon, but Pentium users must have some improvements. please confirm.

[AI]

in last version in russian documentation
two section "bt=3" in part "Plugin has several processing modes at filter stage"
and
no section "bt=4" in part " Function parameters"

PS SSE optimisation is realy faster on my P!!!800 (~15-20%)

ЗЗЫ 2 Fizick - надеюсь ты понял мой ломанный английский :D

PPS 2All Excuse me my english (my language is russian)

[Isochroma]

Your excellent filter has passed muster against all others currently in the contenders' ring! It is the best, and it surpassed the rest by a wide margin.

So, I'm writing to thank you, the developer, for working so hard to help bring world-class code to the masses. Your tool is helping countless people see the digital future with greater clarity and brilliance.

I hope that you see fit to continue developing this great AVISynth plugin; and I'm also sure that for every letter like this one, there are at least a hundred users who also appreciate your work but have chosen not to express their gratitude publicly.

[Fizick]

AI,
Thanks for pointing to the documentation bug. I will correct it.
What is your speed gain with SSE and what is your CPU?

Isochrona,
Thank for your kindly words!
But I hope that soon I will finish this plugin development at last ! :)
(and will develop somithing else.)

[cwk]

Hello Fizick,

I have some questions about the sigmas now available in 1.6.

Does these indicate the sigma that will be used on a particular noise pattern that I should isolate with pframe, px and py? Or do they apply to general low frequency noise that will be determined by the tool?

Also, preliminary tests indicate the following two statements are functionally equivalent:

1. FFT3DFilter(sigma=3, sigma2=2, pframe=100, px=15, py=6, plane=0)
2. FFT3DFilter(sigma=3, plane=0)

Do the sigma2, sigma3 and sigma4 values only apply when they are greater than sigma?

FFT runs now 10-15% faster with version 1.6 on a PentiumIV. I will post more exact results this weekend.

cwk

[Fizick]

cwk,
sigmas will forms pattern.
they apply if any of them is defined not equal to sigma.

[Fizick]

Addition:
sigmas mode does not use pattern position, it is general.
More corrrect: these sigmas define some coefficint matrix (size bh * bw/2), which then used at filtering and inverse FFT stages the same way as pattern spectrum was used.
I forget to say, that prosessing in this (sigmas) mode will depend at frame synthesis stage on wintype parameter (which defines windowing of low frequencies at frame analyze and synthesis stages).
I hope, my addition do not add more confusing. :)
But I do not ask to excuse my english. :)

[DarkFoon]

so, the new sigma values are for the different (noise) frequencies?

using fictional (meaningless) numbers:
frequencies 100 - 200 use "sigma"
frequencies 50 - 99 use "sigma2"
frequencies 25 - 49 use "sigma3"
and frequencies 4 - 24 use "sigma4"

basically, its a way to specify specific sigmas for different frequencies, yes?

if I am understanding correctly, the user needs a way to specify the ranges for each sigma, otherwise, the filter does that, and they have no control over it.

but if that's not the case, it'd be really cool to have as a feature! :)

Imagine it, for the highs, middles, and lows of the frequency spectrum, the user could set specific sigmas.

well, I'm just rambling. sorry :P

[Fizick]

Darkfoon,
You are not quite correct.
The pattern coefficients will be created internally from these sigmas values by interpolation.
So, for intermediate frequency we have intermediate sigma value.
Strictly:
lets maximum (2D) frequency = 1,
at f1=1 (highest frequency) we use "sigma" value,
at f2=0.354 (middle) we use "sigma2" value,
at f3=0.177 (low) we use "sigma3" value,
at f4=0 (lowest frequency) we use "sigma4" value.

At some intermediate frequency we use intermediate sigma value, calculated by linear interpolation between nearest frequencies for sigmas given.
For example, for frequency in the middle between f2=0.354 and f1=1, we have f=(1 + 0.354)/2=0.677, and we use (sigma+sigma2)/2 value.

You can test various sigmas influence by subtract command:

i=avisource("your.avi")
fft3dfilter(i, sigma=0.01, sigma2=2,sigma3=0.01,sigma4=0.01)
subtract(i,last).levels(117,1,137,0,255)

change sigmas values and see what you get (what noise is removed).

Please understand, that this mode is experimental, and i am still not sure, how good and how useful is it.
I create it to simplify the pattern mode using.
In general, it is possible directly specify full pattern spectrum matrix, without interpolation, but it has size bh*bw/2 =1152 :)

[DarkFoon]

ok, i think i understand much better now.
thanks Fizick!! :)

[AS]

Hi, I have a Northwood p4 2.2 GHz and the encoding time dropped by slightly more than an hour using the new version, with the old version taking slightly over 8 hours in an encoding script with multiple filters. So the gain in speed was more than 13%, considering the multiplier effect.

Thank you for your work and effort ^^

[trbarry]

Fizick -

Drat! ;)

This sounds like a nice filter but I just dragged through the entire thread because somewhere a dozen pages back or so you said you might open up the source.

I was thinking about writing a slightly different filter along these lines and was looking for some good open VS6 compat optimized FFT code I could use.

Anybody recommend any?

- Tom

[tsp]

Anybody recommend any?
http://www.fftw.org/

[Fizick]

trbarry,
are you really returned? It will be great!

1. The fastest open FFT code is probably FFTW at http://www.fftw.org
It is comparable by speed with Intel fft library.
I use FFTW in my 3 plugins: Depan, Defreq, and fft3dfilter.
Defreq is open source, so you can look how to use it in VS6.

2. About opening the source.
IMHO, the most important advantage of opensource is team work, contribution.
For example Defreq is open since october 2004, but nobody contribute to it (though it was not perfect and quite draft).
I hope it will be useful for you at least.

But the main reason why i do not open fft3d - i still do not believe in GPL as a real protection from theft my code by some proprietary firm.

[trbarry]

trbarry,
are you really returned? It will be great!

Hi Fizick -

Not really, I'm still too busy but do stop in from time to time.


1. The fastest open FFT code is probably FFTW at http://www.fftw.org
It is comparable by speed with Intel fft library.

I downloaded FFTW a couple weeks ago but never got to it. I thought it said I needed the Intel compiler or vs7 (can't remember) but I don't currently use either.


I use FFTW in my 3 plugins: Depan, Defreq, and fft3dfilter.
Defreq is open source, so you can look how to use it in VS6.


Okay, cool, I'll get Defreq instead. That should be fine.


2. About opening the source.
IMHO, the most important advantage of opensource is team work, contribution.
For example Defreq is open since october 2004, but nobody contribute to it (though it was not perfect and quite draft).
I hope it will be useful for you at least.

But the main reason why i do not open fft3d - i still do not believe in GPL as a real protection from theft my code by some proprietary firm.

No problem. My own stuff is all over the web, sometimes probably in violation of the GPL. But I've found that if I spend too much time worrying about protecting things then I don't get anything useful done anyway. And I've received a number of useful contributions. Even a couple from you IIRC. thx ;)

- Tom

[Fizick]

trbarry,
the most important thing about fftw using:
Do not use (and complile) its source yourself.
Simple use header and already compiled binary. The link is in defreq doc.

[tsp]

only compile it yourself if you want to see what fft3dfilter does send to fftw ;)

[trbarry]

trbarry,
the most important thing about fftw using:
Do not use (and complile) its source yourself.
Simple use header and already compiled binary. The link is in defreq doc.

That's sort of the problem. I like optimizing and tinkering with things. ;)

- Tom

[videoFred]

FFt is a fine denoiser, I use it together with degrainmedian() and limitedsharpen().

But sometimes I get artifacts:
http://users.telenet.be/ho-slotcars/testmap/artefacts.jpg

The artifacts are visible in the dark parts only.
I do not have them with bt=0, but bt=3 or 4 works much better.

Maybe someone could give me some hints?

Thanks!
Fred.

[krieger2005]

I think this is because of sharpening... this problem is because of the combination of the softener and sharpener.... but this could be wrong... try to remove "LimitedSharpen" or any other sharpener... is this affect then there? Another try: sharpen only the luma-(Y)-Channel...

[cwk]

@videoFred:

Although FFT is a fantastic denoiser, it has been known to leave some grid-like artifacts. It could be that these are present after you use FFT, but are only visible after sharpening with limitedsharpen. Have you tried using wintype=2 ? This was designed to help alleviate gridding.

Also, try setting your oh and ow to 1/2(bh and bw). This will slow things down further, but seems to eliminate gridding for me with wintype=1.

Hope this helps!

ck

[DarkFoon]

The type of artifacting you are showing there seems to be consistenet with kreiger2005 is saying. Have you tried the built-in shrpener with FFT3d? Try that instead of another sharpener, and see if it makes a difference.
also, what channels are you processing? just the luma or luma + chroma? (looks to me like luma only)

[Backwoods]

Have you tried the built-in shrpener with FFT3d? Try that instead of another sharpener, and see if it makes a difference.

The same result happens, the grid is there and comes out with any sharpening.

I tried the "wintype=2" and the oh/ow method cwk mention, neither helped with the grid pattern.

Is there any settings anyone can suggest that proved to them to show no grid pattern?

[DarkFoon]

what is your sigma? what bt mode are you using, what processor do you have, any other filters before/after fft3d?

There could be something in common with you and videoFred, because I never get any of these problems.

[Backwoods]

FFT3DFilter()

or

FFT3DFilter(sigma=3, plane=0,ow=8,oh=8)

or

FFT3DFilter(sigma=5,plane=0,ow=24,oh=24,wintype=2)
FFT3DFilter(sigma=5,plane=1,ow=24,oh=24,wintype=2)
FFT3DFilter(sigma=5,plane=2,ow=24,oh=24,wintype=2)

with:
LimitedSharpen(smode=1,strength=500)
to emulate what VideoFred is doing.

P4 2.8

What settings have you used and received good results?

[cwk]

Here is a common setup for me:

FFT3DFilter(sigma=3, bh=60, bw=60, oh=30, ow=30, plane=0, sharpen=1, smin=2, smax=5)

I would imagine the gridding would be worse for me, because the lower smin setting includes more noise. But I don't see any in dark areas.

ck