New very slow FFT denoiser

[egrimisu]

In the anime scene a girls hair is moving, when the hair is moving in the next frame remains a few trails from the previews frame. Like blending but not that intense. If it's necesary i'l post some screens

[fjhdavid]

Dear all,

I read a "denoising comparaison" from soulhunter (http://soulhunter.chronocrossdev.com/dcpart1.htm) , and it is written that between bt=1 end bt=3, the S/N is not so different (you have just to change a litle bit the sigma value).

Do you think the S/N is the only parameter to take into account, as the processing speed difference between bt=1 and bt=3 is very important!

does anybody look carefully at the difference between bt=1 and bt=3?

thanks
Francois

[egrimisu]

What is the BT ? i tried BT=1 and BT=5 and with bt=5 i got a a frame a line that normaly would appear in the next frame or the second frame.

[LaTo]

What is the BT ? i tried BT=1 and BT=5 and with bt=5 i got a a frame a line that normaly would appear in the next frame or the second frame.

BT=1 --> spacial
BT=2 --> spacial + temporal (prev frm)
BT=3 --> spacial + temporal (prev + next frm)
BT=4 --> spacial + temporal (2 prev + next frm)
BT=5 --> spacial + temporal (2 prev + 2 next frm)

[egrimisu]

And why are details from next frame apear in the curent frame using BT=5? is that normal?

[LaTo]

And why are details from next frame apear in the curent frame using BT=5? is that normal?

it's call "ghosting", it's appear with high sigma....

[egrimisu]

It's quite visible with sigma=0.7 , am i doing something wrong?

[LaTo]

It's quite visible with sigma=0.7
So use only spatial mode.

[Fizick]

egrimisu,
what is "quite visible"? provide a script and short source and output clips (or images)

[fjhdavid]

sorry to ask my question a second time.

assuming we are talking about progressive source film (PAL or others progressive source),do you think that 80% or 90% of the "denoising work" is already done with the spatial denoiser setting alone (bt=1)?

thanks for your comments and visual experience
Francois

[Soulhunter]

I read a "denoising comparaison" from soulhunter (http://soulhunter.chronocrossdev.com/dcpart1.htm) , and it is written that between bt=1 end bt=3, the S/N is not so different (you have just to change a litle bit the sigma value).
Actually a difference of ~0.8dB isn't that small! I know, this numbers can be irritating when you are not used to interpret them... But to give you an idea: "The MPEG committee used an informal threshold of 0.5 dB PSNR to decide whether to incorporate a coding optimization because they believed that an improvement of that magnitude would be visible... (from some paper about PSNR)" So, a increase of ~0.8dB can mean a visible improvement! :]


Bye

[fjhdavid]

Soulhunter thanks for the info!

do you know what is the typical lost in term of S/N (in dB) when you are doing a re-encoding with a good mpeg2 encoder like CCE (two pass and best settings)?


Fizick and Soulhunter, with fft3dfilter filter (sharpen only (bt=-1))

is it better to sharpen AFTER or BEFORE a lanczos up-scale resizing (or is there no difference)?

for best sharpening, is it better to process ALL the color planes, or is it better to process ONLY the luma chanel?


Thanks
Francois

[Soulhunter]

do you know what is the typical lost in term of S/N (in dB) when you are doing a re-encoding with a good mpeg2 encoder like CCE (two pass and best settings)?
At what bitrate!? Full SD movie -> DVD5? Dunno... Im not that much in MPEG2 encoding, but I can tell you that a good MPEG4 ASP encoder (XviD) can reach 50dB (thats ~ the threshold I consider as visual identical compared to the source) or more in this scenario!


...is it better to sharpen AFTER or BEFORE a lanczos up-scale resizing (or is there no difference)?
Dunno... Try it n share your results with us! :]


for best sharpening, is it better to process ALL the color planes, or is it better to process ONLY the luma chanel?
Normal shapening wont do much on the chroma planes, at least if you work with YV12 content! YV12 = Strong subsampling -> So, depending on the renderer the output will be either strongly aliased (upsampling without interpolation) or blurry (with interpolation) no matter if you sharpen it beforehand... Only thing that can be advantageous in some cases (chroma bleeding) is to use some warpsharp on the chroma planes. For non subsampled YUV on the other hand, sharpening the U/V planes can help... Tho not as much as on the luma plane, because changes on the chroma are much less visible for the human eye than changes on the luma!


Bye

[fjhdavid]

thanks Soulhunter!

for the mpeg2 re-encoding I leave with the same bitrate (dvd5 -> dvd5), so the S/N ratio would be greater than even 50dB in this case (and with no visual difference)?


I will test sharpening before and after resizing...and I will tell you (but since I read in several places that after is better...)


last question, I read threads about denoising and it appears that most of the "bad noise (most visible)" is in the chroma planes, and that the "grain noise" is in the luma plane.
do you agree?

so can I conclude that I have to use an even stronger sigma on colors planes than on the luma plane?

[Soulhunter]

for the mpeg2 re-encoding I leave with the same bitrate (dvd5 -> dvd5), so the S/N ratio would be greater than even 50dB in this case (and with no visual difference)?
Probably not, as MPEG2 should do worse than MPEG4...

You could do a quick PSNR compare (5000 frames or so) to find out!


last question, I read threads about denoising and it appears that most of the "bad noise (most visible)" is in the chroma planes, and that the "grain noise" is in the luma plane.
do you agree?

so can I conclude that I have to use an even stronger sigma on colors planes than on the luma plane?
For me, chroma noise isnt really much more visible than luma noise, but I find it much more disturbing than luma noise! So, yes... Supposed the amount of noise is equal in Y and U/V, Id denoise the chroma different/more than the luma. But fortunately this situation (more chroma than luma noise) is very very rare! :]


Bye

[fjhdavid]

in your last sentence, you wanted to say "supposed the amount of noise IS NOT equal in Y and U/V" instead of "supposed the amount of noise IS equal in Y and U/V", right?

so, most of the time, I should used the SAME sigma for all the planes (Y, U and V)? right?

[Soulhunter]

in your last sentence, you wanted to say "supposed the amount of noise IS NOT equal in Y and U/V" instead of "supposed the amount of noise IS equal in Y and U/V", right?
I wanted to say that usually the amount of noise (measured/objective amount, not visible/subjective amount) in the chroma is lower than in the luma (also caused by the fact that you work with YV12 content -> Subsampling/Upsampling = Detail loss -> Loss of noise). But if the amount of chroma noise would be at the same level as in the luma, id denoise the chroma stronger than the luma... Because I find chroma noise more disturbing than luma noise! So, I guess the my sentence was right!? >.>


so, most of the time, I should used the SAME sigma for all the planes (Y, U and V)? right?
Supposed you find chroma noise more disturbing than luma noise (like me), yes! Fortunately you wont really notice "oversmoothing" that much on the chroma planes like on the luma planes! So its unlikely you do harm to the chroma by picking a equal value for luma n chroma. Final advise: Better use your eyes, not my suggestions, for tweaking filter settings... After all its a pretty subjective thing (what looks better n what not) eh!? ;]


Bye

[fjhdavid]

Soulhunter thanks one more time for your knowledge! It helps me a lot

I am sorry for my post about bt=1 vs bt=3 speed comparaison. I did tests more carefully yesterday, and the speed difference between bt=1 and bt=3 (with the others settings equal) is not more than 10% ! so don't ask yourself and go directly for bt=3...!

Soulhunter,

I would like to know if your "denoise PSNR results" are DVD or noise dependant, I mean if the sigma ranking and value would be almost the same with an another DVD or noise amplitude value (even if the PSNR value are, of course, different)?

As I have more time, I would like to do PSNR measure myself with my DVD which are records from a digital satellite with an analogue cable (RVB scart). I know that I have to choose a sigma=3 or above (seen in FFT3Dfilter documentation), but I would like to test others parameters like bw, oh...
could you send me your comparaison program or tell me how to measure PSNR directly with a DVD?

thanks
Francois

[Soulhunter]

I would like to know if your "denoise PSNR results" are DVD or noise dependant, I mean if the sigma ranking and value would be almost the same with an another DVD or noise amplitude value (even if the PSNR value are, of course, different)
As you can derivate from my comparison, the ranking doesnt depend much on the footage type (ranking order was almost the same for the film and the anime source iirc). But the kind of noise you cope with could indeed alter the results to some degree (the worst wont get the best or the other way around, but 1st n 2nd could switch places or whatever...). And, as the kind of noise is different from DVD to DVD, results could change here as well!


As I have more time, I would like to do PSNR measure myself with my DVD which are records from a digital satellite with an analogue cable (RVB scart). I know that I have to choose a sigma=3 or above (seen in FFT3Dfilter documentation), but I would like to test others parameters like bw, oh...
could you send me your comparaison program or tell me how to measure PSNR directly with a DVD?
How you wanna compare the different parameters via PSNR in this scenario? I mean, you would need a noise free version of your source (which represents the optimum you try to reach via denoising). Otherwise you would just measure the difference to the source (which can also mean removal of detail etc.) and not the denoising efficiency! Thats why I used a clean source (ClipA) for my compare and added artificial noise resulting in ClipB. Then I applied the different denoisers on ClipB to get ClipC. Finally I compared ClipA vs ClipC... You see the difference? Anyway, if you want to do PSNR comparisons, you can use Avisynths Compare() (http://avisynth.org/oldwiki/index.php?page=Compare) or some tool like this (http://compression.ru/video/quality_measure/video_measurement_tool_en.html).


Bye

[fjhdavid]

thanks Soulhunter,

after three re-encoding I come to the conclusion that there is no difference between sharpening after or before upscale resizing (at least I am not able to see them)

BUT, I have another problem:

I use FFt3Dfilter (sigma=3,bh=32,bw=32,oh=8,ow=8,sharpen=1) (oh=ow=8 for speed reason).

I do then a split with a stackhorizontal and HONESTLY I am not able to see the difference between the original part (on the left) and the filtered part(on the right).

I have a projector (the image is really big) and I just look at the film even close.

Do I have to do still frame and look more carefully? (I remembered pictures on the web, showing big difference).

I don't understand because my film is not perfect and sigma=3 and sharpen=1 are big values).

any comments?

francois

[Boulder]

Please post the complete script that you used for comparison.

[cogman]

Hey all, I was reading through the FFT3dFilter source (specifically FFT3dFilter.cpp) and noticed several places that can be improved by a little math.

For example. I saw a couple of places that look something like something = (((j - i - bh) * 2.0f)/bw)*(((j - i - bh) * 2.0f)/bw);

that could easily be converted to

something = ((j - i - bh) * (j - i - bh) * 4.0f) / (bw * bw);

taking out one division and one multiplication (should be SOME gain when something like this is done in a for loop.)

This could possibly be further speed up by making another variable, j_minus_i_minus_bh = j - i - bh and using that in the calculation for the square.

There are SEVERAL places like this and a lot of them in loops. Do you think it would be worth my time (or yours :P) to go through and simplify math problems like this?

heck, in places where the purpose is the square the results (like the above given) you probably would see an increase if you did
something = ((j - i - bh) * 2.0f)/bw);
something = something*something;
because that would only do the complex math problem once and use the answer to get the rest of the problem.

Your thoughts?

[Gavino]

heck, in places where the purpose is the square the results (like the above given) you probably would see an increase if you did
something = ((j - i - bh) * 2.0f)/bw);
something = something*something;
because that would only do the complex math problem once and use the answer to get the rest of the problem.
A good optimising compiler ought to be able to do this automatically (repeated sub-expression removal).

OTOH it can also make the code easier to read/maintain if the programmer does it explicitly, although if taken to extremes it can actually make the code harder to follow.

[Adub]

I vote do it. It's all in the name of speed!! But of course, Fizick will have the final say.

[Fizick]

i do not consider myself as a programmer guru. But IMHO, other parts of code take the most time.

[Monamona]

The latest fftw3.dll changed its name to libfftw3x-3.dll.
I appreciate it if you modify FFT3DFilter compatible with it without rename.

[Adub]

Why? You don't need the new dll, and supposedly the old one is faster.

[fjhdavid]

a message for Soulhunter:

Soulhunter, I don't know if it requires a lot of work, but in your denoiser comparaison, could you give me, if you can, the value (in dB) of:

Serenity Trailer [2000 Frames]

fft3dfilter(sigma=1.75,bw=32,bh=32,oh=16,ow=16)

I would really appreciate

thanks anyway for all the answers you gave me already

Francois

[fjhdavid]

does anybody notice some visible ringing or haloing beyond a sharpen definite value?

what was this "maximum" sharpen value in your tests?

thanks
Francois

[thetoof]

Like always, it depends a lot on your source. I prefer using an external sharpener than the one of fft3d... Limitedsharpenfaster() helps a LOT reducing the artifacts you're talking about.

[fjhdavid]

honestly, LSF (Smode=3) and fft3Dfilter sharpen look really the same for static images (I use stackhorizontal with DVD-RB and took pictures).

It seems that fft3Dfilter sharpen=0.75 is equivalent to strenth=110 for LSF, but both seems to use "limitedsharpen" feature.

Fizick (if you have time) could you explain me how frequency filter can introduce limitedsharpen?

the only difference which could occur is that fft3d sharpen is only 2D and LSF + Soothe is 3D as Soothe reduce the sharpen strength for motion part of the film.

[Fizick]

sharpening is increasing of high frequencies.
limited sharpening is limited increasing of high frequencies.
limited sharpening is coarcely simulated in limitedsharpen script in spatial domain with operations with neighbors of points ;)

[fjhdavid]

thank you!

does it mean, finally, that fft3dfilter is doing exactly the same job as LSF, except that fft3dfilter works in frequencies domain when LSF work in spatial domain?

and Soothe can be used with fft3dfilter in fact, in order to add a temporal dimension to sharpening?


and you say that:

"Since v.1.7, Gaussian High Pass Filter with variable cutoff frequency is used for sharpening"

does it replace the v1.1 sharpening method you mention in your documentation, or is it an additional filter?

[Fizick]

1. not exactly the same. I do not study LS, but results are differ.
And it is possible to create a lot various implementations of LS.
2. I amm not sure what is "Soothe" method, but probably it si possible to use temporal information for sharpening, but IMO it is sort of terminology (sharpening versus denoising versus superresolution, etc).
3. as i remember, it replaces old method (implementation), but old (since v.1.1) description is correct.

[fjhdavid]

Fizick, just for info about this Soothe "postprocessor" plugin from Didee:

"Soothe() does a very simple job: get the difference between [source] and [sharpened source], apply a TemporalSoften on this difference BUT allow only changes towards 128 ("neutral"), and then apply this temporally calmed difference back to the original clip. Effectively, this will reduce the overall effect of sharpening - less in static areas, and more in moving areas."


do you think It could be helpful as a fft3dfilter postprocessor?


ps: I think your frequency domain approach is really the most stable approach as results are constant and almost predictive and linear regarding each parameters (and in fact almost only 2 (end user) parameters sigma and sharpen)!

[egrimisu]

DUMB QUESTION: what will happen if i set the ow=16, oh=16 to ow=8, oh=8 will it increase the overall quality?

[LaTo]

DUMB QUESTION: what will happen if i set the ow=16, oh=16 to ow=8, oh=8 will it increase the overall quality?

It will decrease overall quality (if you don't touch at bh/bw)

[MADAJ]

hi, a very basic error.

I am missing something in my plugin folder.

here is the error

http://img67.imageshack.us/img67/1870/99507685ma9.jpg

i have loaded fftw3.dll, FFT3DFilter.dll, and libfftw3-3.dll in my AVS plugin dictionary.

but still I get this error.


=====

and what I am aiming for is to get rid of the blue area over the brown guy in this picture.

http://img67.imageshack.us/img67/9370/59062104wf4.png

any suggestions?

[XhmikosR]

You don't have to load these dll in your avs script (only FFT3DFilter.dll but if it's in avisynth plugins folder you don't have to, it's loaded automatically). You have to place fftw3.dll in your system32 folder.
4. Uses fast external FFTW library version 3 (http://www.fftw.org)
as Windows binary DLL (compiled with gcc under MinGW by Alessio Massaro),
which support for threads and have AMD K7 (3dNow!) support in addition to SSE/SSE2.
It may be downloaded from ftp://ftp.fftw.org/pub/fftw/fftw3win32mingw.zip
You MUST put FFTW3.DLL file from this package to some directory in path (for example, C:\WINNT).
Filter will NOT work without it!

[egrimisu]

what if i set them to 8 aswell?

It will decrease overall quality (if you don't touch at bh/bw)

[LaTo]

what if i set them to 8 aswell?
Max quality is overlap = blocksize/2. (32-16 & 16-8 are good value)

[Monamona]

Under YUY2, it seems that FFT3DFilter 2.11 with plane=4
makes Red color deeper.

Please See below.

Correct Image:
http://blog-imgs-41.fc2.com/f/e/l/felmina/ss000000117.jpg

Deeper Red:
http://blog-imgs-41.fc2.com/f/e/l/felmina/ss000000116.jpg

[Fizick]

:script:
(simplest to show effect)

EDIT: I see the script :)
FFT3DFilter(plane=4, bt=3, bw=3, bh=2, ow=1, oh=1, svr=0)

Well, it is really small block size. Never checked this.

[Fizick]

I can not confim chroma dif in my tests.
Try to add coloryuv(analyze=true) in some places to search the reason.
and use as simple test script as possible!

[Monamona]

Thank you for your fast reply.

I simply got this info from some website.
I will re-check and collect more samples.

[saemonz]

I can't find download address？Where is it

[cwk]

I can't find download address？Where is it


http://avisynth.org.ru/fft3dfilter/fft3dfilter.html

[LigH]

Is it just my darn satellite connection here, or is the FFTW FTP server down? Can anyone else get a Win32 version of the FFTW3.dll for me?

http://avisynth.org.ru/fft3dfilter/fft3dfilter.html => ftp://ftp.fftw.org/pub/fftw/fftw3win32mingw.zip => Timeout
http://www.fftw.org/install/windows.html => ftp://ftp.fftw.org/pub/fftw/fftw-3.2-dll.zip => Timeout

[Boulder]

The FTP server seems a bit slow, but here's the file you asked for:

http://www.megaupload.com/?d=LCVJ593H

[LigH]

Thank you. Hooray for mirrors!
__

Meh. I found the reason... My preferred browser uses a proxy (website development environment, proxy usage circumvents the "hosts" file which declares localhost to be the domain I develop the website for, so I can compare test and public server easily). This proxy could not load from this FTP server.