New very slow FFT denoiser

[Isochroma]

Yeah, I thought about that problem, and came up with two solutions:

1. Tell the spacial denoiser to operate on only one axis.

2. Rotate back to normal & then rotate the other way and denoise again, thus balancing x-y denoising. Each pass would be set to 1/2 strength. Only problem is that the temporal dimension would always get 2x the amount of denoising as spacial...

It's always so fun to think of weird ways of doing things, and then weird solutions to the strange problems they generate!

Wow that clip has some really nice noise... I will do voodoo magics with it if I dare!

[DarkFoon]

@ didee
have you tried my FFT3D settings? On delicate anime sources (fine detail, not something like Harvey Birdman) the detail retained is outstanding and the noise reduced was more than I thought possible with FFT3d.

as for the banding on gradients, I had that happen recently. It was irritating. My source was very noisy, but when I ran FFT3D it really messed up this one gradient and made it into a bunch of concentric rings of different shades.
I'll post a before&after as soon as I have it made

[redfordxx]

or the noise *is* reduced (around sigma=7~8), but detail is killed.
Don't use sigma, use patterns...
the difference in quality is huge.
That'swhy I made all these scripts...

Although, it depends on the noise... When it is dots, it's ok.
This here can have up to 8pixel long lines which is difficult to catch as a pattern. However, still quite good job compared to sigma approach.

[Didée]

@ didee
have you tried my FFT3D settings?
I did right now. Most noise is gone, except on some movement. Faces get plastic. Cloths turn into plastic. Fog/nebula turns into milk. And where detailed foreground meets 'flat' background, it leaves back mosquito noise on the background, in rectangular shapes.


@ redforxx

Patterns you say. Where every scene has different amounts & (partly) characteristics of noise (it's basically film grain, after all. Not mpeg artefacts out of thin air, and also not that noise like in analog captures.)
One scene shows noise like that. Next scene is dark, almost without any noise. Next-next scene is even worse than the 1st.
Noise Patterns? Manually set up for ~40 scenes per episode per 22 episodes per 5 seasons?

No, thanks. It's better to produce a general solution. More fun for me, and of more use for others. (somewhen.)

[redfordxx]

as for the banding on gradients, I had that happen recently. It was irritating. My source was very noisy, but when I ran FFT3D it really messed up this one gradient and made it into a bunch of concentric rings of different shades.
That's why I started to use patterns instead of sigma.

[Dark Alchemist]

Thank you didee I always get those mixed up. So the spatial is in 3d and the temporal is the 2d?

[redfordxx]

Original (http://img375.imageshack.us/img375/6075/original1ap.png)
Denoised (http://img375.imageshack.us/img375/5632/fft9cf.png)
Following are Subtract images with 25xcontrast enhancement
Original vs orig-blurred(5x5) (http://img375.imageshack.us/img375/118/g55wh.png)
Denoised vs denoised-blurred(5x5) (http://img355.imageshack.us/img355/5658/f0g51wy.png)
Orig vs. denoised (http://img355.imageshack.us/img355/3407/s01pr.png)
You see the noise disappeared fairly... (strange it is in 16x16blocks..)
On the Orig vs. denoised you see, that the edges not much visible... so, preserved.
On the Denoised vs denoised-blurred you see the vertical lines (sometimes horizontal) I was speaking about. Somehow I must get rid of them.

[redfordxx]

Thank you didee I always get those mixed up. So the spatial is in 3d and the temporal is the 2d?
spatial is 2d
temporal 1d
spatio-temporal 3d

(speaking about flat screens and oldfashioned cinemas...;)

[redfordxx]

Where every scene has different amounts & (partly) characteristics of noise (it's basically film grain, after all. Not mpeg artefacts out of thin air, and also not that noise like in analog captures.)
One scene shows noise like that. Next scene is dark, almost without any noise. Next-next scene is even worse than the 1st.
Noise Patterns? Manually set up for ~40 scenes per episode per 22 episodes per 5 seasons?
Of course, I assumed there is one or two noise patterns per movie.
OTOH, sometimes same noise in different colour or brightness looks different. I though also there are many noises in the movie, until I separated it like in the example Original vs orig-blurred(5x5)

[Mug Funky]

i find sigma to be not precise enough, but patterns to be too precise... you can often see the signature of the pattern in a grid in the output.

the 4 different sigma values are a good compromise - course but not too course. they work pretty well on (grainier than Didée's b5) animation. i will definitely have to post a sample of this noise... the good news with animation of this kind (ie woefully simple) is it doesn't matter if too much noise gets removed, the bad news is the signal to noise ratio is quite a bit below 1... (ie, more noise than signal).

[edit]

oh, and Didée, i wouldn't be too worried about manual NR settings per scene... if i remember that show correctly there were lots of CG sequences which will have zero film grain (only either analog noise or beta SP noise or both), so you'll have to do some chopping anyway.

[Didée]

the 4 different sigma values are a good compromise - course but not too course. they work pretty well on (grainier than Didée's b5) animation.
Well, if you have more noise than I have - fine for you, congratulations! :D
But then, your animations probably don't contain human faces which could end up looking like a bank robber wearing a stocking mask ...

if i remember that show correctly there were lots of CG sequences which will have zero film grain..., so you'll have to do some chopping anyway. The good thing about a good denoiser is that it does only little harm if no noise is present. ;)
(Worse is that the CG's are 'cbbc' fieldblended ... 30p->25i ... DOWN WITH NTSC!)

[Dark Alchemist]

Right temporal being along the timeline but 2d and 3d from a flat scene I still do not understand. I mean how can you makeup a 2nd dimension from a flat source (like film)?

[foxyshadis]

You can think of it as every 2d frame in the video stacked on top of each other, with a temporal width of 40ms or whatever the framerate happens to be. The metaphor isn't perfect, but it's good enough to enable effective temporal filtering and motion vector encoding within a scene.

[Dark Alchemist]

Huh?

So, every frame of the video is stacked one on top of the other then you lost me.

[DarkFoon]

I may not be a moderator or anything, but it seems that we've got 2 different conversations going on here that don't necesarily belong in FFT3D. One belongs in a thread about denoising technologies. And the other Should be in a denoising help thread of some kind.
That being said...
Didée,
I guess I forgot to mention that my FFT3D settings I use on anime. I tried them on a Law & Order: CI episode and they worked well. Especially when not going frame-by-frame (which is a hard thing for me NOT to do)
My cable signal is garbage (especially the L&W episode) tons of noise, and its made worse by the fact that I use a hauppauge winTV-250 which uses a hardware mpeg-2 encoder, so I get lots of that *fun* mpeg-2 artifacting that emulates the "film grain" (in my case, analog crap noise) in the signal. In anime, this is annoying as hell, so I kill it with fire. Anyways, I'm rambling here and I've forgotten where I was going with this... OH! You might want to try peachsmoother (if you haven't already). When FFT3D lets me down (by smoothing away quality or whatnot) peach does an excellent job, because it leaves the detail or "noise" in the image, it makes it static, though. (this is a bad description so don't let it discourage you from trying it if you haven't already) So there is no extra motion(except that actual motion) and the detail is retained. I used it on everything until I found FFT3D.

Anyways.
Good luck!

[levi]

My plugins are at my site http://bag.hotmail.ru and its mirror at http://www.avisynth.org/fizick


I can't find any working links for 1.8.5. Warpenterprises seems to be only 1.6. Any suggestions?

[Revgen]

Download here. (http://www.avisynth.org/fizick/fft3dfilter/fft3dfilter185.zip)

[levi]

that works for you? I'm getting 404.shtml

[DarkFoon]

@levi
I get teh same thing when I click the avisynth link,
but fizick's site works just fine for me.
just scroll down, click on FFT3D, then scroll down until you see the download links at the very bottom.

[foxyshadis]

DF, you should try clicking the links first. ;) Fizick removed the downloads today, for what reason isn't clear yet, but hopefully they'll be back soon.

Fortunately I am not lacking for webspace. Download latest version here (http://foxyshadis.slightlydark.com/random/fft3dfilter185.zip).

[Egh]

You haven't read the site yet? :O Orly? :)

In short: Fizick went donationware. I.e. at the moment all download links are removed from the site.

Long explanation:

http://bag.hotmail.ru/donate.dhtml (with some errors in english though his russian is not really that clear as well :P)

Most important part (text improvements, based on russian version as well, are in bold ^^)

I temporary remove (removed) all my plugins from my site and change (changed) license of all my plugins to donation-ware.
I will make announcement of new developed (for myself) filters versions (i will announce new versions of the filters ) and wait some sum of donations ( and I expect to receive some amount of donations upon announcement). All donors will become the registered users who will immediately get ( receive after donating ) current and next ( all the following ) beta versions of this filter and (will receive) my technical support (by e-mail or private messages). I will minimize the technical support at public forums.

I guess you won't see Fizick here anytime soon :) Nor new versions of fft3d and other filters :(

BTW, not mentioned in english version whatsoever, but in russian it's clear: the sum of money per donation is expected to be no less than 7$ :)

[Revgen]

I don't mind donating, as long as he uses the money to pay for english language courses.:D

[Wilbert]

Hmm, that's bad news :( Last coupled of days I got the feeling something was wrong. Luckely his intention is to keep releasing future versions as OS.

Does someone has his latest mvtools including source (1.0.3 i think)?

[Boulder]

It can be found at Didée's online directory:

http://home.arcor.de/dhanselmann/_stuff/

[BigDid]

http://www.avisynth.org/fizick/
still online
last fft3d 1.85

For mvtools links to http://manao4.free.fr/
but rev 0.991 only

Anyway good luck to Fizick for his projects

Did

[Fizick]

Revgen,
Yes, my English is bad :(.
But if your English is good,
why you never suggest me your help in filters manuals improving (editing)?

[Revgen]

Your English has never been terrible, but it has gotten a lot better since the first time you started posting here. It will improve as you keep using it.

Heck, I've been speaking English all my life and I still haven't figured it out yet.:)

It was kind of a half-joke.



Anyhoo, you're manuals are okay as far as English goes, but they aren't as visually detailed as they could be.

The plugins that you've programmed are very powerful yet also complicated, and any kind of visual cues that you could show us could help make them easier to understand.

For example, in the DeFreq manual it would be better to have a grid pattern graph overlaying a .JPG vidcap to show people a better example of how to set X and Y window values. Giving examples of which settings in Defreq affect which areas in the grid pattern could help people understand the settings easier.

[Egh]

Revgen,
Yes, my English is bad :(.
But if your English is good,
why you never suggest me your help in filters manuals improving (editing)?

First, I don't recall if you actually asked here for that :)

Second, the problem as I see it... Well that the original (aka russian manuals) also need some overhaul aka editing. They certainly need better structural representation, layout and even language corrected.

I recall once I read some paragraph in fft3d manual (in russian :P) and had to read some obscure phrase 3 times finally to get the meaning behind it.

As for DegrainMedian manual, for example, take that part which describes the method used, and e.g. reference to PixieDust has very vague meaning. And I'm very interested in pixiedust comparison and still hope someone would finally implement it as YV12 avs 2.5 plugin. But if you plugin does that functionality, it's very good. But to what extend, how good and so on are quite tough to understand from the manual.

As for help with translation, post the request what to do and I'll take a look.

[Mug Funky]

@ fizick:

i use your stuff commercially, but have no problem with donating :)

i have no credit card at the moment (never liked the idea to be honest), but i'll figure something out. i'll have to pore through the shareit site to see what other options i have.

FWIW, i never really had a problem with your docs - it all made sense, even if i had to re-read a couple of things. structure and flow are all good, and grammar was better than a lot of native english speakers can produce. the good thing about language is there's so much redundancy that context-based error correction is quite easy :)

[Soulhunter]

Hi Fizik! Some time ago Didèe pointed out that FFT3DFilter produces visible ringing in some cases, so I checked his presumption...

Source / FFT3DFilter(BW=48,BH=48,Sigma=5)

http://img13.imageshack.us/img13/1804/normal7fi.th.png (http://img13.imageshack.us/img13/1804/normal7fi.png) http://img13.imageshack.us/img13/1372/zoom9ii.th.png (http://img13.imageshack.us/img13/1372/zoom9ii.png)

Its even more visible in motion: Source (http://www.mytempdir.com/398663) / Filtered (http://www.mytempdir.com/398654)

The only way to get rid of the ringing is to lower the blocksize to something like 8, but then the denoising is to way to weak. So, is there maybe a way to modify FFT3DFilter so that its produces less (or none) ringing?


Tia n' Bye

[DarkFoon]

have you tried different wintype= settings?
2 apparently gives less ringing.

[Didée]

Hmh no, that just makes the ringing more mosquito-like, and masks it by less overall noise reduction compared to wintype=0|1. (Seems like I have to raise sigma's by *1.5 to *2 to get the same NR from wintype=2 as from 0|1 - and if doing so, more detail gets wiped.)

IMO it's a inherent conceptual issue of the window'ed frequency analysis. If there's a detailed foreground object against a flat background, it gets difficult to seperate the frequencies of foreground's detail from those of background's noise, for where both fall into the same window.
Only way to reduce the artefact is to decrease window size ... but going "small enough" will hurt the denoising efficiency.

A source frame with some noise:

http://img8.imageshack.us/img8/7918/thesourceforfft3d4no.th.png (http://img8.imageshack.us/my.php?image=thesourceforfft3d4no.png)



A script with some settings:
Mpeg2Source("I:\_UnderWork\Babylon5\2x05 - Der unsichbare Feind\vts_01.d2v")
crop(0+40,88,-0-40,-88,true) # reduce size for stacking several outputs

o = last

fft1 = o.fft3dfilter(sigma=3,sigma2=4,sigma3=4,sigma4=3,bt=4,bw=32,bh=32,ow=16,oh=16,degrid=1,wintype=0,plane=4)
fft2 = o.fft3dfilter(sigma=3,sigma2=6,sigma3=6,sigma4=3,bt=4,bw=32,bh=32,ow=16,oh=16,degrid=1,wintype=1,plane=4)
fft3 = o.fft3dfilter(sigma=4,sigma2=7,sigma3=7,sigma4=4,bt=4,bw=32,bh=32,ow=16,oh=16,degrid=1,wintype=2,plane=4)

o2 = o.fluxsmoothST(20,2).temporalsoften(2,24,24,24,2)

stackhorizontal( stackvertical(o2 .subtitle("source (heavvy TS)"),fft3.subtitle("wintype=2")),
\ stackvertical(fft1.subtitle("wintype=0"), fft2.subtitle("wintype=1")) )

return last.yv12lut("x 24 +",U=2,V=2) # a dark monitor I have ;)
(Note the sigma's increase along with wintype.)

The result:

http://img41.imageshack.us/img41/2609/fft3dwintypes2xh.th.png (http://img41.imageshack.us/my.php?image=fft3dwintypes2xh.png)

Look around all edges of faces and uniforms, and in particular beneath the emblem on Ivanova's sleeve.

[Lemonzest]

where can i download version 1.85 (latest??) cant get it any where :(

[Wilbert]

I attached version 1.85 for you.

[Lemonzest]

thanks :)

[Fizick]

As for help with translation, post the request what to do and I'll take a look.

I do not know what is bad in current translations. So, I can not request. But any specific (concrete) suggestions and corrections to any filters manuals are welcome.
Such small corrections I can put to manuals.


Soulhunter,
I know about this "ringing". It is compromise (as was noted by Didee)

[Egh]

I do not know what is bad in current translations. So, I can not request. But any specific (concrete) suggestions and corrections to any filters manuals are welcome.
Such small corrections I can put to manuals.



Well generally it's needs more editing, even in russian. So I'd suggest you first find someone volunteering to edit manuals to make them more readable and comprehensive. Not that they are bad for freeware, lots of programs don't have any manuals whatsoever ^^ But I'd say for any commercial software that documentation would be far from being enough. For example that fragment from DegrainManual referencing PixieDust. I probably still don't understand fully the meaning of that phrase :)

[Soulhunter]

Soulhunter,
I know about this "ringing". It is compromise (as was noted by Didee)
So, nothing that could be done about it? :\


Bye

[Soulhunter]

FFT3DFilter(Sigma=5,BW=48,BH=48) (http://img229.imageshack.us/img229/1606/fft3dfiltersigma5bw48bh480tm.png) / Zoomed (http://img229.imageshack.us/img229/7082/zoomfft3dfiltersigma5bw48bh488.png)

FFT3DFilter(Sigma=3,BW=16,BH=16).Deen("c3d",1,3,3,25) (http://img229.imageshack.us/img229/6831/fft3dfiltersigma3bw16bh16deenc.png) / Zoomed (http://img229.imageshack.us/img229/8734/zoomfft3dfiltersigma3bw16bh16d.png)

Seems to work, equal compressibility but near like no ringing... ^_^


Bye

[Soulhunter]

Wouldnt it be possible to use smaller blocks around edges and big blocks for the rest?

http://img98.imageshack.us/img98/1823/normal7fi9dp.png


Bye

[Didée]

Wouldnt it be possible to use smaller blocks around edges and big blocks for the rest?

http://img98.imageshack.us/img98/1823/normal7fi9dp.png


Bye

Hihi ... in an ancient time where a basic DOS-shell just had appeared ("Windows", 15 to 20 years ago), this method was already used in FractInt. They called it "tesselation".

Basically, the method should be implementable into FFT3D. Use a robust edgemask like Prewitt and perform tesselation on the basic blocks, breaking a new piece when a boundary of a tesseled piece hits an edge value bigger than <threshold>.

[tsp]

it might be a problem to get blocks of different size to overlap the right way (that is a gentle roll-off to black at the edge and when the overlapped blocks are added togetger it should give the same intensity)

[Fizick]

Recently new version 3.1 of fftw library is released at http://www.fftw.org
Provided new binary (after renaming to fftw3.dll ) can be used with fft3dfilter,
but the speed is not faster with my Athlon.

[zyrill]

Regarding Soulhunter's problem with ringing around the edges: I encountered the same problem while using FFT3D but found a way to counter it: I now invoke FFT3D twice, once with a block size of 48 and once with a block size of 4. It's dreadfully inefficient but I don't really care since I want the best possible quality and using FFT3DGPU framerate is not a concern anymore (following values only for reference since taken from VDub while encoding to XviD. FFT3D: ~2.5fps; FFT3DGPU: ~12fps working on a 712 px by 420 px file).
Prior to looking more deeply into FFT3D I tried the whole spread of denoising-solutions available and after having seen that FFT3D yielded the best results, I tried refining the parameters to achieve the best possible result. The result is actually so good that, given any (even a recent) DVD, the output of the avisynth script looks much cleaner and clearer then the input - it's like having pulled the curtains away. And while getting rid of all the noise, no detail is lost! It's really most amazing and I'm most thankful that such a plugin exists.
Anyway, without much further ado, here's the script I ripped together:
# Denoising every color plane discretely by multiple FFTs using motion compensation
Motion=DepanEstimate(trust=2.5, fftw=true)
DepanInterleave(data=Motion)
YToUV(FFT3DGPU(sigma=3, plane=1, bw=32, bh=32).UToY,\
FFT3DGPU(sigma=3, plane=2, bw=32, bh=32).VToY,\
FFT3DGPU(sigma=2, plane=0, bw=32, bh=32))
YToUV(FFT3DGPU(sigma=3, plane=1, bw=4, bh=4).UToY,\
FFT3DGPU(sigma=3, plane=2, bw=4, bh=4).VToY,\
FFT3DGPU(sigma=2, plane=0, bw=4, bh=4))
SelectEvery(3,1)
I hope you'll find it equally useful.

Cheers!

[ariga]

Recently new version 3.1 of fftw library is released at http://www.fftw.org
Provided new binary (after renaming to fftw3.dll ) can be used with fft3dfilter,
but the speed is not faster with my Athlon.
The new library didn't work even after renaming! Strange... Is a reboot required ?

[Fizick]

ariga, see here
http://forum.doom9.org/showthread.php?p=781111#post781111

[ariga]

Thanks. Used the *f.dll and it works fine. Wonder why the other .dll didn't work.

[Fizick]

Other new dll includes not single prcecision, but double precision functions with other (different) names and calling parameters.

[AI]

2 zyrill

very small comments, becose I am not English man
------------- First --------------

# Denoising every color plane discretely by multiple FFTs using motion compensation
Motion=DepanEstimate(trust=2.5, fftw=true)
DepanInterleave(data=Motion)
MergeChroma(FFT3DGPU(sigma=2, plane=0, bw=32, bh=32),\
FFT3DGPU(sigma=3, plane=1, bw=32, bh=32))
SelectEvery(3,1)

(see Bold) - becose in GPU version
"plane=1" = "plane=2" = "plane=3" - alwase use both U and V

-------- Second ----------
if you use 2 or more temporal filters in
DepanInterleave(...)
....
SelectEvery(...)needDePanInterleave(...)
filtre1()
selectevery(3,1)
DePanInterleave(...)
filtre2()
selectevery(3,1)
DePanInterleave(...)
filtre3()
selectevery(3,1)orDePanInterleave(prev=3, next=3)
filtre1()
filtre2()
filtre3()
selectevery(7,3)orDePanInterleave(prev=1, next=1)
MergeChroma(filtre1(),filtre2())
selectevery(3,1)

[leonid_makarovsky]

I've tried the latest version of Fizick's fft3d filter plugin. I found it better than the previous versions. I can't measure how fast it is alone 'cause I use it with bunch of other stuff. Yet, I am not using any sharpening features on that one. And yet I do NOT use fft3dfilter alone. Before that I use PixieDust for VHS noise removal.

This is the code fragment I use to clean the factory made NTSC or PAL interlaced VHS cassettes with new like condition:

myclip = SeparateFields(myclip)
myclip = PixieDust(myclip, limit = 4)
myclip = fft3dfilter(myclip, sigma = 2, bt = 4, plane = 0, sharpen = 0, svr = 0)
myclip = LimitedSharpen(myclip, ss_x = 1, ss_y = 1, Smode = 3, dest_width = 352)
myclip = Weave(myclip)

Separating fields and using fft3dfilter as progressive gives me better results for my projects.

If I have a progressive scan PAL VHS, then I am now trying to decide whether to treat as if it's interlaced or to do something like this:

# PixieDust cleans 2 separate fields of a progressive frame
# better then it cleans a progressive frame
myclip = SeparateFields(myclip)
myclip = PixieDust(myclip, limit = 4)
myclip = Weave(myclip)

# use fft3dfilter to feed a progressive frame
myclip = fft3dfilter(myclip, sigma = 2, bt = 4, plane = 0, sharpen = 0, svr = 0)

# LimitedSharpen sharpens 2 separate fields of a progressive
# frame better then it sharpens just a progressive frame
myclip = SeparateFields(myclip)
myclip = LimitedSharpen(myclip, ss_x = 1, ss_y = 1, Smode = 3, dest_width = 352)
myclip = Weave(myclip)

--Leonid

PS. I feed this code to TMPGEnc. 97 minutes and 18 seconds of PAL video takes 28 hours to process from 704x576 Huffyuv AVI.