Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 24th June 2012, 02:26   #821  |  Link
librarian
Registered User
 
Join Date: Nov 2011
Posts: 63
This sample script should give a (approximately) Wiener only temporal filter:

Code:
avisource("")
clp=last
fq=fft3dfilter(sigma=6,bt=1,plane=0)#Use current frame data only
ft= fft3dfilter(sigma=6,bt=5,plane=0)#bt= 2...5
difffq=mt_makediff(clp,fq,y=3,u=2,v=2)#only luma is filtered then u=v=2
tmp=mt_adddiff(ft,difffq,y=3,u=2,v=2)#but this remove the filtering on current frame too

return tmp
bt=0 is the "Temporal Kalman filter for spectrum data" (see the doc).

Last edited by librarian; 24th June 2012 at 02:58.
librarian is offline   Reply With Quote
Old 24th June 2012, 11:27   #822  |  Link
dadix
Registered User
 
Join Date: Aug 2011
Posts: 49
thank you

@Fizick
for noise pattern can be implemented to have a option to add a frequency range and multiple areas of frequency range?
dadix is offline   Reply With Quote
Old 20th March 2013, 07:19   #823  |  Link
Jenyok
Warm and fuzzy
 
Join Date: Apr 2010
Location: Moscow, Russia
Posts: 201
There are some questions to you.
.
There is a function RemoveGrain(mode=17) .
.
Could I use function FFT3dFilter(...) instead of function RemoveGrain(mode=17) to achive equal result of function RemoveGrain(mode=17) (result FF3dFilter(...) = result RemoveGrain(mode=17)) ?
Which parameters and values of those parameters could I use in function FFT3dFilter(...) to achive this result ?
.
.
There are another similar questions.
.
There is a function RemoveGrain(mode=1) .
There is a function RemoveGrain(mode=2) .
There is a function RemoveGrain(mode=3) .
There is a function RemoveGrain(mode=5) .
.
result FF3dFilter( ??? ) = result RemoveGrain(mode=1) ?
Parameters and values of those parameters to function FFT3dFiler() in this case ?
.
result FF3dFilter( ??? ) = result RemoveGrain(mode=2) ?
Parameters and values of those parameters to function FFT3dFiler() in this case ?
.
result FF3dFilter( ??? ) = result RemoveGrain(mode=3) ?
Parameters and values of those parameters to function FFT3dFiler() in this case ?
.
result FF3dFilter( ??? ) = result RemoveGrain(mode=5) ?
Parameters and values of those parameters to function FFT3dFiler() in this case ?
.
FFT3dFilter( ??? ) similar to and another "mode=" of RemoveGrain() function ?
mode=7
mode=9
mode=11
mode=19
.
Thanks.
.
__________________
Warm and fuzzy (white and fluffy)

Last edited by Jenyok; 20th March 2013 at 08:37.
Jenyok is offline   Reply With Quote
Old 28th March 2013, 13:14   #824  |  Link
Jenyok
Warm and fuzzy
 
Join Date: Apr 2010
Location: Moscow, Russia
Posts: 201
And there is a silence. ??? !!!
.
Didee, Gavino, Fizick
.
Where are you, mates ?
.
__________________
Warm and fuzzy (white and fluffy)

Last edited by Jenyok; 28th March 2013 at 13:16.
Jenyok is offline   Reply With Quote
Old 21st January 2017, 20:58   #825  |  Link
aerDNA
Registered User
 
aerDNA's Avatar
 
Join Date: Apr 2009
Location: Rijeka, Croatia
Posts: 10
Sorry for necrobumping like this but plugin documentation points to this thread and I wanted to share some insight that might be helpful to new users (if there are any) in setting it up.
This is an excellent denoiser and my no. 1 choice for a long time now. It used to be painfully slow if you wanted to squeeze the most out of it but on today's powerful CPUs it performs ok. I mainly wanted to comment on default/recommended values vs what I use.

- Documentation suggests sigma=1.5-2.5 for digital sources and 3 or more for analog. If detail retention is a priority, my recommendation is not to go above 2, unless you're encoding at low bitrate in which case fine detail is gone anyhow and stronger denoising will help compressibility. I normally use 1.5; it removes a substantial amount of noise without compromising on quality.

- Chroma planes tolerate much stronger denoising than luma so it's a good idea to process them separately. sigma=1.5, plane=0 (luma) + sigma=3, plane=3 (chroma) will be visually indistinguishable from sigma=1.5, plane=4 (all) but it will compress better. If you're afraid it doubles the time required because you need to apply FFT3DFilter twice, don't worry, it takes the same.

- Suggested optimal block size/overlap is bw,bh=32, ow,oh=16. This was written years ago with tolerable speed in mind, I presume. With a modern CPU, use bw,bh=16, ow,oh=8. It produces superior results.

- For 3D Wiener, bt=5 is suggested as best. This is actually the main thing that made me write this post, because I've always been using bt=5 and only today did I do some proper experimenting with this value. After quite a bit of testing, one thing I can objectively say is that bt=5, compared to default bt=3, consistently produces frames that are slightly less compressible. This means it washes out less data, but what is the nature of it in terms of noise/detail? Visual inspection of frames revealed no difference in most cases, and where there was a visible difference, quality seemed to be on the side of bt=3, surprisingly. So my conlusion is that bt=5 actually has slightly worse noise/detail removal ratio. Even if I'm wrong on this, perceptible difference really is insignificant, and considering that bt=5 comes with a pretty massive speed penalty, I say stick with the default.

- On multicore, be sure to set the ncpu= param, you'll get a decent speed boost.

tl;dr - my "best" settings:
FFT3DFilter(sigma=1.5, plane=0, bw=16, bh=16, ow=8, oh=8, ncpu=x)
FFT3DFilter(sigma=3, plane=3, bw=16, bh=16, ow=8, oh=8, ncpu=x)

Finally, use the ICL build by Groucho2004 - it's faster and there's also a 64-bit build for AviSynth+. To use both versions on Win64 (I use 32-bit for scripting/preview in VirtualDub and 64-bit for final encoding) you also need both libfftw3f-3.dll versions from fftw.org. Renamed to fftw3.dll, 64-bit dll goes to Windows\System32 and 32-bit dll goes to Windows\SysWOW64.
aerDNA is offline   Reply With Quote
Old 21st January 2017, 22:36   #826  |  Link
manono
Moderator
 
Join Date: Oct 2001
Location: Hawaii
Posts: 7,406
Thank you very much for the tips. I use the filter a lot and always at the 'best settings', usually with Sigma set for 1.5 or 2.0. I'll try out your suggestions the next time I use it.
manono is offline   Reply With Quote
Old 21st January 2017, 23:33   #827  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by aerDNA View Post
To use both versions on Win64 (I use 32-bit for scripting/preview in VirtualDub and 64-bit for final encoding) you also need both libfftw3f-3.dll versions from fftw.org. Renamed to fftw3.dll, 64-bit dll goes to Windows\System32 and 32-bit dll goes to Windows\SysWOW64.
Just a little remark - With the ICL builds I made renaming the fftw DLLs is not necessary, they accept either libfftw3f-3.dll or fftw3.dll.
__________________
Groucho's Avisynth Stuff

Last edited by Groucho2004; 22nd January 2017 at 15:04.
Groucho2004 is offline   Reply With Quote
Old 22nd January 2017, 09:59   #828  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Quote:
Originally Posted by aerDNA View Post
Sorry for necrobumping like this but plugin documentation points to this thread and I wanted to share some insight that might be helpful to new users (if there are any) in setting it up.
This is an excellent denoiser and my no. 1 choice for a long time now. It used to be painfully slow if you wanted to squeeze the most out of it but on today's powerful CPUs it performs ok. I mainly wanted to comment on default/recommended values vs what I use.

- Documentation suggests sigma=1.5-2.5 for digital sources and 3 or more for analog. If detail retention is a priority, my recommendation is not to go above 2, unless you're encoding at low bitrate in which case fine detail is gone anyhow and stronger denoising will help compressibility. I normally use 1.5; it removes a substantial amount of noise without compromising on quality.

- Chroma planes tolerate much stronger denoising than luma so it's a good idea to process them separately. sigma=1.5, plane=0 (luma) + sigma=3, plane=3 (chroma) will be visually indistinguishable from sigma=1.5, plane=4 (all) but it will compress better. If you're afraid it doubles the time required because you need to apply FFT3DFilter twice, don't worry, it takes the same.

- Suggested optimal block size/overlap is bw,bh=32, ow,oh=16. This was written years ago with tolerable speed in mind, I presume. With a modern CPU, use bw,bh=16, ow,oh=8. It produces superior results.

- For 3D Wiener, bt=5 is suggested as best. This is actually the main thing that made me write this post, because I've always been using bt=5 and only today did I do some proper experimenting with this value. After quite a bit of testing, one thing I can objectively say is that bt=5, compared to default bt=3, consistently produces frames that are slightly less compressible. This means it washes out less data, but what is the nature of it in terms of noise/detail? Visual inspection of frames revealed no difference in most cases, and where there was a visible difference, quality seemed to be on the side of bt=3, surprisingly. So my conlusion is that bt=5 actually has slightly worse noise/detail removal ratio. Even if I'm wrong on this, perceptible difference really is insignificant, and considering that bt=5 comes with a pretty massive speed penalty, I say stick with the default.

- On multicore, be sure to set the ncpu= param, you'll get a decent speed boost.

tl;dr - my "best" settings:
FFT3DFilter(sigma=1.5, plane=0, bw=16, bh=16, ow=8, oh=8, ncpu=x)
FFT3DFilter(sigma=3, plane=3, bw=16, bh=16, ow=8, oh=8, ncpu=x)

Finally, use the ICL build by Groucho2004 - it's faster and there's also a 64-bit build for AviSynth+. To use both versions on Win64 (I use 32-bit for scripting/preview in VirtualDub and 64-bit for final encoding) you also need both libfftw3f-3.dll versions from fftw.org. Renamed to fftw3.dll, 64-bit dll goes to Windows\System32 and 32-bit dll goes to Windows\SysWOW64.
If you do care about quality, fft3d is really just a pretty bad choice, it's 2017 now and there are denoising algorithms out there MUCH more advanced than simple dumb frequency domain filtering (and also DFTTest is better quality wise in this category).

Some better options:
Motion Compensation (MDeGrain)
NLMeans (KNLMeansCL, TNLMeans)
BM3D (available in vaporsynth)
feisty2 is offline   Reply With Quote
Old 22nd January 2017, 18:48   #829  |  Link
aerDNA
Registered User
 
aerDNA's Avatar
 
Join Date: Apr 2009
Location: Rijeka, Croatia
Posts: 10
DFTTest is marginally better but much slower. TNLMeans is better but much, much, much slower. When you enable temporal (it's not nearly as good without it) it goes from unacceptably slow to 48hrs./encode slow. KNLMeansCL should be faster but I'm without a discrete GPU at the moment and apparently it lacks temporal mode. I wanted to give MDeGrain a go but it seems I'd have to do some digging through forum just to piece together the right plugin and dependencies that have different versions and mods of their own. I also don't feel like getting into VaporSynth because there's nothing I really need but don't get from AviSynth(+), which I'm accustomed to, so no BM3D for me. I will test KNLMeansCL and if turns out to be a spectacular performance improvement over TNLMeans while offering same quality, I'll gladly go with that. Right now I don't see anything offering me the kind of quality vs speed balance that FFT3D does.
aerDNA is offline   Reply With Quote
Old 22nd January 2017, 19:08   #830  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Quote:
and apparently it lacks temporal mode.
Apparently the wrong fact.
KNLMeansCL is a 3D NLMeans filter, the "d" parameter is the temporal radius for NLMeans.

Last edited by feisty2; 22nd January 2017 at 19:15.
feisty2 is offline   Reply With Quote
Old 22nd January 2017, 19:24   #831  |  Link
manono
Moderator
 
Join Date: Oct 2001
Location: Hawaii
Posts: 7,406
This is the FFT3DFilter thread, feisty2. You can promote your favorite filters somewhere else. Please.
manono is offline   Reply With Quote
Old 23rd January 2017, 00:23   #832  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
Yeah, the FFT3DFilter bashing seems to be totally out of place, not only because this is the main support thread for the plugin, but also because there are still situations where it is quite useful (e.g., as a pre-filter to create an estimation clip).

As for this quote:

Quote:
I wanted to give MDeGrain a go but it seems I'd have to do some digging through forum just to piece together the right plugin and dependencies that have different versions and mods of their own.
pretty much anything you do in AVISynth is going to require that you have a few plugins in addition to the main one. In this case you need MVTools2, along with whatever else it requires. I'll agree that there are a LOT of mods and branches to the MVTools2 development, but I have a package that works well for me, and if you want I would be happy to zip the various things together, along with a couple of denoising sample scripts. My samples are not too much different than the cookbook examples in the MVTools2 documentation, but I have done a few tweaks for speed and also for better performance for the SD (VHS/Beta/8mm) video that I often am asked to restore.

Last edited by johnmeyer; 23rd January 2017 at 00:24. Reason: Hit Save instead of Preview and still had more editing to do
johnmeyer is offline   Reply With Quote
Old 23rd January 2017, 00:50   #833  |  Link
GMJCZP
Registered User
 
GMJCZP's Avatar
 
Join Date: Apr 2010
Location: I have a statue in Hakodate, Japan
Posts: 744
FFT3dfilter is good too for in some cases for repair edges in conjuction with Camembert or TCannyMod.
__________________
By law and justice!

GMJCZP's Arsenal
GMJCZP is offline   Reply With Quote
Old 23rd January 2017, 03:23   #834  |  Link
aerDNA
Registered User
 
aerDNA's Avatar
 
Join Date: Apr 2009
Location: Rijeka, Croatia
Posts: 10
@johnmeyer Thanks, if you have a convenient MDeGrain setup package I'll try it out, but to be honest you probably shouldn't bother because somehow I think I'll be sticking with FFT3DFilter for the time being.
I feel I should explain my rationale and why I'm satisfied with FFT3DFilter. If I denoise, I do so to help compressibility and my usual goal is not to remove all noise but only up to the point where perceptible quality trade-off begins (and by quality I specifically mean detail retention). With FFT3DFilter that point is around sigma=1.5 for my eyes. At near-zero impact on quality, good amount of noise is removed and relatively fast. A more advanced filter will remove more noise but at the cost of speed. On the clip I tested on, FFT3DFilter with params stated above is ~4x faster than DFTTest default and ~15x faster than TNLMeans default. Whatever noise I'm left with I'll just pay for in bitrate, so FFT3DFilter will save me some less bitrate than a more efficient filter would, but it still saved a lot and for a much smaller speed penalty. As feisty2 pointed out, it's 2017. I don't have to target 700MB. Storage/bandwidth is not a problem and I'll gladly have a little bigger output if it means reasonable encoding times.
aerDNA is offline   Reply With Quote
Old 23rd January 2017, 03:57   #835  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
I have been a big fan of (Spatial) VagueDenoiser, and (S/T) FFT3DFilter for some long time.
FFT3D, usually Sigma between about 1.5 and 1.75 (occasionally a little higher), for what it does, is really pretty fast compared to others.

I also like MCDegrainSharp(frames=1), and sometimes use it before a weak FFT3dFilter (sigma= ~ 1.5), usually if some other processing between
the degraining and output.

MCDegrainSharp requires only mvtools2 so not too difficult to set up:- http://forum.doom9.org/showthread.ph...38#post1508638

EDIT: Also see plain MCDegrain() a few posts before above linked.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 23rd January 2017 at 04:01.
StainlessS is offline   Reply With Quote
Old 23rd January 2017, 06:04   #836  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Quote:
Originally Posted by aerDNA View Post
@johnmeyer Thanks, if you have a convenient MDeGrain setup package I'll try it out, but to be honest you probably shouldn't bother because somehow I think I'll be sticking with FFT3DFilter for the time being.
I feel I should explain my rationale and why I'm satisfied with FFT3DFilter. If I denoise, I do so to help compressibility and my usual goal is not to remove all noise but only up to the point where perceptible quality trade-off begins (and by quality I specifically mean detail retention). With FFT3DFilter that point is around sigma=1.5 for my eyes. At near-zero impact on quality, good amount of noise is removed and relatively fast. A more advanced filter will remove more noise but at the cost of speed. On the clip I tested on, FFT3DFilter with params stated above is ~4x faster than DFTTest default and ~15x faster than TNLMeans default. Whatever noise I'm left with I'll just pay for in bitrate, so FFT3DFilter will save me some less bitrate than a more efficient filter would, but it still saved a lot and for a much smaller speed penalty. As feisty2 pointed out, it's 2017. I don't have to target 700MB. Storage/bandwidth is not a problem and I'll gladly have a little bigger output if it means reasonable encoding times.
the quality of a denoising algorithm is typically defined as "how good the result looks still after complete removal of the noise" and fft3d is real lame by that definition, I thought you were talking about that definition of quality earlier, anyways, not the point.

I never used fft3d cuz this kind of filters are generally extremely toxic to high frequency components which leads to edge distortion and ringing and crap like that.

with DFTTest you can simply avoid filtering on high frequency components with the "sstring" parameter, not sure if such policy exists in fft3d, if it does not, you can always do it manually.
Code:
src = last
fft = fft3dfilter(xxx)
hif = mt_makediff(src, src.removegrain(11).removegrain(11))
fft.removegrain(11).removegrain(11).mt_adddiff(hif)
Quote:
Originally Posted by johnmeyer View Post
but also because there are still situations where it is quite useful (e.g., as a pre-filter to create an estimation clip).
apparently he was using it as a main denoiser.

Last edited by feisty2; 23rd January 2017 at 06:08.
feisty2 is offline   Reply With Quote
Old 23rd January 2017, 13:38   #837  |  Link
aerDNA
Registered User
 
aerDNA's Avatar
 
Join Date: Apr 2009
Location: Rijeka, Croatia
Posts: 10
Quote:
Originally Posted by StainlessS View Post
MCDegrainSharp requires only mvtools2 so not too difficult to set up:- http://forum.doom9.org/showthread.ph...38#post1508638
EDIT: Also see plain MCDegrain() a few posts before above linked.
I'll try it, thanks.

Quote:
Originally Posted by feisty2 View Post
the quality of a denoising algorithm is typically defined as "how good the result looks still after complete removal of the noise." and fft3d is real lame by that definition, I thought you were talking about that definition of quality earlier, anyways, not the point.
That's filter quality, I was talking about resulting image quality with regard to detail. I figured there was a misunderstanding - my fault.

Quote:
Originally Posted by feisty2 View Post
I never used fft3d cuz this kind of filters are generally extremely toxic to high frequency components which leads to edge distortion and ringing and crap like that.
with DFTTest you can simply avoid filtering on high frequency components with the "sstring" parameter, not sure if such policy exists in fft3d
It does, there's sigma, sigma2, sigma3, sigma4.
I may not understand the math behind filters but I'm the world's leading expert in determining what looks good to me. I spent a lot of time switching betwen different filter outputs and staring at frames like a twonk. If your opinion is based on math and/or calendar instead of actual experience with the filter, you really should refrain from telling people it's a bad choice. It's a good denoiser that sits in quality vs performance sweet spot. NLMeans takes a lot more processing for moderate gains and whatever comes next will be even worse, because we're deep in the zone of diminishing returns here.
aerDNA is offline   Reply With Quote
Old 23rd January 2017, 14:29   #838  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: void
Posts: 2,633
Quote:
Originally Posted by aerDNA View Post
If your opinion is based on math and/or calendar instead of actual experience with the filter, you really should refrain from telling people it's a bad choice.
don't wanna hijack this thread so please continue here

Quote:
Originally Posted by aerDNA View Post
but I'm the world's leading expert in determining what looks good to me. I spent a lot of time switching betwen different filter outputs and staring at frames like a twonk.
and I couldn't agree with you on this, I'm also a quality freak, and 10 times slower is okay to me longs as there's 1% of quality improvement.

Last edited by feisty2; 23rd January 2017 at 14:35.
feisty2 is offline   Reply With Quote
Old 23rd January 2017, 14:59   #839  |  Link
aerDNA
Registered User
 
aerDNA's Avatar
 
Join Date: Apr 2009
Location: Rijeka, Croatia
Posts: 10
Nah, I'm done here, said what I wanted and plenty more. And frankly I don't like your cocky fanboyish style. Engaging in further discussion wouldn't bring me any kind of satisfaction.
aerDNA is offline   Reply With Quote
Old 16th February 2017, 21:21   #840  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
New thread for plugin version update

Just made a new thread for an update of the plugin, so it doesn't get buried in this long thread
martin53 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 10:41.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.