Sharpen Filter Suggestions

[kandrey89]

I'm having a hard time with sharpen filters, primarily which ones to use,
import VDub filters,
Avisynth internals,
other Avisynth file filters,
ongoing almost finished good sharpen filters code here on forums.

Could you guys throw some names that you know do excellent filtering both sharpening and denoising for FILM?

Thank You

[Blue_MiSfit]

Some good sharpeners worth looking at:
LimitedSharpenFaster
SeeSaw
Unfilter

Some denoisers:
FFT3D[filter/gpu]
FRFun
DFTTest
DeGrainMedian
HQDN3D
TNLMeans (for the masochistic)

~MiSfit

[mahsah]

Well, if you want sharpening AND denoising in the same filter, I think fft3dfilter/fft3dgpu is the only one that does that. But it can be a bit heavy by default.

Personally, I think the denoiser that takes away the least detail is mvdegrain (use 1, 2, or 3 depending on how fast your PC is). Removegrain is also nice if you need something really speedy.

My favorite sharpener is Limitedsharpenfaster; it can do pretty much anything. I also like combining it with the "Soothe" filter.

[fibbingbear]

I do not recommend using FFT3DFilter's sharpener. It may be a great denoiser, but it's sharpener leaves a lot to be desired.

I sometimes use msharpen, although it's best applied to special cases.

[Blue_MiSfit]

Yeah fft3dfilter's sharpener is only good if used VERY gently.

MVDegrain2 is ... amazing.

But, it's also hideously slow :)

~MiSfit

[Piper]

I agree with fibbingbear and Blue_MiSfit usage of fft3dfilter's sharpener (or lack thereof). There are better sharpeners out there. Here's an example of fft3dfilter and and seesaw which I'm currently using on a 70's Britcom which I continuously toy with...

a = last
b=a.fft3dfilter(plane=0, sigma=1).fft3dfilter(plane=3, sigma=2, bw=32, bh=32, bt=4,)
SeeSaw(a,b, NRlimit=6, NRlimit2=7, Sstr=1.5, Slimit=6, Spower=8, Sdamplo=5, Szp=16)

[Blue_MiSfit]

SeeSaw is great - at least when you preview it in an AviSynth script. For some reason, I'm never happy with encoded results, no matter how much bitrate I throw at it...

~MiSfit

[Piper]

@Blue_MiSfit: Perhaps bitrate isn't your issue. ;)

There's another alternative that I've been using for a while:
mc_spuds(frames=2,strength=2,sharpp=2) Like fft3dfilter, mc_spuds has a "built in" sharpener.

I used this for quite a while before I was lured away by the speed of fft3dgpu (the temptress!). Still, there's a quality to mc_spuds which appeals to me more than fft3dgpu/fft3dfilter, at least so far.

Still, I admit to not using fft3dgpu very much, so I could very well not be using comparable settings. But if I can find settings with fft3dgpu which denoises as well and looks as sharp as the above mc_spuds example, I'll probably switch. It's just that much faster. My previous post was an example of such an effort...

[anubhavrocker]

Some good sharpeners worth looking at:
LimitedSharpenFaster
SeeSaw
Unfilter

Some denoisers:
FFT3D[filter/gpu]
FRFun
DFTTest
DeGrainMedian
HQDN3D
TNLMeans (for the masochistic)

~MiSfit

Hello :)
I have seen your many posts you are so helping to everyone. I have just one request from you. Can you please arrange(either decreasing or increasing order) both sharpeners and denoisers according to their end results, i mean to say according to your experience of ripping which do you favours more.
Thanks in advance.

(PS-I am new to all this ripping, but want to rip like professional rippers. I know it requires hard work but i ll do it)

[xbox360]

try asharp I loved it.

[*.mp4 guy]

Heavyweight Denoisers, in order of percieved quality (to me):

-mvdenoise1/2/3 (pick your slowness) + fft3d for high sad picture areas

-mvdenoise1/2/3

-mvdenoise1/2/3 + predenoised motion prediction clip (stronger denoising)

-tnlmeans (technically this can ouperform mvdegrain, but its just too crazily slow with those esettings)

- Two stage fft3d (to avoid ringing, example later)

-dfttest

- fft3d

-tbilateral (sometimes better then fft3d on some content)

-Vaguedenoiser with prediction (sometimes better then fft3d, fast)

-dctfun (basically just blurs)

-frfun7 (basically just blurs and turns things to oil painting squiggles)

-deen various thresholded blurs, good when combined with more sophisticated methods



Lightweight Denoisers, in order of percieved quality (to me):

-ttempsmooth great temporal lockdown/stabilization, nonexisten/minor artifacts with default settings

-denoise3d (via ffavisynth) fast, powerful spatio temporal smoother, I have never seen it cause temporal artifacts.

-fluxsmoothST good spatio temporal 3x3x3 smoother

-degrainmedian Very powerful, fast, 3x3x3 median filter, tends to cause artifacts in anything less then mode=3, not very powerful in mode=3

-deen (in one of its temporal modes) very strong, but lots of artifacts



two stage fft3d example script:
strength = 6
fft3dfilter(bw=6, bh=6, ow=3, oh=3, plane=0, bt=1, sigma=strength)
fft3dfilter(bw=216, bh=216, ow=108, oh=108, plane=0, bt=1, sigma=strength/8, sigma2=strength/4, sigma3=strength/2, sigma4=strength)

Vaugedenoiser with prediction and temporal stabilization:
Aux = Deen("c3d", 1, 4, 8)#.ttempsmooth(maxr=7)
vaguedenoiser(nsteps=8, auxclip=aux, wiener=true, wratio=1.0, chromaT=1, threshold=1)
FluxsmoothT(3)
gradfun2db(1.51)

mvdegrain2+fft3d+temporal stabilization and debanding:
source = last
pred = source # to get stronger denoising, put denoisers here, they will change how motion vectors are predicted

backward_vec2 = pred.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
backward_vec1 = pred.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec1 = pred.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec2 = pred.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)

maskp1 = mvmask(kind=1, vectors=forward_vec1, ysc=255).UtoY()
maskp2 = mvmask(kind=1, vectors=forward_vec2).UtoY()
maskp3 = mvmask(kind=1, vectors=backward_vec1, ysc=255).UtoY()
maskp4 = mvmask(kind=1, vectors=backward_vec2).UtoY()
maskf = average(maskp1, 0.25, maskp2, 0.25, maskp3, 0.25, maskp4, 0.25).spline36resize(source.width, source.height)

smooth = pred.fft3dfilter(bw=16, bh=16, ow=8, oh=8, bt=1, sigma=4, plane=0)
source2 = maskedmerge(source, smooth, maskf)

source3 = source2.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=2)
source3
ttempsmooth(maxr=7)
gradfun2db(1.51)

[anubhavrocker]

Heavyweight Denoisers, in order of percieved quality (to me):

-mvdenoise1/2/3 (pick your slowness) + fft3d for high sad picture areas

-mvdenoise1/2/3

-mvdenoise1/2/3 + predenoised motion prediction clip (stronger denoising)

-tnlmeans (technically this can ouperform mvdegrain, but its just too crazily slow with those esettings)

- Two stage fft3d (to avoid ringing, example later)

-dfttest

- fft3d

-tbilateral (sometimes better then fft3d on some content)

-Vaguedenoiser with prediction (sometimes better then fft3d, fast)

-dctfun (basically just blurs)

-frfun7 (basically just blurs and turns things to oil painting squiggles)

-deen various thresholded blurs, good when combined with more sophisticated methods



Lightweight Denoisers, in order of percieved quality (to me):

-ttempsmooth great temporal lockdown/stabilization, nonexisten/minor artifacts with default settings

-denoise3d (via ffavisynth) fast, powerful spatio temporal smoother, I have never seen it cause temporal artifacts.

-fluxsmoothST good spatio temporal 3x3x3 smoother

-degrainmedian Very powerful, fast, 3x3x3 median filter, tends to cause artifacts in anything less then mode=3, not very powerful in mode=3

-deen (in one of its temporal modes) very strong, but lots of artifacts



two stage fft3d example script:
strength = 6
fft3dfilter(bw=6, bh=6, ow=3, oh=3, plane=0, bt=1, sigma=strength)
fft3dfilter(bw=216, bh=216, ow=108, oh=108, plane=0, bt=1, sigma=strength/8, sigma2=strength/4, sigma3=strength/2, sigma4=strength)

Vaugedenoiser with prediction and temporal stabilization:
Aux = Deen("c3d", 1, 4, 8)#.ttempsmooth(maxr=7)
vaguedenoiser(nsteps=8, auxclip=aux, wiener=true, wratio=1.0, chromaT=1, threshold=1)
FluxsmoothT(3)
gradfun2db(1.51)

mvdegrain2+fft3d+temporal stabilization and debanding:
source = last
pred = source # to get stronger denoising, put denoisers here, they will change how motion vectors are predicted

backward_vec2 = pred.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
backward_vec1 = pred.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec1 = pred.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec2 = pred.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)

maskp1 = mvmask(kind=1, vectors=forward_vec1, ysc=255).UtoY()
maskp2 = mvmask(kind=1, vectors=forward_vec2).UtoY()
maskp3 = mvmask(kind=1, vectors=backward_vec1, ysc=255).UtoY()
maskp4 = mvmask(kind=1, vectors=backward_vec2).UtoY()
maskf = average(maskp1, 0.25, maskp2, 0.25, maskp3, 0.25, maskp4, 0.25).spline36resize(source.width, source.height)

smooth = pred.fft3dfilter(bw=16, bh=16, ow=8, oh=8, bt=1, sigma=4, plane=0)
source2 = maskedmerge(source, smooth, maskf)

source3 = source2.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=2)
source3
ttempsmooth(maxr=7)
gradfun2db(1.51)

Excellent info mate, thanks for all this. I am gonna try them all.
Well my main concern was dvd9s as i rip only from dvd9 sources. So, what will u prefer for dvd9 source dvdrips ?

[saint-francis]

mvdegrain2+fft3d+temporal stabilization and debanding:
source = last
pred = source # to get stronger denoising, put denoisers here, they will change how motion vectors are predicted

backward_vec2 = pred.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
backward_vec1 = pred.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec1 = pred.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec2 = pred.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)

maskp1 = mvmask(kind=1, vectors=forward_vec1, ysc=255).UtoY()
maskp2 = mvmask(kind=1, vectors=forward_vec2).UtoY()
maskp3 = mvmask(kind=1, vectors=backward_vec1, ysc=255).UtoY()
maskp4 = mvmask(kind=1, vectors=backward_vec2).UtoY()
maskf = average(maskp1, 0.25, maskp2, 0.25, maskp3, 0.25, maskp4, 0.25).spline36resize(source.width, source.height)

smooth = pred.fft3dfilter(bw=16, bh=16, ow=8, oh=8, bt=1, sigma=4, plane=0)
source2 = maskedmerge(source, smooth, maskf)

source3 = source2.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=2)
source3
ttempsmooth(maxr=7)
gradfun2db(1.51)

I tried this and I get an error saying that "there is no function named average".

[jeffy]

http://forum.doom9.org/showthread.php?t=118430

[*.mp4 guy]

@anubhavrocker

Ehm, thats far far too little information for me to make an informed recomendation on. Essentially all commercial dvds are dvd9's, and commercial disks vary extremely widely in quality, no to mention different types of content. I'll outline what I generally use on the most common types of sources.

I usually don't process filmed content, unless there are source artifacts, such as macroblocking, or banding, in which case I consider how much I like the source, and choose a script based on speed/quality tradeoffs, I can't make that decision for you. In my case, I'm never happy with anything less then mvdegrain2, so Its either that, or nothing in my case.

For anime, I usually stick to light Tbilateral+ttempsmooth+gradfun . If the source has heavy noise or artifacts, then I will use either, 2 stage fft3d, or vaguedenoiser, or, If the source has a lot of detail, the mvdegrain2 script I posted, but followed by Tbilateral+ttempsmooth+gradfun, instead of just ttempsmooth and gradfun.

Of course, these are just starting points to build from, I almost never end up using a script without modifying it a bit to fit the specific circumstances.

I should mention that I'm extremely picky, I can't stand fft3d, or degrainmedian on film, the two most popular quick fix filters. However plenty of people use them and are perfectly happy with the results.

@jeffy
Thanks, I always forget about that.

[Avenger007]

Heavyweight Denoisers, in order of percieved quality (to me):

-mvdenoise1/2/3 (pick your slowness) + fft3d for high sad picture areas

-mvdenoise1/2/3

-mvdenoise1/2/3 + predenoised motion prediction clip (stronger denoising)

-tnlmeans (technically this can ouperform mvdegrain, but its just too crazily slow with those esettings)

- Two stage fft3d (to avoid ringing, example later)

-dfttest

- fft3d

-tbilateral (sometimes better then fft3d on some content)

-Vaguedenoiser with prediction (sometimes better then fft3d, fast)

-dctfun (basically just blurs)

-frfun7 (basically just blurs and turns things to oil painting squiggles)

-deen various thresholded blurs, good when combined with more sophisticated methods



Lightweight Denoisers, in order of percieved quality (to me):

-ttempsmooth great temporal lockdown/stabilization, nonexisten/minor artifacts with default settings

-denoise3d (via ffavisynth) fast, powerful spatio temporal smoother, I have never seen it cause temporal artifacts.

-fluxsmoothST good spatio temporal 3x3x3 smoother

-degrainmedian Very powerful, fast, 3x3x3 median filter, tends to cause artifacts in anything less then mode=3, not very powerful in mode=3

-deen (in one of its temporal modes) very strong, but lots of artifacts



two stage fft3d example script:
strength = 6
fft3dfilter(bw=6, bh=6, ow=3, oh=3, plane=0, bt=1, sigma=strength)
fft3dfilter(bw=216, bh=216, ow=108, oh=108, plane=0, bt=1, sigma=strength/8, sigma2=strength/4, sigma3=strength/2, sigma4=strength)

Vaugedenoiser with prediction and temporal stabilization:
Aux = Deen("c3d", 1, 4, 8)#.ttempsmooth(maxr=7)
vaguedenoiser(nsteps=8, auxclip=aux, wiener=true, wratio=1.0, chromaT=1, threshold=1)
FluxsmoothT(3)
gradfun2db(1.51)

mvdegrain2+fft3d+temporal stabilization and debanding:
source = last
pred = source # to get stronger denoising, put denoisers here, they will change how motion vectors are predicted

backward_vec2 = pred.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
backward_vec1 = pred.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec1 = pred.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec2 = pred.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)

maskp1 = mvmask(kind=1, vectors=forward_vec1, ysc=255).UtoY()
maskp2 = mvmask(kind=1, vectors=forward_vec2).UtoY()
maskp3 = mvmask(kind=1, vectors=backward_vec1, ysc=255).UtoY()
maskp4 = mvmask(kind=1, vectors=backward_vec2).UtoY()
maskf = average(maskp1, 0.25, maskp2, 0.25, maskp3, 0.25, maskp4, 0.25).spline36resize(source.width, source.height)

smooth = pred.fft3dfilter(bw=16, bh=16, ow=8, oh=8, bt=1, sigma=4, plane=0)
source2 = maskedmerge(source, smooth, maskf)

source3 = source2.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=2)
source3
ttempsmooth(maxr=7)
gradfun2db(1.51)
:goodpost: but where's MC_Spuds (http://forum.doom9.org/showthread.php?t=131279) :scared:

[saint-francis]

http://forum.doom9.org/showthread.php?t=118430

Right. Thanks.

[fibbingbear]

Heavyweight Denoisers, in order of percieved quality (to me):
two stage fft3d example script:
strength = 6
fft3dfilter(bw=6, bh=6, ow=3, oh=3, plane=0, bt=1, sigma=strength)
fft3dfilter(bw=216, bh=216, ow=108, oh=108, plane=0, bt=1, sigma=strength/8, sigma2=strength/4, sigma3=strength/2, sigma4=strength)


bt=1 implies that you're using it is a purely spatial denoiser. Do you not find it's temporal denoising satisfying?

[*.mp4 guy]

@Avenger007
mc spuds is an automated mvdegrain2 script. I only looked at the first version, but that one was worse quality then plain mvdegrain2.

@fibbingbear

I only use fft3d on anime, and on anime it always causes a sort of "temporal ringing" artifact on characters in motion. I don't usually see that happen on film, but I don't use fft3d on film.

[fibbingbear]

Thank you for your reply *.mp4 guy :) I was going to post a question about FFT3DFilter because I noticed the *exact* same phenomenon. I was curious what other settings people used with it, and why they chose them. Looks like you answered my questions before I even posted them :-p

In followup, do you use any sort of temporal filter in conjunction with fft3dfilter for anime?

[*.mp4 guy]

Sometimes I will use fft3d with bt=0 and a low sigma (bt=0 uses kalman filtering, and is good for locking down mild noise) but usually I use something like this.

fluxsmoothST(5,5)

strength = 4
fft3dfilter(bw=6, bh=6, ow=3, oh=3, plane=0, bt=1, sigma=strength)
fft3dfilter(bw=216, bh=216, ow=108, oh=108, plane=0, bt=1, sigma=strength/8, sigma2=strength/4, sigma3=strength/2, sigma4=strength)

fluxsmoothT(3)
ttempsmooth(maxr=7)
gradfun2db(1.51)

[edit]I forgot to mention, that without a temporal component fft3d will leave low level low frequency noise behind, Vaguedenoiser has essentially the same problem, fluxsmoothT at low settings is perfect for removing these artifacts.

[Blue_MiSfit]

I think we should sticky this... m4g's list is very helpful

[fibbingbear]

I've seen people specify FFT3DFilter with plane=0 (luma) or plane=3 (chroma). Is there any reason to use either one, as opposed to plane=4 (all of them)? I can see how plane=0 would be fast and probably give the best bang for the buck, and how if you know you only have chroma noise*, to use plane=3, but if time is not an issue, why not use plane=4?



*Btw, if anyone has a link to a good example of chroma noise, it'd be helpful. I'm still not 100% sure how to identify it.

[mikeytown2]

I think we should sticky this... m4g's list is very helpful

make a wiki page (http://avisynth.org/mediawiki/Denoisers) of it, i think i should have a link from the Denoisers section (http://avisynth.org/mediawiki/External_filters#Denoisers) to it.

[*.mp4 guy]

@fibbingbear

Denoising a plane that is not noisy will only result in a loss of quality (blurring, banding, temporal anomolies, etc.), so if you only have luma noise, you should only denoise the luma plane. A lot of people go so far as to denoise chroma and luma planes seperately, even when all of the planes contain noise, this is done so that the strength of the denoising can be set individually for each plane.

Whereas Luma noise manifests as variances in brightness, chroma noise is variance in color. The "rainbowing" artifacts found on analog cpatures from composite video are a prime example of chrominance noise, blue/yellow white noise can often be found on film sources, Dv cameras often exhibit low frequency chroma noise that apear like color "blotches" on flat areas, such as walls, and sky.

[Terka]

very useful! maybe also add some examples videos to wiki?

[Blue_MiSfit]

Yup. M4G sums it up very well.

I almost ALWAYS apply some form of chroma denoising - and always separate from luma denoising.

~MiSfit

[oso2k6]

Hello, thanks *.mp4 guy for that script using mvdegrain2+fft3d+...etc, i used it to encode the R1 berserk dvds that are grainy and i liked the result a lot, i changed some parameters to gain more speed, like overlap 2 instead of 4 and ttempsmoothF instead of the regular ttempsmooth, btw, can you suggest a similar script but for filmed content please? Thanks in advance

[*.mp4 guy]

You can use mvdegrain on film, and it will work well. no need to change anything from the example I posted, really (the changes you made are fine aswell).

[rkalwaitis]

MCTemporalDenoise with LSFMod are a nice combo, although a bit slow. I like Mvdegrain as well.

denoised=DegrainMedian(mode=3)

backward_vec2 = MVAnalyse(denoised,isb=true, truemotion=true, chroma=true, delta=2, pel=2, blksize=8, overlap=4, sharp=1, idx=1)
backward_vec1 = MVAnalyse(denoised,isb=true, truemotion=true, chroma=true, delta=1, pel=2, blksize=8, overlap=4, sharp=1, idx=1)
forward_vec1 = MVAnalyse(denoised,isb=false, truemotion=true, chroma=true, delta=1, pel=2, blksize=8, overlap=4, sharp=1, idx=1)
forward_vec2 = MVAnalyse(denoised,isb=false, truemotion=true, chroma=true, delta=2, pel=2, blksize=8, overlap=4, sharp=1, idx=1)
MVDegrain2(source,backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=250, idx=2)

FFT3DFilter(sigma=2.7,bt=4,plane=3) #just chroma


@MP4 guy. Is there a way to improve on my useage of FFT3dfilter and MvDegrain. You are a most reliable source and almost any suggestion you may have would be something to listen to.

K

[thetoof]

I'd recommend using MVTools2:
source_super=MSuper(sharp=1,pel=2,levels=1)

denoised=DegrainMedian(mode=3)
denoised_super=denoised.MSuper(sharp=1,pel=2)

backward_vec2 = MAnalyse(denoised_super,isb=true, truemotion=true, chroma=true, delta=2, blksize=8, overlap=4)
backward_vec1 = MAnalyse(denoised_super,isb=true, truemotion=true, chroma=true, delta=1, blksize=8, overlap=4)
forward_vec1 = MAnalyse(denoised_super,isb=false, truemotion=true, chroma=true, delta=1, blksize=8, overlap=4)
forward_vec2 = MAnalyse(denoised_super,isb=false, truemotion=true, chroma=true, delta=2, blksize=8, overlap=4)
MDegrain2(source,source_super,backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=250)

FFT3DFilter(sigma=2.7,bt=4,plane=3) #just chroma

You can change your predenoising filter depending on your source (fft3d with different sigma/2/3/4 to adapt to the noise pattern for optimal denoising is great on tricky sources).., though talking about denoisers here is a *tad* off topic.

[*.mp4 guy]

You could try prepending fft3d to the degrainmedian line, since you have chroma set to true in MvAnalyse, and you are already applying fft3d directly to the video, it could get you a better result by reducing the influence of chroma noise on the selected motion vectors. You might have to tweak the sigma a bit to get the exact same level of denoising as you had before, but it should be slightly better.

Honestly, there isn't really anything to do to your script. Any real improvements would slow it down, and without knowing exactly what you want it to do i can't give better advice.

fft3d = FFT3DFilter(sigma=2.7,bt=4,plane=3) #just chroma
denoised=fft3d.DegrainMedian(mode=3)

backward_vec2 = MVAnalyse(denoised,isb=true, truemotion=true, chroma=true, delta=2, pel=2, blksize=8, overlap=4, sharp=1, idx=1)
backward_vec1 = MVAnalyse(denoised,isb=true, truemotion=true, chroma=true, delta=1, pel=2, blksize=8, overlap=4, sharp=1, idx=1)
forward_vec1 = MVAnalyse(denoised,isb=false, truemotion=true, chroma=true, delta=1, pel=2, blksize=8, overlap=4, sharp=1, idx=1)
forward_vec2 = MVAnalyse(denoised,isb=false, truemotion=true, chroma=true, delta=2, pel=2, blksize=8, overlap=4, sharp=1, idx=1)
MVDegrain2(fft3d,backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=250, idx=2)

[rkalwaitis]

thetoof and mp4 guy, ill give your suggestions a try.

I tried the below script as I respect the mp4 guys words of wisdom and could not get it to work. besides the obvious dlls (gradfun, ttempsmooth) what else do I need. Which versions of masktools and MVtools? I didnt understand the link above that you had mentioned that you sometimes overlook. Is it connected to this.




mvdegrain2+fft3d+temporal stabilization and debanding:

Code:
source = last
pred = source # to get stronger denoising, put denoisers here, they will change how motion vectors are predicted

backward_vec2 = pred.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
backward_vec1 = pred.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec1 = pred.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec2 = pred.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)

maskp1 = mvmask(kind=1, vectors=forward_vec1, ysc=255).UtoY()
maskp2 = mvmask(kind=1, vectors=forward_vec2).UtoY()
maskp3 = mvmask(kind=1, vectors=backward_vec1, ysc=255).UtoY()
maskp4 = mvmask(kind=1, vectors=backward_vec2).UtoY()
maskf = average(maskp1, 0.25, maskp2, 0.25, maskp3, 0.25, maskp4, 0.25).spline36resize(source.width, source.height)

smooth = pred.fft3dfilter(bw=16, bh=16, ow=8, oh=8, bt=1, sigma=4, plane=0)
source2 = maskedmerge(source, smooth, maskf)

source3 = source2.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=2)
source3
ttempsmooth(maxr=7)
gradfun2db(1.51)

[rkalwaitis]

mp4 guy,
Nevermind I figured it out. The dll was hidden in the src folder. It produced a really pleasing image. Do you use this on anime? No?

[rkalwaitis]

nevermind figured it out.

[xekon]

I know this is an old thread, but I have been experimenting with denoisers for like 2 weeks(bluray Anime, originally on DVD, 1992 Source), and very luckily stumbled across this thread from google. Using a very small variation of this script I have accomplished the amount of cleanup I was going for without blurring out the details. (which is what keeps happening with every other combo I have tried for the last two weeks)

Sometimes I will use fft3d with bt=0 and a low sigma (bt=0 uses kalman filtering, and is good for locking down mild noise) but usually I use something like this.

fluxsmoothST(5,5)

strength = 4
fft3dfilter(bw=6, bh=6, ow=3, oh=3, plane=0, bt=1, sigma=strength)
fft3dfilter(bw=216, bh=216, ow=108, oh=108, plane=0, bt=1, sigma=strength/8, sigma2=strength/4, sigma3=strength/2, sigma4=strength)

fluxsmoothT(3)
ttempsmooth(maxr=7)
gradfun2db(1.51)

[edit]I forgot to mention, that without a temporal component fft3d will leave low level low frequency noise behind, Vaguedenoiser has essentially the same problem, fluxsmoothT at low settings is perfect for removing these artifacts.

here is my small modification:
fluxsmoothST(5,5)
strength = 4
fft3dgpu(bw=8, bh=8, ow=4, oh=4, plane=0, bt=1, sigma=strength)
fft3dgpu(bw=256, bh=256, ow=128, oh=128, plane=0, bt=1, sigma=strength/8, sigma2=strength/4, sigma3=strength/2, sigma4=strength)
fluxsmoothT(3)
ttempsmooth(maxr=7)
AutoCrop(0,2,2,0,0,0,0,30,5,0,-1,0)
AWarpSharp2(128,4,0,44)
DeHalo_alpha(brightstr=2, ss=2.0)
GradFun3(0.4)

[xekon]

bdtest.avs :
A = RaWavSource("E:\AviSynth\job.wav")
V = FFVideoSource("E:\AviSynth\job.mkv")
AudioDub(V, A)
Trim(3414,3625)
fluxsmoothST(5,5)
strength = 4
fft3dgpu(bw=8, bh=8, ow=4, oh=4, plane=4, bt=1, sigma=strength)
fft3dgpu(bw=256, bh=256, ow=128, oh=128, plane=4, bt=1, sigma=strength/8, sigma2=strength/4, sigma3=strength/2, sigma4=strength)
fluxsmoothT(3)
ttempsmooth(maxr=7)
AutoCrop(0,2,2,0,0,0,0,30,5,0,-1,0)
AWarpSharp2(128,4,0,44)
DeHalo_alpha(brightstr=2,ss=2.0)
GradFun3(0.4)

Without GradFun3(0.4) there is pretty bad banding on certain scenes in 1080p Yu Yu Hakusho Anime.

After adding GradFun3(0.4) it looks AMAZING in AvsP, the dither looks just great visually. But AvsP is looking at the video before compression with x264, what setting can you recommend to save the dither? Because after running it through x264 the banding is right back.

here is my current x264 command:
E:\AviSynth\x264 --preset veryslow --crf 22.0 --ref 5 --bframes 16 --no-fast-pskip --output "jobt.mkv" "E:\AviSynth\bdtest.avs"

[gyth]

what setting can you recommend to save the dither?
10bit x264.

[cretindesalpes]

what setting can you recommend to save the dither?
Increase the bitrate (reduce the CRF), 22 is too high to keep the dithering efficiently. You can try to increase the psy-trellis and the AQ strength, but there is a risk of dirtying the sharp elements like edges and lines. Another possibility: use the --zones option in x264 to increase the bitrate locally, if you can identify all the scenes that need debanding.

[xekon]

10bit x264.

WOW! Huge difference! changed nothing with my setting or script, just used 10bit x264 and that was all it took!

Nearly all of the dither was preserved! I am going to drop the crf from 22 to 20 and see if it get any better.

[xekon]

Increase the bitrate (reduce the CRF), 22 is too high to keep the dithering efficiently. You can try to increase the psy-trellis and the AQ strength, but there is a risk of dirtying the sharp elements like edges and lines. Another possibility: use the --zones option in x264 to increase the bitrate locally, if you can identify all the scenes that need debanding.

10bit looks great on my high end computer, but it cannot play back on my media computer at an acceptable speed. it stutters badly. I am going to try upping the AQ strength i guess...

What crf do you recommend, I guess i am going to just keep lowering it to see what kind of effect it has.

[Nevilne]

If you don't need device compatibility then I'd suggest (from top to bottom more likely to work - less likely to work):
-encode in hi10p but lower bframes, reframes, maybe even drop cabac, make sure to decode with latest lav filters.
-for 8bit, expand to pc levels (full range), dither and encode in full range
-just apply ton of dithering and encode in higher bitrate

[xekon]

-for 8bit, expand to pc levels (full range), dither and encode in full range

I am very interested in trying this, What do I change with my current method to expand? and also to encode in full range?

the dither that I applied is with GradFun3(0.4) so I know how to do that part.

[cretindesalpes]

At the end of your script:
Dither_lut8 (yexpr="x 16 - 219 / 65535 *", expr="x 128 - 112 / 32767 * 32768 +", y=3, u=3, v=3)
GradFun3 (0.465, lsb_in=true)

— OR —

Put this just before your filtering chain (FluxSmoothST):
SmoothCurve (Ycurve="0-0;16-0;235-255;255-255", Ucurve="0-0;16-0;240-255;255-255", Vcurve="0-0;16-0;240-255;255-255", interp=0)and set the GradFun3 thr parameter to 0.465.

The latter is preferred because it won't introduce dither patterns in perfectly flat areas. Encode with: --fullrange on

Anyway, I don't think this will make any big difference regarding the dither preservation.

[xekon]

I just tried all the methods suggested here.

You are correct, the 10bit encoding does the best at preserving the dither out of everything.

So I am just going to put a newer CPU in my media PC, stupid crappy pentium D, probably replace it with an E6700 DUO.

I only have one problem left to figure out. The sharp/clean lines that look wonderful in Avisynth AvsP look very bad in x264, like when you have a sharp transition from one color to another.

I have uploaded three .bmp all exported from AvsP. All on the same frame number. there is the original, the result with the avs script, and then the result after encoding the avs to X264.

In these examples, if you look at the outline of the clouds in the x264 bmp, you can see that it is no longer a nice clean line, instead it looks very messy, what can be changed with my script or x264 settings to correct this? I assume its all x264 to blame because the image looks great in AvsP before encoding, but then again I had no idea about SmoothCurve() either.

http://sites.google.com/site/jakejsite/x264artifacts.7z

here is my current script bdtest.avs:
A = RaWavSource("E:\AviSynth\job.wav")
V = FFVideoSource("E:\AviSynth\job.mkv")
AudioDub(V, A)
Trim(3414,3625)
SmoothCurve(Ycurve="0-0;16-0;235-255;255-255", Ucurve="0-0;16-0;240-255;255-255", Vcurve="0-0;16-0;240-255;255-255", interp=0)
fluxsmoothST(5,5)
strength = 4
fft3dgpu(bw=8, bh=8, ow=4, oh=4, plane=4, bt=1, sigma=strength)
fft3dgpu(bw=256, bh=256, ow=128, oh=128, plane=4, bt=1, sigma=strength/8, sigma2=strength/4, sigma3=strength/2, sigma4=strength)
fluxsmoothT(3)
ttempsmooth(maxr=7)
AutoCrop(0,2,2,0,0,0,0,30,5,0,-1,0)
AWarpSharp2(128,4,0,44)
Masked_DHA()
GradFun3(0.465)

and x264:
E:\AviSynth\x264_10 --preset veryslow --crf 22.0 --ref 5 --bframes 16 --no-fast-pskip --output "jobt.mkv" "E:\AviSynth\bdtest.avs"

[*.mp4 guy]

Lower AQ Lower CRF Less Warpsharp. More Trellis, More Subme, better motion search.

--crf 18.5 --weightp 2 --trellis 2 --subme 10 --aq-strength 0.7If trellis messes things up, try adding:--psy-rd 0.2:0.115If that messes things up, reduce the numbers (0.2, 0.115) until things look bad in a way you find acceptable. The crf value is just a guess, you might be able to get away with something in the twenties.

If none of that works to your liking, you will have to post a sample, I'm not going to suggest anything weirder then that unless I can be sure it works in your case.

[xekon]

If none of that works to your liking, you will have to post a sample, I'm not going to suggest anything weirder then that unless I can be sure it works in your case.

Thanks for the response!

I gave --crf 18.5 --aq-strength 0.7 a try, and it helped a little bit.

Next I added on --psy-rd 0.2:0.115, this did a LOT to fix the problem. There is no noise surrounding the lines now, which makes it look a lot better.

The lines themselves however still look jagged or aliased after encoding with x264, prior to encoding they looked smooth in AvsP.

SAMPLE: "old link" (x264 crf 0 before any processing with avisynth)

With that sample, here is my current AVS script that looks great in AvsP, but once encoded in x264 the lines like the edges of the cloud are jagged/aliased looking:
FFVideoSource("E:\AviSynth\job.mkv")
SmoothCurve(Ycurve="0-0;16-0;235-255;255-255", Ucurve="0-0;16-0;240-255;255-255", Vcurve="0-0;16-0;240-255;255-255", interp=0)
fluxsmoothST(5,5)
strength = 4
fft3dgpu(bw=8, bh=8, ow=4, oh=4, plane=4, bt=1, sigma=strength)
fft3dgpu(bw=256, bh=256, ow=128, oh=128, plane=4, bt=1, sigma=strength/8, sigma2=strength/4, sigma3=strength/2, sigma4=strength)
fluxsmoothT(3)
ttempsmooth(maxr=7)
AutoCrop(0,2,2,0,0,0,0,30,5,0,-1,0)
AWarpSharp2(128,4,0,44)
Masked_DHA()
GradFun3(0.465)


PS: --preset veryslow automatically sets the other stuff you recommended: --weightp 2 --trellis 2 --subme 10

with x264_10 --preset veryslow --crf 22.0 --ref 5 --bframes 16 --no-fast-pskip
I get mediaInfo:
cabac=1 / ref=5 / deblock=1:0:0 / analyse=0x3:0x133 / me=umh / subme=10 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-2 / threads=6 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=22.0 / qcomp=0.60 / qpmin=0 / qpmax=81 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00