I want to use FFT3DFILTER to dehalo, but I don't know the right parameters.

[Jeremy Duncan]

Here's a unprocessed d2v clip.
I included some sample avs files.
Use virtualdub and the avs called tweak to open the d2v I included to see some bad haloing.

I'm gonna try and get fft3dfilter to fix this.

Rapidshare (http://rapidshare.com/files/56845016/Ghost_in_the_Shell_Innocence.zip.html)
Mega Upload (http://www.megaupload.com/?d=ARMDS9T8)

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\DGDecode.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MT.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RepairSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\decomb.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\aWarpSharp.dll")

MPEG2Source("C:\Documents and Settings\Dell\Desktop\Ghost in the Shell Innocence\Ghost in the Shell Innocence.d2v")

SetMemoryMax(1024)

AssumeTFF()
Telecide(chroma=false)
decimate()

MT("FFT3DFILTER(sigma=1,bw=14,bh=14,ow=7,oh=7,bt=2,sharpen=1.0,dehalo=1,hr=2,ht=0)",3,2)

MT("aWarpSharp(depth=20, blurlevel=2, thresh=0.21, cm=0)",3,2)

lanczosresize(1680,952)

I want to use FFT3DFILTER to dehalo, but I don't know the right parameters.

Originally, this was a reply to Demi9OD's FFT3DFILTER thread, but I decided to open my own, since al I need is dehalo settings, I thought it would be easier to start my own thread.

If you don't know how to use FFT3DFILTER.dll, you need to put FFTW3.DLL into the system32 folder, and the FFT3DFILTER.dll into the avisynth folder.

Version 2.1.1 (http://avisynth.org.ru/fft3dfilter/fft3dfilter.html)

[Jeremy Duncan]

You'll need to make a new d2v to open the avs up in virtualdub.
Use dgmpegdec:Use DGIndex (http://neuron2.net/dgmpgdec/dgmpgdec148.zip)

I set Video: Field Operation: Forced film.

[Demi9OD]

By the time I can get rid of the Halo's with fft3d it's already blurred the picture back down to 0 sharpening. I really don't think fft3d's dehalo algo is of much use.

[Jeremy Duncan]

Dehalo_alpha isn't much good on this clip either.

[Jeremy Duncan]

Anybody feel free to chime in with a parameter suggestion other than FFT3DFILTER.

[talen9]

BlindDeHalo3 by Didée?

[Didée]

OMG. Halos, echoes of halos, and echoes of echoes of halos. Nice.

Actually, these artefacts can be removed very easily:

http://img527.imageshack.us/img527/4973/removejeremyshalossw6.th.png (http://img527.imageshack.us/my.php?image=removejeremyshalossw6.png)

All halos gone. :D

[Jeremy Duncan]

OMG. Halos, echoes of halos, and echoes of echoes of halos. Nice.

Actually, these artefacts can be removed very easily:

http://img527.imageshack.us/img527/4973/removejeremyshalossw6.th.png (http://img527.imageshack.us/my.php?image=removejeremyshalossw6.png)

All halos gone. :D

That destroys the detail/sharpness.
A avisynth script would be better than a FFDshow blur setting.

Didée,
I understand your making a New Seesaw. Maybe with a dehalo function.
Maybe you can use the clip I uploaded for a sample to test how well your it dehalo's.

I know you don't use FFDshow in realtime, so I felt it was the right thing to tell you your suggestion destroyed the detail.
It looks like a vcr tape from the 80's. :D

[foxyshadis]

Betcha that was his point. And if Herr Didée thinks it hopeless, I doubt anyone else will be able to help it. =p

[MOmonster]

@Jeremy Duncan
Don´t know why, but in your (really) short sample the haloes aren´t so annoying for me. FFT3D won´t do better than Dehalo_Alpha against haloes. If Dehalo_Alpha don´t works for you, you could try this function:
# eee is a fast alternative dehalo function.
# eee means edge enhancment erasing
#
# it can be used against really light halos until heavy over enhancment, but therefore some more parameters were
# necessary
# maybe you will need some time for tweaking
#
# version:
# - eee v0.4a
#
# needed filters:
# - masktools v2.0
#
# parameter description:
# - rx,ry blurring radius - higher values enable stronger blurring [1.2...8.0 -> 2.5]
#
# - dstr,bstr strength for darken and brighten blurring [0.0...1.0 -> 1.0, 0.7]
#
# - prot this parameter works like a source protection "power" ( result=pow(effect,1-prot/100) ;-)
# higher values reduce the blurring effect on edges [0...100 -> 7.0]
#
# - dprot the detail protection can be increased to keep more details [0...60 -> 24.0]
#
# - blimit this parameter decrease the blur weighting for heavier blurred pixels
# that higher you set this parameter that smaller (!!) is it´s influence
# for blimit>1000 the inluence very small [40...10000 -> 300]
#
# - rthr the repair-threshold is used for the post-repair-effect
# lower thresholds will repair more details, but if you set this parameter
# too low, the dehalo effect will be reduced [-128...255 -> 10]
#
# - rrange the range for the post-repair-effect is used for the effect weighting
# higher values will smooth and lower the repair effect, smaller values
# increase and sharpen the repair-effect
# a negative rrange will disable the post repairing and speed up the function
# but edges will be more soften and you have to configure the other
# parameters (higher prot and so on) [..0...1000 -> 50]
#
# - chroma you can decide if you want to process chroma too [bool -> false]
#
# - fclip the last parameter (input clip isn´t named) is a alternative blurred clip
# eee just creates a internal blurred clip and weights it with the source
# the blurring is really simple and not always the best, if you want to use
# your own blurring function but the masking of eee, you can use this parameter



function eee(clip clp, "rx", "ry", "dstr", "bstr", "prot", "dprot", int "blimit", int "rthr", "rrange", bool "chroma", clip "fclip")
{
#### parameters ####
rx = 0.25 / default( rx, 2.5 )
ry = isfloat(ry) ? 0.25/ry : rx

rrange = default(rrange,50)

uv = default(chroma,false)==true ? 3 : 1


#### prepared string parameters ####
D_STR = string(-1*default(dstr,1.0))
B_STR = string(default(bstr,0.7))

DLEVEL = string(default(dprot,24.0)+0.1)
PWR = string(1.0-default(prot,7.0)/100.0)
HIBLUR = string(default(blimit,300))

R_THR = string(default(rthr,10)+128)
RFACTOR = string(255.0/rrange)


#### other needed variables ####
ox = clp.width()
oy = clp.height()


#### working strings ####
WEIGHT = "x y - 2 ^ x y - 2 ^ "+DLEVEL+" + / x y - abs "+HIBLUR+" / 4 ^ 1 + /"
PROCESS= "x y > "+D_STR+" "+B_STR+" ? x y - abs "+PWR+" ^ * "+WEIGHT+" * x +"


#### basic filtering ####
fclip = isclip(fclip) ? fclip : clp.bicubicresize(int(ox*rx)*4,int(oy*ry)*4).bicubicresize(ox,oy,1,0)

dehalo = mt_lutxy(clp,fclip,PROCESS,U=uv,V=uv)

rmask = mt_makediff(dehalo.mt_expand(),dehalo.mt_inpand()).mt_inpand().mt_lut("x "+R_THR+" - "+RFACTOR+" *")

dehalo = rrange<0 ? dehalo : mt_merge(dehalo,clp,rmask,u=2,v=2)

return dehalo
}For example:eee(dprot=60)
But clearing the haloes without any losing of details is nearly impossible. Didee said it allready:D.

[Jeremy Duncan]

eee()

I can't find the link the the necessary masktools.

[talen9]

eee()

I can't find the link the the necessary masktools.

http://forum.doom9.org/showthread.php?t=98985 ;)

[Jeremy Duncan]

I tried 2.0.30.0 and 2.0.32.0 mt_masktools and I couldn't open eee().
Check out my script.

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\DGDecode.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MT.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RepairSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\decomb.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\aWarpSharp.dll")

MPEG2Source("C:\Documents and Settings\Dell\Desktop\Ghost in the Shell Innocence\Ghost in the Shell Innocence.d2v")

SetMemoryMax(1024)

AssumeTFF()
Telecide(chroma=false)
decimate()

MT("FFT3DFILTER(sigma=1,bw=14,bh=14,ow=7,oh=7,bt=2,sharpen=1.0)",3,2)

MT("aWarpSharp(depth=20, blurlevel=2, thresh=0.21, cm=0)",3,2)

eee(dprot=60)

lanczosresize(1680,952)

I tried eee.avs and eee.avsi
This is one of the errors I get:
Avisynth open failure
Avisynth:script open failure

[Jeremy Duncan]

I see no mention of eee() in the doom9 search.
I think your fooling around. There is no eee() is there.
I looked at the person who makes masktools, and there is no masktools V2.0
Google has no record of eee() being mentioned on doom9.
I don't appreciate tomfoolery.

[Demi9OD]

Anyone ever play world of warcraft? What we need here is a WowAceUpdater for Avisynth dlls and scripts...

[Dreassica]

I see no mention of eee() in the doom9 search.
I think your fooling around. There is no eee() is there.
I looked at the person who makes masktools, and there is no masktools V2.0
Google has no record of eee() being mentioned on doom9.
I don't appreciate tomfoolery.
Try adding closing } ad end of that function, as script loads and works fine with that added.

[*.mp4 guy]

Try this, I have no idea if it will look good on the whole film, but it did alright on the tiny little sample.

function halomask1(clip c, int "rad", float "ss"){

ss = default(ss, 1.5)
rad = default(rad, 7)
owidth = c.width
oheight = c.height


ssample = spline36resize(c, round((c.width*SS)/4)*4, round((c.height*SS)/4)*4)
mask1 = mt_lutxy(ssample, Xsharpen(ssample, 127, 255), "x y - abs 1 - 63 *", u=1, v=1)
mask2 = (rad >= 1) ? mask1.mt_expand(u=1, v=1) : mask1
mask2 = (rad >= 2) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 3) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 4) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 5) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 6) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 7) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 8) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 9) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 10) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 11) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 12) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 13) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 14) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 15) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 16) ? mask2.mt_expand(u=1, v=1) : mask2
mask3 = mt_lutXY(mask1, mask2, "y x - 63 - 1.5 *", u=1, v=1).spline36resize(owidth, oheight)

return(mask3)}

MT_Merge(last.blur(-0.7), last.blur(1.42), last.halomask1(ss=1.75, rad=6).frfun7(1.01, 256, 0))

requires MT_Masktolls (masktools 2), frfun7 and Xsharpen

[Jeremy Duncan]

Try adding closing } ad end of that function, as script loads and works fine with that added.

Yes, your right.
I'm sorry for being rough.
Thank you for the function, it does get rid of a good portion of the halo's.

:o

[Jeremy Duncan]

No EEE
Link (http://fileserver2.jpghosting.com/images/NoEEE_ecbf98bdb24f80b5a45a3f3d4a585268.PNG)

EEE
Link (http://fileserver2.jpghosting.com/images/EEE_23d96ffca09d5a2cfe4b3eb5128b102c.PNG)


SetMemoryMax(1024)
AssumeTFF() # If it stutters try AssumeBFF instead of AssumeTFF.
Telecide(chroma=false)
decimate() # Delete this line if your playing a pal dvd, or if changing TFF to BFF doesn't stop the stutter.
FluxSmoothT(3)
ColorMatrix(Mode="Rec.601->Rec.709",clamp=false,threads=2)
MT("SeeSaw(NRlimit=0, NRlimit2=5, Sstr=2.22, Szp=1, Slimit=9, Sdamplo=16, Spower=1, sootheT=95, SdampHi=39)",3)
aWarpSharp(depth=20, blurlevel=2, thresh=0.21, cm=0)
eee(dprot=60)

I don't think I'll use eee.

[MOmonster]

Sorry, a copy and paste mistake.
EEE is nowhere mentioned because it was only on my hd. eee(dprot=60) is just a how-to-use sample and not setted for your sample. But indeed with dehalo_alpha I get often better results, especially on non-anime sources. The main focus was a fast dehalo function.

[Jeremy Duncan]

*.mp4 guy,

I went to get Xsharpen from neuron2's website. But it wasn't a dll, it was a .vdf

I put it into the avisynth plugins folder.
I have frfun7 and MT_Masktools.
This is the avsi function.

function halomask1(clip c, int "rad", float "ss"){

ss = default(ss, 1.5)
rad = default(rad, 7)
owidth = c.width
oheight = c.height


ssample = spline36resize(c, round((c.width*SS)/4)*4, round((c.height*SS)/4)*4)
mask1 = mt_lutxy(ssample, Xsharpen(ssample, 127, 255), "x y - abs 1 - 63 *", u=1, v=1)
mask2 = (rad >= 1) ? mask1.mt_expand(u=1, v=1) : mask1
mask2 = (rad >= 2) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 3) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 4) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 5) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 6) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 7) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 8) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 9) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 10) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 11) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 12) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 13) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 14) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 15) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 16) ? mask2.mt_expand(u=1, v=1) : mask2
mask3 = mt_lutXY(mask1, mask2, "y x - 63 - 1.5 *", u=1, v=1).spline36resize(owidth, oheight)

return(mask3)}

This is the avs I open with virtualdub:
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\DGDecode.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MT.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RepairSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\decomb.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\aWarpSharp.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\frfun7.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\xsharpen.vdf")

MPEG2Source("C:\Documents and Settings\Dell\Desktop\Ghost in the Shell Innocence\Ghost in the Shell Innocence.d2v")

SetMemoryMax(1024)



MT_Merge(last.blur(-0.7), last.blur(1.42), last.halomask1(ss=1.75, rad=6).frfun7(1.01, 256, 0))
lanczosresize(1680,952)

I can't use the function you made. Virtualdub won't open the avs.

MOmonster,
Thanks for the function anyway. :)

[foxyshadis]

Xsharpen is inside the one of the warpsharp packages - http://avisynth.org/warpenterprises/files/warpsharppackage_25_dll_20031103.zip

[Jeremy Duncan]

Ok. I fugure that since people don't complain about haloing on my seesaw config, that if I calibrated fft3dfilter to the same amout of haloing it would be nice.
I found that dehaloing was really just blurring the picture in the examples Didée and MOmonster gave.
So I just lowered the sharpness and set dehalo to one to decrease the resolution to get rid of the extreme halo.
I calibrated the sharpness against the seesaw config to make sure it wasn't too blurry, I want fft3dfilter to be a upgrade from my seesaw config.

So after calibrating the halo and sharness I ran the result through subtract() and I got a config that dehalo's better and is sharper than the seesaw config.

I'll post those subtract pictures now.
Thanks for the help.
Without your help I wouldn't have know that dehaloing reduces detail and I wouldn't have been able to calibrate fft3dfilter.

One (http://fileserver2.jpghosting.com/images/Subtract1_c9c6dbd326a5e7ea19f0c58337feebff.PNG)
Two (http://fileserver2.jpghosting.com/images/Subtract2_8b62760e326be2da50b4783343bcb873.PNG)

Tweak config
SetMemoryMax(1024)

AssumeTFF()
Telecide(chroma=false)
decimate()

MT("FFT3DFILTER(sigma=1,bw=14,bh=14,ow=7,oh=7,bt=2,dehalo=1,sharpen=0.9)",3,2)

MT("aWarpSharp(depth=20, blurlevel=2, thresh=0.21, cm=0)",3,2)

Jeremy config
SetMemoryMax(1024)
AssumeTFF()
Telecide(chroma=false)
decimate()
FluxSmoothT(3)
MT("SeeSaw(NRlimit=0, NRlimit2=5, Sstr=2.22, Szp=1, Slimit=9, Sdamplo=16, Spower=1, sootheT=95, SdampHi=39)",3)
aWarpSharp(depth=20, blurlevel=2, thresh=0.21, cm=0)