Dehalo_alpha introducing artifact, what can I change?

[xekon]

I have .bmp made by video, save image from AvsP but they are about 6 megs each, so I saved as .jpg the problem exists in the .bmp as well.

here is my script:
FFVideoSource("E:\AviSynth\job.mkv")
Trim(3414,3625)
SmoothCurve(Ycurve="0-0;16-0;235-255;255-255", Ucurve="0-0;16-0;240-255;255-255", Vcurve="0-0;16-0;240-255;255-255", interp=0)
fluxsmoothST(5,5)
strength = 4
fft3dgpu(bw=8, bh=8, ow=4, oh=4, plane=4, bt=1, sigma=strength)
fft3dgpu(bw=256, bh=256, ow=128, oh=128, plane=4, bt=1, sigma=strength/8, sigma2=strength/4, sigma3=strength/2, sigma4=strength)
fluxsmoothT(3)
ttempsmooth(maxr=7)
AutoCrop(0,2,2,0,0,0,0,30,5,0,-1,0)
AWarpSharp2(128,4,0,44)
#DeHalo_alpha(brightstr=2,ss=2.0)
GradFun3(0.465)

If you look at the two pictures, anywhere there is a narrow amount of color between black lines... it basically creates an additional line that was not there... Now I would really like to use DeHalo_alpha so if anyone has any clues for me why this would happen then please let me know. That is the only thing that changes between the two pics. the first one I have the DeHalo_alpha line commented out. Same frame number and everything.

without DeHalo_alpha(brightstr=2,ss=2.0):
http://sites.google.com/site/jakejsite/nodehalo.jpg

with DeHalo_alpha(brightstr=2,ss=2.0)
http://sites.google.com/site/jakejsite/dehalo.jpg

DeHalo_alpha:

function DeHalo_alpha(clip clp, float "rx", float "ry", float "darkstr", float "brightstr", float "lowsens", float "highsens", float "ss")
{
rx = default( rx, 2.0 )
ry = default( ry, 2.0 )
darkstr = default( darkstr, 1.0 )
brightstr = default( brightstr, 1.0 )
lowsens = default( lowsens, 50 )
highsens = default( highsens, 50 )
ss = default( ss, 1.5 )

LOS = string(lowsens)
HIS = string(highsens/100.0)
DRK = string(darkstr)
BRT = string(brightstr)
ox = clp.width()
oy = clp.height()
uv = 1
uv2 = (uv==3) ? 3 : 2

halos = clp.bicubicresize(m4(ox/rx),m4(oy/ry)).bicubicresize(ox,oy,1,0)
are = mt_lutxy(clp.mt_expand(U=uv,V=uv),clp.mt_inpand(U=uv,V=uv),"x y -","x y -","x y -",U=uv,V=uv)
ugly = mt_lutxy(halos.mt_expand(U=uv,V=uv),halos.mt_inpand(U=uv,V=uv),"x y -","x y -","x y -",U=uv,V=uv)
so = mt_lutxy( ugly, are, "y x - y 0.001 + / 255 * "+LOS+" - y 256 + 512 / "+HIS+" + *" )
lets = mt_merge(halos,clp,so,U=uv,V=uv)
remove = (ss==1.0) ? clp.repair(lets,1,0)
\ : clp.lanczosresize(m4(ox*ss),m4(oy*ss))
\ .mt_logic(lets.mt_expand(U=uv,V=uv).bicubicresize(m4(ox*ss),m4(oy*ss)),"min",U=uv2,V=uv2)
\ .mt_logic(lets.mt_inpand(U=uv,V=uv).bicubicresize(m4(ox*ss),m4(oy*ss)),"max",U=uv2,V=uv2)
\ .lanczosresize(ox,oy)
them = mt_lutxy(clp,remove,"x y < x x y - "+DRK+" * - x x y - "+BRT+" * - ?",U=2,V=2)

return( them )
}

function m4(float x) {return(x<16?16:int(round(x/4.0)*4))}


EDIT: just gave Masked_DHA() a try and the artifacts are gone now, I am not sure if Masked_DHA() is as effective at removing halos or not, but anything is better than the artifacts I was getting so I guess I will just use this for now unless somebody can find out what the problem was.

[mandarinka]

Sorry bu I'll be blunt.

Looking at your script, by the time DHA comes into action, the footage has been already detroyed many times over. I suspect you wouldn't have to solve your DHA problem if you did some sane stuff in the first place. If you want to smooth a hand-drawn anime source, consider mdegrain2 for example (mvtools) instead of heap of smoothing. And please forget that (a)warpsharp exists.

PS. When I used DHA last time, brightstr over 1 would cause trouble to me too, try to keep it <1.

[SEt]

(a)warpsharp can produce very nice results if used correctly and not on max settings. It's especially great after downscaling in antialiasing.

[xekon]

aWarpSharp2() takes a depth of: -128..127

my 44 depth would be equivalent to a value of 22 on the original awarpsharp (44=22*4/2):

Original aWarpSharp compatibility:
Mapping from original aWarpSharp parameters:
thresh = thresh*256
blur = blurlevel
depth = depth*blurlevel/2
chroma = 0->2, 1->4, 2->3

but I am using Masked_DHA() instead now.

[SEt]

Depth 44 is very strong and results in line deformations. When I use it I use something around 4 (though with a bit different version to public one).