Halo killings

[pila13]

Hi, i try kill halo in anime, i use dehalo_alfa -

Assumeframebased()
SeparateFields()
c=last
d=mt_edge()
DeHalo_alpha(rx=0.6,ry=0.6,darkstr=0.4,ss=2)
mt_merge(last,c,d)
DeHalo_alpha(rx=0.1,ry=0.1,darkstr=0.2,ss=2)
em=c.mt_edge("prewitt",64,128).mt_expand().mt_expand()
mt_merge(c,last,em,true)
Weave()

But, i get ugly halos


Clean video simple - http://file.qip.ru/file/VZjFMh43/testdemuxed.html(to download click "скачать"

Hope for you help!

[Didée]

Your script is bad. The first part, up to the 1st mt_merge, uses dehalo for the not-edge areas. Then you follow up with a ridiculously-low-radius dehalo for the edge areas (2nd mt_merge).

Did you check how the result is with just dehalo_alpha, without any ill-formed edge masking?


Perhaps it is sufficient to replace

mt_merge(last,c,d)

with

mt_merge(last,d,c)

because it really seems to me that you got it the wrong way round ...

[pila13]

Assumeframebased()
SeparateFields()
c=last
d=mt_edge()
DeHalo_alpha(rx=1.2,ry=1.2,darkstr=0.3,ss=2)
mt_merge(last,d,c)
DeHalo_alpha(rx=0.8,ry=0.8,darkstr=0.6,ss=2)
em=c.mt_edge("prewitt",64,128).mt_expand().mt_expand()
mt_merge(c,last,em,true)
Weave()

i Get this -


____
Use -

DeHalo_alpha(rx=1.2,ry=1.2,darkstr=0.3,ss=2)

Only
___
Nothing changed

[Didée]

Originally Posted by Didée

replace mt_merge(last,c,d) with mt_merge(last,d,c)

Originally Posted by pila13

i Get this -

Ah! Purrfecct! That's how it has to be!

Well, my fault. I actually meant to use mt_merge(c,last,d)

But I think the problem rather is the halo² situation (halos have sharp borders with 2nd-order halos), which isn't exactly a big strength of DH_alpha.
Also, it could be that some of the more broad halos are partly contained in the chroma planes, like the bright ones on reddish backgrounds. Such can happen if the halos were created while the source was in RGB colorspace. Not quite sure if chroma really is involved - that needs closer inspection, and I didn't download your sample yet. But IF chroma is involved, then DH_alpha won't suffice, because it doesn't process YV12's chroma at all.

I've no specific advice for the moment, except for toying around with various filters and their settings.

Alas, I've tight schedule for the next week or two. You've to investigate on your own, or wait for someone else making good sugestions.

[pila13]

Sorry, my mistake :

I use -

Assumeframebased()
SeparateFields()
c=last
d=mt_edge()
DeHalo_alpha(rx=1.6,ry=1.6,darkstr=0.3,ss=2)
mt_merge(c,last,d)
DeHalo_alpha(rx=0.8,ry=0.8,darkstr=0.6,ss=2)
em=c.mt_edge("prewitt",64,128).mt_expand().mt_expand()
mt_merge(c,last,em,true)
Weave()

But.....nothink chenge -

Upload Clean video simpe on mediafile

[Didée]

Random dehalo musing during lunchbreak.

Source -------------- Result

Not perfect on the reddish areas, dunno. But not bad either.

    # mt_MaskTools-25.dll
    # RemoveGrainSSE2.dll
    # RepairSSE2.dll
    # MedianBlur.dll
    

    o = last

    e1 = o.mt_edge("min/max",0,255,0,255).mt_lut("x 4 - 4 *")#.greyscale
    e2 = e1.mt_expand().mt_expand()
    e2 = e2.mergeluma(e2.mt_expand(),0.5).mt_inflate()

    e3 = e2.mergeluma(e2.mt_expand(),0.5).mt_lutxy(e1.mt_deflate(),"x y 1.2 * -").mt_inflate()#.greyscale
#   e3 = e2.mergeluma(e2.mt_expand(),0.5).mt_lutxy(e1.mt_deflate().mt_deflate(),"x y 1.2 * -").mt_inflate().mt_inflate().greyscale

#   e3 = e3.nnedi2_rpow2(2,cshift="bicubicresize").bicubicresize(o.width,o.height)

    blurr = o.minblur(1).removegrain(11,-1).removegrain(11,-1)

    dh1   = o.repair(o.removegrain(2),1).mt_merge(blurr,e3,U=2,V=2)
    dh1D  = mt_makediff(o,dh1)

    tmp   = dh1.sbr()
    med2D = mt_makediff(tmp,tmp.medianblur(2,0,0))
    DD    = mt_lutxy(dh1D,med2D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs 2 * < x y 128 - 2 * 128 + ? ?")
    dh2   = dh1.mt_adddiff(DD,U=2,V=2)

    interleave(o,dh2)
    return(last)

#======================

    function sbr(clip o) {
    rg11  = o.removegrain(11)
    rg11D = mt_makediff(o,rg11)
    rg11DD= mt_makediff(rg11D,rg11D.removegrain(11)).mt_lutxy(rg11D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")
    o.mt_makediff(rg11DD,U=2,V=2)
    }
#----------------------
    function MinBlur(clip clp, int r, int "uv")
    {
    uv   = default(uv,3)
    uv2  = (uv==2) ? 1 : uv
    rg4  = (uv==3) ? 4 : -1
    rg11 = (uv==3) ? 11 : -1
    rg20 = (uv==3) ? 20 : -1
    medf = (uv==3) ? 1 : -200

    RG11D = (r==1) ? mt_makediff(clp,clp.removegrain(11,rg11),U=uv2,V=uv2)
     \    : (r==2) ? mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20),U=uv2,V=uv2)
     \    :          mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20).removegrain(20,rg20),U=uv2,V=uv2)
    RG4D  = (r==1) ? mt_makediff(clp,clp.removegrain(4,rg4),U=uv2,V=uv2)
     \    : (r==2) ? mt_makediff(clp,clp.medianblur(2,2*medf,2*medf),U=uv2,V=uv2)
     \    :          mt_makediff(clp,clp.medianblur(3,3*medf,3*medf),U=uv2,V=uv2)
    DD    = mt_lutxy(RG11D,RG4D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2)
    clp.mt_makediff(DD,U=uv,V=uv)
    return(last)
    }

[pila13]

I try, sorry, but what line change FPS?

Thx you!

[Didée]

The line

interleave( o, dh2 )

doubles the output fps. This is meant for easier before/after comparison, by single-stepping frame by frame in e.g. VirtualDub.

[pila13]

Ok, i try differents config, and get this -

Clean-

___
1)Use Dehalo-

Assumeframebased()
SeparateFields()
c=last
d=mt_edge()
DeHalo_alpha(rx=1.6,ry=1.6,darkstr=0.4,ss=2)
mt_merge(c,last,d)
DeHalo_alpha(rx=0.8,ry=0.8,darkstr=0.3,ss=2)
em=c.mt_edge("prewitt",64,128).mt_expand().mt_expand()
mt_merge(c,last,em,true)
Weave()

Result -

__
2)Try use some different script -

toon(0.5)
DeHalo_alpha(rx=1.2,ry=2.0,ss=2)
mergechroma(mt_convolution(horizontal="1",vertical="-8 0 0 128 0 0 -8"),last)
DeHalo_alpha(rx=1.2,ry=2.0,ss=2)
stackhorizontal(crop(0,0,8,0),crop(8,0,-8,0).\
mergechroma(crop(8,0,-8,0).awarpsharp(depth=10,cm=1))
\,crop(last.width-8,0,0,0))

Result -

_____
3) Use Didée scrip -

# mt_MaskTools-25.dll
# RemoveGrainSSE2.dll
# RepairSSE2.dll
# MedianBlur.dll


o = last

e1 = o.mt_edge("min/max",0,255,0,255).mt_lut("x 4 - 4 *")#.greyscale
e2 = e1.mt_expand().mt_expand()
e2 = e2.mergeluma(e2.mt_expand(),0.5).mt_inflate()

e3 = e2.mergeluma(e2.mt_expand(),0.5).mt_lutxy(e1.mt_deflate(),"x y 1.2 * -").mt_inflate()#.greyscale
# e3 = e2.mergeluma(e2.mt_expand(),0.5).mt_lutxy(e1.mt_deflate().mt_deflate(),"x y 1.2 * -").mt_inflate().mt_inflate().greyscale

# e3 = e3.nnedi2_rpow2(2,cshift="bicubicresize").bicubicresize(o.width,o.height)

blurr = o.minblur(1).removegrain(11,-1).removegrain(11,-1)

dh1 = o.repair(o.removegrain(2),1).mt_merge(blurr,e3,U=2,V=2)
dh1D = mt_makediff(o,dh1)

tmp = dh1.sbr()
med2D = mt_makediff(tmp,tmp.medianblur(2,0,0))
DD = mt_lutxy(dh1D,med2D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs 2 * < x y 128 - 2 * 128 + ? ?")
dh2 = dh1.mt_adddiff(DD,U=2,V=2)

interleave(o,dh2)
return(last)

#======================

function sbr(clip o) {
rg11 = o.removegrain(11)
rg11D = mt_makediff(o,rg11)
rg11DD= mt_makediff(rg11D,rg11D.removegrain(11)).mt_lutxy(rg11D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")
o.mt_makediff(rg11DD,U=2,V=2)
}
#----------------------
function MinBlur(clip clp, int r, int "uv")
{
uv = default(uv,3)
uv2 = (uv==2) ? 1 : uv
rg4 = (uv==3) ? 4 : -1
rg11 = (uv==3) ? 11 : -1
rg20 = (uv==3) ? 20 : -1
medf = (uv==3) ? 1 : -200

RG11D = (r==1) ? mt_makediff(clp,clp.removegrain(11,rg11),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20).removegrain(20,rg20),U=uv2,V=uv2)
RG4D = (r==1) ? mt_makediff(clp,clp.removegrain(4,rg4),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.medianblur(2,2*medf,2*medf),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.medianblur(3,3*medf,3*medf),U=uv2,V=uv2)
DD = mt_lutxy(RG11D,RG4D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2)
clp.mt_makediff(DD,U=uv,V=uv)
return(last)
}

Result -

=========================
Now i try use script + script №1, get this =
____
1)Script 1+2

Assumeframebased()
SeparateFields()
c=last
d=mt_edge()
DeHalo_alpha(rx=1.6,ry=1.6,darkstr=0.4,ss=2)
mt_merge(c,last,d)
DeHalo_alpha(rx=0.8,ry=0.8,darkstr=0.2,ss=2)
em=c.mt_edge("prewitt",64,128).mt_expand().mt_expand()
mt_merge(c,last,em,true)
Weave()

toon(0.5)
DeHalo_alpha(rx=1.2,ry=2.0,ss=2)
mergechroma(mt_convolution(horizontal="1",vertical="-8 0 0 128 0 0 -8"),last)
DeHalo_alpha(rx=1.2,ry=2.0,ss=2)
stackhorizontal(crop(0,0,8,0),crop(8,0,-8,0).\
mergechroma(crop(8,0,-8,0).awarpsharp(depth=10,cm=1))
\,crop(last.width-8,0,0,0))

Result -

____
2)Last test - script 1+3

Assumeframebased()
SeparateFields()
c=last
d=mt_edge()
DeHalo_alpha(rx=0.6,ry=0.6,darkstr=0.4,ss=2)
mt_merge(c,last,d)
DeHalo_alpha(rx=0.1,ry=0.1,darkstr=0.2,ss=2)
em=c.mt_edge("prewitt",64,128).mt_expand().mt_expand()
mt_merge(c,last,em,true)
Weave()


o = last

e1 = o.mt_edge("min/max",0,255,0,255).mt_lut("x 4 - 4 *")#.greyscale
e2 = e1.mt_expand().mt_expand()
e2 = e2.mergeluma(e2.mt_expand(),0.5).mt_inflate()

e3 = e2.mergeluma(e2.mt_expand(),0.5).mt_lutxy(e1.mt_deflate(),"x y 1.2 * -").mt_inflate()#.greyscale
# e3 = e2.mergeluma(e2.mt_expand(),0.5).mt_lutxy(e1.mt_deflate().mt_deflate(),"x y 1.2 * -").mt_inflate().mt_inflate().greyscale

# e3 = e3.nnedi2_rpow2(2,cshift="bicubicresize").bicubicresize(o.width,o.height)

blurr = o.minblur(1).removegrain(11,-1).removegrain(11,-1)

dh1 = o.repair(o.removegrain(2),1).mt_merge(blurr,e3,U=2,V=2)
dh1D = mt_makediff(o,dh1)

tmp = dh1.sbr()
med2D = mt_makediff(tmp,tmp.medianblur(2,0,0))
DD = mt_lutxy(dh1D,med2D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs 2 * < x y 128 - 2 * 128 + ? ?")
dh2 = dh1.mt_adddiff(DD,U=2,V=2)

#interleave(o,dh2)
return(last)

#======================

function sbr(clip o) {
rg11 = o.removegrain(11)
rg11D = mt_makediff(o,rg11)
rg11DD= mt_makediff(rg11D,rg11D.removegrain(11)).mt_lutxy(rg11D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")
o.mt_makediff(rg11DD,U=2,V=2)
}
#----------------------
function MinBlur(clip clp, int r, int "uv")
{
uv = default(uv,3)
uv2 = (uv==2) ? 1 : uv
rg4 = (uv==3) ? 4 : -1
rg11 = (uv==3) ? 11 : -1
rg20 = (uv==3) ? 20 : -1
medf = (uv==3) ? 1 : -200

RG11D = (r==1) ? mt_makediff(clp,clp.removegrain(11,rg11),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20).removegrain(20,rg20),U=uv2,V=uv2)
RG4D = (r==1) ? mt_makediff(clp,clp.removegrain(4,rg4),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.medianblur(2,2*medf,2*medf),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.medianblur(3,3*medf,3*medf),U=uv2,V=uv2)
DD = mt_lutxy(RG11D,RG4D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2)
clp.mt_makediff(DD,U=uv,V=uv)
return(last)
}

REsult -

_______________
Fuf, The best result I think this script 1 +3, but the picture looks very "soapy". Perhaps there are other ideas?
Thank you very much - Didée and nonsens112
For your help!

[Vitaliy Gorbatenko]

Maybe I'm blind, but in the second example I'm do not see Halo artifacts. But the ringing artifact is present. Accordingly, а combine 2 dehalo scripts make no sense.
To remove ringing and clean lines a'm use this: http://www.mediafire.com/?oam2mnd1lui
Try:
DeRing_and_Halo(2, 10, Maska=false)
or/and
AdaptiveEdgeCleaner2(1, 24)

[dansrfe]

Forgive me Didée, but can you please explain in-depth what you are doing in the script? I really would like to have a better idea of how the script is actually working in order to enhance my understand the specific reasons why what is being used at its respective location in the script.

[Didée]

@ Vitaliy:

Funny to see that many-years-old code of mf to come up again. Back then I told him that all the masking stuff could be greatly simplyfied and sped up, but it seems the old whacky code still persists.
(But apart from that, the good functionality is out of question.)


@ dansfre

Briefly, it is ..

1) edgemask#1-> (gives one line at edges)

2) edgemask#2(edgemask#1) -> (gives hull edge around emask#1)

3) median+blur -> (clean halo/ringing artifacts)

4) copy 3) onto source via 2)

5) enhance 4) via median-sharpen (to counter any not-wished blurring that might've been created)


No 5) uses the property that a median-sharpen will never-ever introduce halos IF a source is free of halos to begin with.

[pila13]

[foreign language removed]

[pila13]

The best result that I turned, changed

LimitedSharpenFaster() ===> LSFmod() +
EdgeCleaner()+BlindDeHalo3(PPmode=-3,strength=150,rx=2.5,ry=2.5)+DeHalo_alpha

[Didée]

"Your best result" still has halos.

[pila13]

I know, can't 100% kill halos
Now i try your script + BlindDeHalo3, maybe rx=.1.8, ry=2.0, strength=200

[pila13]

No more ideas?

[*.mp4 guy]

These sorts of artifacts are annoying because there isn't any cheap way to kill them, and they aren't huge, so its easy to justify ignoring them. Anyway. The compression artifacts are also annoying.

[Vitaliy Gorbatenko]

@ Didée:
I'm can't get the same mask as Camembert with mask tool's. I'm know that is slow.. But it almost perfect..

[pila13]

These sorts of artifacts are annoying because there isn't any cheap way to kill them, and they aren't huge, so its easy to justify ignoring them. Anyway. The compression artifacts are also annoying.

You use awarpsharp().awarpsharp2()?