Fine anime antialiasing

[Mystery Keeper]

I am trying to encode .hack//SIGN and I've got an encode by KAA to compare. Those guys have lost awfully lot of details, but, on the bright side, they've got characters edge strokes smooth and thin. I tried many AA methods, and none could make the same result.

Here's KAA's encode
http://ipicture.ru/uploads/080804/8073/thumbs/4dqX1JoR08.png (http://ipicture.ru/Gallery/Viewfull/4606990.html)

Here's my frame without AA.
http://ipicture.ru/uploads/080804/8073/thumbs/z7Sxw7kfbk.png (http://ipicture.ru/Gallery/Viewfull/4606993.html)

Here's my frame with eedi2().TurnRight().eedi2().TurnLeft()
http://ipicture.ru/uploads/080804/8073/thumbs/SA13fN1TUg.png (http://ipicture.ru/Gallery/Viewfull/4606996.html)

And here's my frame with triple tweaked "antialiasing()" script
http://ipicture.ru/uploads/080804/8073/thumbs/W6zW8EiFVQ.png (http://ipicture.ru/Gallery/Viewfull/4606999.html)

As you can see, antialiasing() gives rather good result. BUT! It came to be rather unstable and gives heavy flickering in some motion scenes.

Does anyone know how I can make the edge strokes as thin and smooth? I tried asking around KAA, and they said their encoder is WoW addict, which they can't contact for months T_T

[Sagekilla]

Try using Toon() I remember that being very useful.

[Mystery Keeper]

Try using Toon() I remember that being very useful.

Very nice for darkening strokes. But 'tis not what I need.

[Comatose]

I think what you need is some kind of warpsharp, this is too big for AA filters to handle :E

[egrimisu]

i like kaa encode :) maybe warpsharp will help a bit like Comatose said

[Mystery Keeper]

I think what you need is some kind of warpsharp, this is too big for AA filters to handle :E

Hey, it's getting closer! But too strong. Carves out some objects too.

Upd: Thanks! I guess I can come up with something nice if I play with that.
Anyone got other suggestions?

[Mystery Keeper]

Ok, this draft gives nice result:

a=last
w=width(a)
h=height(a)
b=a.Warpsharp(depth=100).TurnLeft().EEDI2().TurnRight().EEDI2()
m=logic( b.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
\ ,b.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
\ ,"max").greyscale.levels(0,0.8,128,0,255,false).spline64resize(w,h)
b=b.spline64resize(w,h).Toon(0.5)
MaskedMerge(a,b,m,Y=3,U=3,V=3)

But it still gives a little flicker. Maybe it can be motion compensated?

[Sagekilla]

Sure, if you do this:

bvec = a.MVAnalyse(isb=true,delta=1,idx=1,ov=4,pel=2)
fvec = a.MVAnalyse(isb=false,delta=1,idx=1,ov=4,pel=2)
bcomp = a.MVFlow(bcomp, idx=1, thSCD1=500)
fcomp= a.MVFlow(fcomp, idx=1, thSCD1=500)
b = Interleave(bcomp,a,fcomp)

w=width(b)
h=height(b)
c=b.Warpsharp(depth=100).TurnLeft().EEDI2().TurnRight().EEDI2()
m=logic( c.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
\ ,c.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
\ ,"max").greyscale.levels(0,0.8,128,0,255,false).spline64resize(w,h)
c=c.spline64resize(w,h).Toon(0.5)
MaskedMerge(b,c,m,Y=3,U=3,V=3)
TemporalRepair(b,last)

That should work, I believe.

[Mystery Keeper]

Gives errors. If I correct them, it crashes.

[Didée]

@ Sagekilla: Not quite what I would've expected, but try & see how that turns out. Downside is that the EEDI2 interpolation now is done on 3 times more frames than before ...
Also, me thinks there's a SelectEvery(3,1) missing somewhere, to undo the interleave? ;)


My go (not tested though): MC'ed Soothe.

a=last.assumetff()
w=width(a)
h=height(a)
b=a.Warpsharp(depth=100).TurnLeft().EEDI2().TurnRight().EEDI2()
m=logic( b.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
\ ,b.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
\ ,"max").greyscale.levels(0,0.8,128,0,255,false).spline64resize(w,h)
b=b.spline64resize(w,h,0.5,-0.5,2*w+.001,2*h+.001).Toon(0.5)
MaskedMerge(a,b,m,Y=3,U=3,V=3)

sD=mt_makediff(a,last)

fv2 = a.MVAnalyse(isb=false,delta=2,idx=13)
fv1 = a.MVAnalyse(isb=false,delta=1,idx=13)
bv1 = a.MVAnalyse(isb=true, delta=1,idx=13)
bv2 = a.MVAnalyse(isb=true, delta=2,idx=13)

sDD = sD.MVDegrain2(bv1,fv1,bv2,fv2,idx=14)

reduc = 0.4
sDD = mt_lutxy(sD,sDD,"x 128 - abs y 128 - abs < x y ?") .mergeluma(sDD,1.0-reduc)

a.mt_makediff(sDD,U=2,V=2)

@ MysteryKeeper: you've had a 1/2-pel registration error between the original (a) and antialiased (b) clip, due to EEDI2. Corrected by the blue parts.

[Mystery Keeper]

Thank thee, Didée. I'll try to encode with that script and see if 'tis effective enough. Weird. I found an interlaced frame in the middle of anime o_0

[Sagekilla]

Ah yes, I forgot about SelectEvery(3,1). Also, why is it necessary for the AssumeTFF()? I was just assuming this source was progressive, and wouldn't need any interlace badgering.

Edit: Interlacing inside of anime isn't as uncommon as you might think. Unfortunately very annoying stuff happens in anime all the time :(

[Didée]

Progressive is what I expected, too. The AssumeTFF was just to make sure in which direction the frame gets shifted by EEDI2. It's a bad shortcut, yes, I'm too lazy and tired to look up the necessary parameters to make it the "proper" way. :p

[Mystery Keeper]

@Sagekilla, interlacing is usually introduced in the places of montage. Well, that single frame was at the scene change.

Well, thanks guys. I see that my script is highly imperfect, but as I said, 'twas draft. Now, thanks to you all, I've got the wrapper I need and can play with it and make an adjustable function... tomorrow. Good night ^_^

[Mystery Keeper]

Now this looks very good ^_^ Though prone to some details carving.

function SharpAAMC(clip orig, int "aastr", float "ds", int "SPre", int "SPost", int "BPost")
{
aastr=default(aastr,255) #antialiasing strength
ds=default(ds,0.2) #strokes darkening strength
SPre=default(SPre,100) #Presharpening
SPost=default(SPost,50) #Postsharpening
BPost=default(Bpost,10) #Postbluring

a=orig
w=width(a)
h=height(a)
m=logic( a.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
\ ,a.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
\ ,"max").greyscale.levels(0,0.8,128,0,255,false)
b=a.Toon(ds).Warpsharp(depth=SPre)
b=b.spline64resize(w*2,h*2).TurnLeft().SangNom(aa=aastr).TurnRight().SangNom(aa=aastr)
b=b.Warpsharp(depth=SPost,blur=BPost).spline64resize(w,h)

MaskedMerge(a,b,m,Y=3,U=3,V=3)
sD=mt_makediff(a,last)
fv2 = a.MVAnalyse(isb=false,delta=2,idx=13)
fv1 = a.MVAnalyse(isb=false,delta=1,idx=13)
bv1 = a.MVAnalyse(isb=true, delta=1,idx=13)
bv2 = a.MVAnalyse(isb=true, delta=2,idx=13)

sDD = sD.MVDegrain2(bv1,fv1,bv2,fv2,idx=14)

reduc = 0.4
sDD = mt_lutxy(sD,sDD,"x 128 - abs y 128 - abs < x y ?").mergeluma(sDD,1.0-reduc)

a.mt_makediff(sDD,U=2,V=2)

return last
}

Feel free to tune it and make more versatile!

[Mystery Keeper]

It probably needs some other postsharpening method. But it must still leave lines smooth. Any suggestions?

[Sagekilla]

LSF tends to work great, I've used LimitedSharpenFaster(edgemode=1,strength=160,overshoot=1,soft=15) and it tends to avoid haloing while adding some nice sharpness.

[Mystery Keeper]

Ok. It's getting better and better. But now it seems to be sharpness VS strokes smoothness.

function SharpAAMC(clip orig, int "aastr", float "ds", int "ShPre", int "ShPost", int "SmPost")
{
aastr=default(aastr,255) #antialiasing strength
ds=default(ds,0.2) #strokes darkening strength
ShPre=default(ShPre,80) #Presharpening
ShPost=default(ShPost,300) #Postsharpening
SmPost=default(SmPost,100) #Postsmoothing

a=orig
w=width(a)
h=height(a)
m=logic( a.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
\ ,a.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
\ ,"max").greyscale.levels(0,0.8,128,0,255,false)
b=a.Toon(ds).Warpsharp(depth=ShPre)
b=b.spline64resize(w*2,h*2).TurnLeft().SangNom(aa=aastr).TurnRight().SangNom(aa=aastr)
b=b.spline64resize(w,h).LimitedSharpenFaster(edgemode=1,strength=ShPost,overshoot=1,soft=SmPost)


MaskedMerge(a,b,m,Y=3,U=3,V=3)
sD=mt_makediff(a,last)
fv2 = a.MVAnalyse(isb=false,delta=2,idx=13)
fv1 = a.MVAnalyse(isb=false,delta=1,idx=13)
bv1 = a.MVAnalyse(isb=true, delta=1,idx=13)
bv2 = a.MVAnalyse(isb=true, delta=2,idx=13)

sDD = sD.MVDegrain2(bv1,fv1,bv2,fv2,idx=14)

reduc = 0.4
sDD = mt_lutxy(sD,sDD,"x 128 - abs y 128 - abs < x y ?").mergeluma(sDD,1.0-reduc)

a.mt_makediff(sDD,U=2,V=2)

return last
}

[Sagekilla]

Well 300 is a bit -excessive- for sharpening. You can always do a second toon or use AAA() to smooth out the lines a bit after LSF.

[Mystery Keeper]

Nah. It's tuned to sharpen the eyes area. Greater SmPost will smooth the lines nice and clean, but blur that area again. What's bothering me a bit is that some spots in eyes get lessened. Well, fine with me though. All the parameters can be tuned to make lesser damage in exchange of being less effective.

[DarkT]

I dunno... I'm getting nice results with:

Function SAA(Clip Clp,Int"SS",Bool"CP")
{

OX = Clp.Width
OY = Clp.Height
SS = Default(SS,2)
CP = Default(CP,True)


Clp = Clp.IsYV12() ? Clp : Clp.ConvertToYV12()

Clp.BilinearResize(OX*SS,OY*SS).SangNom().TurnRight()
\ .SangNom().TurnLeft().Spline36Resize(OX,OY)

CP ? Last : MergeChroma(Clp)

Return(Last)

}

ImageSource("untitled.bmp", start=1, end=587,fps=23.976023976023976023976023976024).converttoyv12()

SAA()
Toon(strength=0.3)
WarpSharp(96, 10, 128, -0.6)
LimitedSharpenFaster(smode=4, strength=77)

Anywya, my 2 cents...

[Adub]

Yeah, with my passed experiences with Sagnom, I lost way to many details, and with Spline36resize on top of that, it starts to look to blurry for my personal taste.

[Mystery Keeper]

Yeah, with my passed experiences with Sagnom, I lost way to many details, and with Spline36resize on top of that, it starts to look to blurry for my personal taste.

That's why 'tis masked. Well, warpsharp in this function still kills some details (carves edges), but it can be controlled by ShPre parameter.

Folk, critique is good, but don't just criticize, suggest improvements.

[Adub]

My post was referring to the post by DarkT directly above mine. I haven't even looked at your script. I will now though.

Edit: Wow!! This works wonders on a source I was looking at! It had some nasty jaggies from a bad deinterlace, but this pretty much eliminated them. It's a bit slow though, but for the quality, I'll take it.

Edit2: Okay, it is REALLY SLOW. I am getting around 0.39fps on a lossless encode of a 720p source. Anyway to speed this up? Maybe use the Masktools 2 version of DEdgemask? (If it even exists, I don't know)

[Mystery Keeper]

If you comment the lines between "MaskedMerge()" and "return last", it will disable the motion compensation and probably add you some speed.

Sorry, I'm not well versed in AviSynth ^_^'

[Adub]

Yes, but I like the motion comp. If there is nothing that can be done, I will except my fate. But I'll keep my fingers crossed!!

[Seraphic-]

Sorry, I'm not well versed in AviSynth ^_^'

You seem to know enough to be able to write an impressive script.
Did you find a guide or put it together from bits and pieces from searching around?

Also, could you post your full script code? I was trying to run your script from post #18, but I'm not able to get it to load.
I am most likely missing a few need things.

[martino]

Maybe use the Masktools 2 version of DEdgemask? (If it even exists, I don't know)
Nope, it doesn't. But with some fiddling you could get something similar with mt_edge's prewitt mode.

[TheRyuu]

Can we get a demuxed m2v or something I can play around with? (preferable with a couple scenes, 30 seconds or something, I dunno)

[Adub]

Seraphic, what was your problem? Meaning, what error were you getting?

[Seraphic-]

Seraphic, what was your problem? Meaning, what error were you getting?

"Return value was not a video clip"

[Adub]

Try copying and pasting the script again. Make sure you scroll down all the way to the bottom as well. It sounds like you missed a few lines.

[Comatose]

Mystery Keeper: Why not replace MaskedMerge(a,b,m,Y=3,U=3,V=3) with mt_merge(a,b,m,Y=3,U=3,V=3)?
Also, you're getting the eye "sparkle" loss because of SangNom ;_; They're small spots so it eats them :3 Search for starfield (in this forum).

[Mystery Keeper]

@Seraphic, I have only done the AA part, using people recommendations even there, and made an adjustable function. All other parts are written by somebody else.

How can I split a vob file? I'd love to give it to you for experiments. Folk, I know that's a wrong thread for it, but I'm desperate! I can't denoise it. Whatever I try spoils the video. Grid artifacts, ghosts, washed out details. Nothing seems to work for it.

[Mystery Keeper]

Mystery Keeper: Why not replace MaskedMerge(a,b,m,Y=3,U=3,V=3) with mt_merge(a,b,m,Y=3,U=3,V=3)?

Done. Thanks.

[Seraphic-]

Try copying and pasting the script again. Make sure you scroll down all the way to the bottom as well. It sounds like you missed a few lines.

Yes, seems I did miss a few lines.
It loads now, but doesn't seem to be changing the appearance of the video at all.
Was something else left out?

@Seraphic, I have only done the AA part, using people recommendations even there, and made an adjustable function. All other parts are written by somebody else.

The AA technique that I am familiar with is "EEDI2().TurnRight().EEDI2().TurnLeft()" then "spline36resize" down to the correct resolution.
It does a good job and removes a fair amount of edge jaggies, but also adds a fair amount of blur.

[Mystery Keeper]

The AA technique that I am familiar with is "EEDI2().TurnRight().EEDI2().TurnLeft()" then "spline36resize" down to the correct resolution.
It does a good job and removes a fair amount of edge jaggies, but also adds a fair amount of blur.

'Twas in the initial version of the script. Blur is dealt with alright. 'Tis just that using EEDI2 will make it much slower while not smoothing the lines as good as sangnom. Maybe I'm wrong. Need to test.

[martino]

You never called the function itself.

It does a good job and removes a fair amount of edge jaggies, but also adds a fair amount of blur.
By no way as bad as with SangNom, blur-wise.

[45tripp]

How can I split a vob file? I'd love to give it to you for experiments. .

dgindex.
'[' ']' select section, file -> 'save and demux'

[Seraphic-]

You never called the function itself.

How do you call the function?

[Mystery Keeper]

Just write SharpAAMC() somewhere in your script. Better do it after deinterlacing and denoising and before resizing.

[Mystery Keeper]

Ok, Seraphic, here's the function using EEDI2 for you. Twice as slow.
function ESharpAAMC(clip orig, float "ds", int "ShPre", int "ShPost", int "SmPost")
{
ds=default(ds,0.2) #strokes darkening strength
ShPre=default(ShPre,80) #Presharpening
ShPost=default(ShPost,300) #Postsharpening
SmPost=default(SmPost,100) #Postsmoothing

a=orig.assumetff()
w=width(a)
h=height(a)
m=logic( a.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
\ ,a.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
\ ,"max").greyscale.levels(0,0.8,128,0,255,false)
b=a.Toon(ds).Warpsharp(depth=ShPre)
b=b.TurnLeft().EEDI2().TurnRight().EEDI2().spline64resize(w,h,0.5,-0.5,2*w+.001,2*h+.001)
b=b.LimitedSharpenFaster(edgemode=1,strength=ShPost,overshoot=1,soft=SmPost)


mt_merge(a,b,m,Y=3,U=3,V=3)
sD=mt_makediff(a,last)
fv2 = a.MVAnalyse(isb=false,delta=2,idx=13)
fv1 = a.MVAnalyse(isb=false,delta=1,idx=13)
bv1 = a.MVAnalyse(isb=true, delta=1,idx=13)
bv2 = a.MVAnalyse(isb=true, delta=2,idx=13)

sDD = sD.MVDegrain2(bv1,fv1,bv2,fv2,idx=14)

reduc = 0.4
sDD = mt_lutxy(sD,sDD,"x 128 - abs y 128 - abs < x y ?").mergeluma(sDD,1.0-reduc)

a.mt_makediff(sDD,U=2,V=2)

return last
}

And now compare them. Left - sangnom, right - eedi2.

http://ipicture.ru/uploads/080807/8073/thumbs/AihJn6JCxS.png (http://ipicture.ru/Gallery/Viewfull/4864548.html)

[Adub]

Hell, I can't tell the difference. I'll stick with the sangnom one.

Although, I got maa() (from the animeivtc package) and some other filters to achieve essentially the same results, with a much faster render time. Sure, it's not exactly the same, but it works.

[Seraphic-]

Ok, Seraphic, here's the function using EEDI2 for you. Twice as slow.
function ESharpAAMC(clip orig, float "ds", int "ShPre", int "ShPost", int "SmPost")
{
ds=default(ds,0.2) #strokes darkening strength
ShPre=default(ShPre,80) #Presharpening
ShPost=default(ShPost,300) #Postsharpening
SmPost=default(SmPost,100) #Postsmoothing

a=orig.assumetff()
w=width(a)
h=height(a)
m=logic( a.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
\ ,a.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
\ ,"max").greyscale.levels(0,0.8,128,0,255,false)
b=a.Toon(ds).Warpsharp(depth=ShPre)
b=b.TurnLeft().EEDI2().TurnRight().EEDI2().spline64resize(w,h,0.5,-0.5,2*w+.001,2*h+.001)
b=b.LimitedSharpenFaster(edgemode=1,strength=ShPost,overshoot=1,soft=SmPost)


mt_merge(a,b,m,Y=3,U=3,V=3)
sD=mt_makediff(a,last)
fv2 = a.MVAnalyse(isb=false,delta=2,idx=13)
fv1 = a.MVAnalyse(isb=false,delta=1,idx=13)
bv1 = a.MVAnalyse(isb=true, delta=1,idx=13)
bv2 = a.MVAnalyse(isb=true, delta=2,idx=13)

sDD = sD.MVDegrain2(bv1,fv1,bv2,fv2,idx=14)

reduc = 0.4
sDD = mt_lutxy(sD,sDD,"x 128 - abs y 128 - abs < x y ?").mergeluma(sDD,1.0-reduc)

a.mt_makediff(sDD,U=2,V=2)

return last
}

And now compare them. Left - sangnom, right - eedi2.

http://ipicture.ru/uploads/080807/8073/thumbs/AihJn6JCxS.png (http://ipicture.ru/Gallery/Viewfull/4864548.html)

You didn't have to do that.
All I meant was I used EEDI2 to do some testing here and there.

I'm getting a "no function named DEdgeMask" error.
I have LimitedSharpenFaster, mt_masktools-26.dll, and warpsharp.dll
During a search said you needed those three, but still returns the error.

[martino]

You need the old masktools for DEdgeMask.


P.S. You need Toon, EEDI2 and MVTools too.

[Seraphic-]

You need the old masktools for DEdgeMask.


P.S. You need Toon, EEDI2 and MVTools too.

Yup, have those. Just having a hard time finding Toon.

[Mystery Keeper]

http://mf.creations.nl/avs/filters/Toon-v1.0.dll

[Seraphic-]

http://mf.creations.nl/avs/filters/Toon-v1.0.dll

Thanks. Quite an impressive script as the default setting removed a lot o the jagged edges without much smoothing.
It over smooths on some of my other videos, so I'll have to mess around with the settings.

I tried to load your EEDI2 version from post #42, but get no function named "SharpAAMC".
However, after searching, I found nothing on it so I'm not sure what I'm missing.

(No AA Top - With AA Bottom)
http://www.temp.seraphicgate.com/noaa.jpg
http://www.temp.seraphicgate.com/withaa.jpg

Edit2: Okay, it is REALLY SLOW. I am getting around 0.39fps on a lossless encode of a 720p source. Anyway to speed this up? Maybe use the Masktools 2 version of DEdgemask? (If it even exists, I don't know)

I'm getting 1.55fps on one of my lossless 720p captures.
For a 3:30 long video, that works out to about two hours. :p

[Mystery Keeper]

EEDI2 version is called ESharpAAMC.

[Seraphic-]

EEDI2 version is called ESharpAAMC.

Ah, see what you mean now. Thanks.
Yeah, it is much slower as you said.
Getting 1.15fps, down from 1.55fps with the other.

During your testing, how did you find these scripts to handle fast motion?