Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 3rd August 2008, 21:57   #1  |  Link
Mystery Keeper
Beyond Kawaii
 
Mystery Keeper's Avatar
 
Join Date: Feb 2008
Location: Russia
Posts: 724
Fine anime antialiasing

I am trying to encode .hack//SIGN and I've got an encode by KAA to compare. Those guys have lost awfully lot of details, but, on the bright side, they've got characters edge strokes smooth and thin. I tried many AA methods, and none could make the same result.

Here's KAA's encode


Here's my frame without AA.


Here's my frame with eedi2().TurnRight().eedi2().TurnLeft()


And here's my frame with triple tweaked "antialiasing()" script


As you can see, antialiasing() gives rather good result. BUT! It came to be rather unstable and gives heavy flickering in some motion scenes.

Does anyone know how I can make the edge strokes as thin and smooth? I tried asking around KAA, and they said their encoder is WoW addict, which they can't contact for months T_T
__________________
...desu!
Mystery Keeper is offline   Reply With Quote
Old 3rd August 2008, 22:14   #2  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
Try using Toon() I remember that being very useful.
Sagekilla is offline   Reply With Quote
Old 3rd August 2008, 22:30   #3  |  Link
Mystery Keeper
Beyond Kawaii
 
Mystery Keeper's Avatar
 
Join Date: Feb 2008
Location: Russia
Posts: 724
Quote:
Originally Posted by Sagekilla View Post
Try using Toon() I remember that being very useful.
Very nice for darkening strokes. But 'tis not what I need.
__________________
...desu!
Mystery Keeper is offline   Reply With Quote
Old 3rd August 2008, 22:50   #4  |  Link
Comatose
Registered User
 
Join Date: Dec 2007
Posts: 639
I think what you need is some kind of warpsharp, this is too big for AA filters to handle :E
Comatose is offline   Reply With Quote
Old 3rd August 2008, 22:54   #5  |  Link
egrimisu
Registered User
 
egrimisu's Avatar
 
Join Date: Jan 2008
Location: Romania - neighbor of Dracula
Posts: 327
i like kaa encode maybe warpsharp will help a bit like Comatose said
egrimisu is offline   Reply With Quote
Old 3rd August 2008, 22:55   #6  |  Link
Mystery Keeper
Beyond Kawaii
 
Mystery Keeper's Avatar
 
Join Date: Feb 2008
Location: Russia
Posts: 724
Quote:
Originally Posted by Comatose View Post
I think what you need is some kind of warpsharp, this is too big for AA filters to handle :E
Hey, it's getting closer! But too strong. Carves out some objects too.

Upd: Thanks! I guess I can come up with something nice if I play with that.
Anyone got other suggestions?
__________________
...desu!

Last edited by Mystery Keeper; 3rd August 2008 at 23:00.
Mystery Keeper is offline   Reply With Quote
Old 3rd August 2008, 23:34   #7  |  Link
Mystery Keeper
Beyond Kawaii
 
Mystery Keeper's Avatar
 
Join Date: Feb 2008
Location: Russia
Posts: 724
Ok, this draft gives nice result:

Code:
a=last
w=width(a)
h=height(a)
b=a.Warpsharp(depth=100).TurnLeft().EEDI2().TurnRight().EEDI2()
m=logic( b.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
 \       ,b.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
 \       ,"max").greyscale.levels(0,0.8,128,0,255,false).spline64resize(w,h)
b=b.spline64resize(w,h).Toon(0.5)
MaskedMerge(a,b,m,Y=3,U=3,V=3)
But it still gives a little flicker. Maybe it can be motion compensated?
__________________
...desu!

Last edited by Mystery Keeper; 3rd August 2008 at 23:50.
Mystery Keeper is offline   Reply With Quote
Old 4th August 2008, 00:09   #8  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
Sure, if you do this:

bvec = a.MVAnalyse(isb=true,delta=1,idx=1,ov=4,pel=2)
fvec = a.MVAnalyse(isb=false,delta=1,idx=1,ov=4,pel=2)
bcomp = a.MVFlow(bcomp, idx=1, thSCD1=500)
fcomp= a.MVFlow(fcomp, idx=1, thSCD1=500)
b = Interleave(bcomp,a,fcomp)

w=width(b)
h=height(b)
c=b.Warpsharp(depth=100).TurnLeft().EEDI2().TurnRight().EEDI2()
m=logic( c.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
\ ,c.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
\ ,"max").greyscale.levels(0,0.8,128,0,255,false).spline64resize(w,h)
c=c.spline64resize(w,h).Toon(0.5)
MaskedMerge(b,c,m,Y=3,U=3,V=3)
TemporalRepair(b,last)

That should work, I believe.
Sagekilla is offline   Reply With Quote
Old 4th August 2008, 00:17   #9  |  Link
Mystery Keeper
Beyond Kawaii
 
Mystery Keeper's Avatar
 
Join Date: Feb 2008
Location: Russia
Posts: 724
Gives errors. If I correct them, it crashes.
__________________
...desu!
Mystery Keeper is offline   Reply With Quote
Old 4th August 2008, 00:21   #10  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,393
@ Sagekilla: Not quite what I would've expected, but try & see how that turns out. Downside is that the EEDI2 interpolation now is done on 3 times more frames than before ...
Also, me thinks there's a SelectEvery(3,1) missing somewhere, to undo the interleave?


My go (not tested though): MC'ed Soothe.

Code:
a=last.assumetff()
w=width(a)
h=height(a)
b=a.Warpsharp(depth=100).TurnLeft().EEDI2().TurnRight().EEDI2()
m=logic( b.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
 \       ,b.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
 \       ,"max").greyscale.levels(0,0.8,128,0,255,false).spline64resize(w,h)
b=b.spline64resize(w,h,0.5,-0.5,2*w+.001,2*h+.001).Toon(0.5)
MaskedMerge(a,b,m,Y=3,U=3,V=3)

sD=mt_makediff(a,last)

fv2 = a.MVAnalyse(isb=false,delta=2,idx=13)
fv1 = a.MVAnalyse(isb=false,delta=1,idx=13)
bv1 = a.MVAnalyse(isb=true, delta=1,idx=13)
bv2 = a.MVAnalyse(isb=true, delta=2,idx=13)

sDD = sD.MVDegrain2(bv1,fv1,bv2,fv2,idx=14)

reduc = 0.4
sDD = mt_lutxy(sD,sDD,"x 128 - abs y 128 - abs < x y ?") .mergeluma(sDD,1.0-reduc)

a.mt_makediff(sDD,U=2,V=2)
@ MysteryKeeper: you've had a 1/2-pel registration error between the original (a) and antialiased (b) clip, due to EEDI2. Corrected by the blue parts.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 4th August 2008, 00:35   #11  |  Link
Mystery Keeper
Beyond Kawaii
 
Mystery Keeper's Avatar
 
Join Date: Feb 2008
Location: Russia
Posts: 724
Thank thee, Didée. I'll try to encode with that script and see if 'tis effective enough. Weird. I found an interlaced frame in the middle of anime o_0
__________________
...desu!
Mystery Keeper is offline   Reply With Quote
Old 4th August 2008, 00:56   #12  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
Ah yes, I forgot about SelectEvery(3,1). Also, why is it necessary for the AssumeTFF()? I was just assuming this source was progressive, and wouldn't need any interlace badgering.

Edit: Interlacing inside of anime isn't as uncommon as you might think. Unfortunately very annoying stuff happens in anime all the time
Sagekilla is offline   Reply With Quote
Old 4th August 2008, 01:03   #13  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,393
Progressive is what I expected, too. The AssumeTFF was just to make sure in which direction the frame gets shifted by EEDI2. It's a bad shortcut, yes, I'm too lazy and tired to look up the necessary parameters to make it the "proper" way.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 4th August 2008, 01:52   #14  |  Link
Mystery Keeper
Beyond Kawaii
 
Mystery Keeper's Avatar
 
Join Date: Feb 2008
Location: Russia
Posts: 724
@Sagekilla, interlacing is usually introduced in the places of montage. Well, that single frame was at the scene change.

Well, thanks guys. I see that my script is highly imperfect, but as I said, 'twas draft. Now, thanks to you all, I've got the wrapper I need and can play with it and make an adjustable function... tomorrow. Good night ^_^
__________________
...desu!

Last edited by Mystery Keeper; 4th August 2008 at 01:55.
Mystery Keeper is offline   Reply With Quote
Old 4th August 2008, 13:37   #15  |  Link
Mystery Keeper
Beyond Kawaii
 
Mystery Keeper's Avatar
 
Join Date: Feb 2008
Location: Russia
Posts: 724
Now this looks very good ^_^ Though prone to some details carving.

Code:
function SharpAAMC(clip orig, int "aastr", float "ds", int "SPre", int "SPost", int "BPost")
{
aastr=default(aastr,255) #antialiasing strength
ds=default(ds,0.2) #strokes darkening strength
SPre=default(SPre,100) #Presharpening
SPost=default(SPost,50) #Postsharpening
BPost=default(Bpost,10) #Postbluring

a=orig
w=width(a)
h=height(a)
m=logic( a.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
 \       ,a.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
 \       ,"max").greyscale.levels(0,0.8,128,0,255,false)
b=a.Toon(ds).Warpsharp(depth=SPre)
b=b.spline64resize(w*2,h*2).TurnLeft().SangNom(aa=aastr).TurnRight().SangNom(aa=aastr)
b=b.Warpsharp(depth=SPost,blur=BPost).spline64resize(w,h)

MaskedMerge(a,b,m,Y=3,U=3,V=3)
sD=mt_makediff(a,last)
fv2 = a.MVAnalyse(isb=false,delta=2,idx=13)
fv1 = a.MVAnalyse(isb=false,delta=1,idx=13)
bv1 = a.MVAnalyse(isb=true, delta=1,idx=13)
bv2 = a.MVAnalyse(isb=true, delta=2,idx=13)

sDD = sD.MVDegrain2(bv1,fv1,bv2,fv2,idx=14)

reduc = 0.4
sDD = mt_lutxy(sD,sDD,"x 128 - abs y 128 - abs < x y ?").mergeluma(sDD,1.0-reduc)

a.mt_makediff(sDD,U=2,V=2)

return last
}
Feel free to tune it and make more versatile!
__________________
...desu!

Last edited by Mystery Keeper; 4th August 2008 at 13:42.
Mystery Keeper is offline   Reply With Quote
Old 4th August 2008, 14:23   #16  |  Link
Mystery Keeper
Beyond Kawaii
 
Mystery Keeper's Avatar
 
Join Date: Feb 2008
Location: Russia
Posts: 724
It probably needs some other postsharpening method. But it must still leave lines smooth. Any suggestions?
__________________
...desu!
Mystery Keeper is offline   Reply With Quote
Old 4th August 2008, 15:06   #17  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
LSF tends to work great, I've used LimitedSharpenFaster(edgemode=1,strength=160,overshoot=1,soft=15) and it tends to avoid haloing while adding some nice sharpness.
Sagekilla is offline   Reply With Quote
Old 4th August 2008, 15:54   #18  |  Link
Mystery Keeper
Beyond Kawaii
 
Mystery Keeper's Avatar
 
Join Date: Feb 2008
Location: Russia
Posts: 724
Ok. It's getting better and better. But now it seems to be sharpness VS strokes smoothness.

Code:
function SharpAAMC(clip orig, int "aastr", float "ds", int "ShPre", int "ShPost", int "SmPost")
{
aastr=default(aastr,255) #antialiasing strength
ds=default(ds,0.2) #strokes darkening strength
ShPre=default(ShPre,80) #Presharpening
ShPost=default(ShPost,300) #Postsharpening
SmPost=default(SmPost,100) #Postsmoothing

a=orig
w=width(a)
h=height(a)
m=logic( a.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
 \       ,a.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
 \       ,"max").greyscale.levels(0,0.8,128,0,255,false)
b=a.Toon(ds).Warpsharp(depth=ShPre)
b=b.spline64resize(w*2,h*2).TurnLeft().SangNom(aa=aastr).TurnRight().SangNom(aa=aastr)
b=b.spline64resize(w,h).LimitedSharpenFaster(edgemode=1,strength=ShPost,overshoot=1,soft=SmPost)


MaskedMerge(a,b,m,Y=3,U=3,V=3)
sD=mt_makediff(a,last)
fv2 = a.MVAnalyse(isb=false,delta=2,idx=13)
fv1 = a.MVAnalyse(isb=false,delta=1,idx=13)
bv1 = a.MVAnalyse(isb=true, delta=1,idx=13)
bv2 = a.MVAnalyse(isb=true, delta=2,idx=13)

sDD = sD.MVDegrain2(bv1,fv1,bv2,fv2,idx=14)

reduc = 0.4
sDD = mt_lutxy(sD,sDD,"x 128 - abs y 128 - abs < x y ?").mergeluma(sDD,1.0-reduc)

a.mt_makediff(sDD,U=2,V=2)

return last
}
__________________
...desu!

Last edited by Mystery Keeper; 4th August 2008 at 16:08.
Mystery Keeper is offline   Reply With Quote
Old 4th August 2008, 16:26   #19  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
Well 300 is a bit -excessive- for sharpening. You can always do a second toon or use AAA() to smooth out the lines a bit after LSF.

Last edited by Sagekilla; 4th August 2008 at 16:29.
Sagekilla is offline   Reply With Quote
Old 4th August 2008, 16:42   #20  |  Link
Mystery Keeper
Beyond Kawaii
 
Mystery Keeper's Avatar
 
Join Date: Feb 2008
Location: Russia
Posts: 724
Nah. It's tuned to sharpen the eyes area. Greater SmPost will smooth the lines nice and clean, but blur that area again. What's bothering me a bit is that some spots in eyes get lessened. Well, fine with me though. All the parameters can be tuned to make lesser damage in exchange of being less effective.
__________________
...desu!
Mystery Keeper is offline   Reply With Quote
Reply

Tags
anime, antialiasing, avisynth

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:23.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.