Anti-aliasing script

[maki]

Hi me again XD!!

Can someone help!

I wan to fix the lines of this video. Here's a pic:

http://img684.imageshack.us/i/yuki1.png/

And also I used warpsharp at depth=50 and I was wondering if it's too much. Here's the pic with warpsharp, and also this is the one I want to edit:

http://img710.imageshack.us/i/yuki2b.png/

I meant to say-I want to fix the jagginess of the eyebrows and glasses and I know I need to de_halo it, but right I just want to fix the lines.

another thing here's script and I want to ask why does this happen( I am still a noob at this, but I try to read a little everyday, so please be patience with me XD).

P.S. sorry for posting the pics its just that imageshack wouldn't let me link to the image, I will fix it when they fix the problem.

without sharpaamcmod():

deblock_qed( quant1=25,quant2=25, aOff1=8,bOff1=8,aOff2=8,bOff2=8, uv=3 )
o = last
DH = o.dehalo_alpha(rx=1.75,ry=2.8,darkstr=0)
bb = DH.removegrain(11)
xD = mt_makediff(bb,bb.repair(bb.repair(bb.medianblur(2,-333,-333),1),1))
\ .mt_lut("x 128 - 2.49 * 128 +")
xDD = mt_lutxy(xD,mt_makediff(o,DH),"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")

DH.mt_adddiff(xDD,U=2,V=2)

FluxSmoothST(7,7)
WarpSharp(depth=20)
HQDering(strength=255, overall =0)

LimitedSharpenfaster( ss_x=1.0, ss_y=2.0, dest_x=1280, dest_y=720,
\ Smode=3, strength=255, radius=0,
\ Lmode=1, wide=true, overshoot=1,
\ edgemode=1, special=false,
\ exborder=1)

#deinterlace
#crop
LanczosResize(1280,720) # Lanczos (Sharp)
sharpaamcmod()
#denoise
ConvertToYV12()

http://img407.imageshack.us/i/yukiwithoutsharpmod.png/

with sharpaamcmod()

deblock_qed( quant1=25,quant2=25, aOff1=8,bOff1=8,aOff2=8,bOff2=8, uv=3 )
o = last
DH = o.dehalo_alpha(rx=1.75,ry=2.8,darkstr=0)
bb = DH.removegrain(11)
xD = mt_makediff(bb,bb.repair(bb.repair(bb.medianblur(2,-333,-333),1),1))
\ .mt_lut("x 128 - 2.49 * 128 +")
xDD = mt_lutxy(xD,mt_makediff(o,DH),"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")

DH.mt_adddiff(xDD,U=2,V=2)

FluxSmoothST(7,7)
WarpSharp(depth=20)
HQDering(strength=100, overall =0)

LimitedSharpenfaster( ss_x=1.0, ss_y=2.0, dest_x=1280, dest_y=720,
\ Smode=3, strength=255, radius=0,
\ Lmode=1, wide=true, overshoot=1,
\ edgemode=1, special=false,
\ exborder=1)

#deinterlace
sharpaamcmod()
#crop
LanczosResize(1280,720) # Lanczos (Sharp)
sharpaamcmod()
#denoise
ConvertToYV12()

P.S. I changed HQ to 100 I gain some of the dark line quality since this image.

I know that it isn't limitedsharpen since I changed each one of its settings.

http://img35.imageshack.us/i/yukiwithsharpmod.png/

and also is it possible to get the results of the second image without the loss of details and curvature lines.

Oh! look what I found. Somehow the same effect as the above image, but with jagginess and a bit thicker lines:

deblock_qed( quant1=25,quant2=25, aOff1=8,bOff1=8,aOff2=8,bOff2=8, uv=3 )
o = last
DH = o.dehalo_alpha(rx=1.75,ry=2.8,darkstr=0)
bb = DH.removegrain(11)
xD = mt_makediff(bb,bb.repair(bb.repair(bb.medianblur(2,-333,-333),1),1))
\ .mt_lut("x 128 - 2.49 * 128 +")
xDD = mt_lutxy(xD,mt_makediff(o,DH),"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")

DH.mt_adddiff(xDD,U=2,V=2)
FluxSmoothST(7,7)
WarpSharp(depth=20)

HQDering(strength=255, overall =0)
LimitedSharpenfaster( ss_x=1.0, ss_y=2.0, dest_x=1280, dest_y=720,
\ Smode=3, strength=255, radius=0,
\ Lmode=1, wide=true, overshoot=1,
\ edgemode=1, special=false,
\ exborder=1)
function ESharpAAMC(clip orig, float "ds", int "ShPre", int "ShPost", int "SmPost")
{
ds=default(ds,0.2) #strokes darkening strength
ShPre=default(ShPre,80) #Presharpening
ShPost=default(ShPost,300) #Postsharpening
SmPost=default(SmPost,100) #Postsmoothing

a=orig.assumetff()
w=width(a)
h=height(a)
m=logic( a.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
\ ,a.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
\ ,"max").greyscale.levels(0,0.8,128,0,255,false)
b=a.Toon(ds).Warpsharp(depth=ShPre)
b=b.TurnLeft().EEDI2().TurnRight().EEDI2().spline64resize(w,h,0.5,-0.5,2*w+.001,2*h+.001)
b=b.LimitedSharpenFaster(edgemode=1,strength=ShPost,overshoot=1,soft=SmPost)


mt_merge(a,b,m,Y=3,U=3,V=3)
sD=mt_makediff(a,last)
fv2 = a.MVAnalyse(isb=false,delta=2,idx=13)
fv1 = a.MVAnalyse(isb=false,delta=1,idx=13)
bv1 = a.MVAnalyse(isb=true, delta=1,idx=13)
bv2 = a.MVAnalyse(isb=true, delta=2,idx=13)

sDD = sD.MVDegrain2(bv1,fv1,bv2,fv2,idx=14)

reduc = 0.4
sDD = mt_lutxy(sD,sDD,"x 128 - abs y 128 - abs < x y ?").mergeluma(sDD,1.0-reduc)

a.mt_makediff(sDD,U=2,V=2)

return last
}

LanczosResize(1280,720) # Lanczos (Sharp)
#denoise
ConvertToYV12()

http://img715.imageshack.us/i/yukiwithnewscript.png/

I still prefer the second option. Also I still want to remove the jaggies. Help would be greatly appreciated.

[list]

can't see any artifact there
BTW...

#FASTER ANTI ALIASING FILTER
#REQUIERED:
#    SangNom
#    mt_masktools

function FAA(clip input,int "str") {
	str = Default(str, 50)
	vid = input.IsYV12() ? input : input.ConvertToYV12()
	w=width(input)
	h=height(input)

	aa=vid.pointResize(w*2,h) . TurnLeft() . SangNom(AA=str) . TurnRight(). lanczosResize(w,h)

	return mt_merge(vid , aa , vid.mt_edge().mt_inflate() )
}

Maybe it's not the best anti aliasing filter, but 3x faster, and the first i can use in real time with some sources.

[MenotBumba]

http://img196.imageshack.us/img196/8...0111004222.png

I used these two lines, but the aliasing problem is still not solved as you can see on the screenshot above.

AnimeIVTC(mode=1,aa=4,killcomb=2,chrfix=false,ifade=false)
AAA(640,480,15,15,1,2,false)

Anyone knows how to fix it?

[mastrboy]

That looks more like a dotcrawl problem, little hard to tell by an image only...

Have a look at Tcomb(mode=0) or Checkmate()... If i remember correctly setting chrfix=true in AnimeIVTC also activates some dotcrawl removal? Or was it derainbowing?

[Bloax]

[old]Oh hey. I for some reason decided to mess around with this little script.

The end result ended up being twice as fast, though only because I reduced the initial resizing.

Well, basically you get a pair of extra options, and that's about it. (You can see it here.) <- Cleaned the post up a bit.

Oh shameless me, but I also wanted to stabilize the antialiasing. (Which is pretty spatial to me.)

So I ended up doing my very first 'real' script edit. Feel free to flame the hell out of my clumsy and useless edit.

(Found here)

All I seem to accomplish is create quite a bunch of artifacts.
Though it does seem a bit more stable.

I'm such a shameless bugga. ;4 [/old]

Edit:I made a new script today, with most of the dirty masktools business similar to these. (Updated, old version here.)
EditEdit: Another update, old version (Here)
Edit^3: Updated once again, old script here.

function m4(float x) {return(x<16?16:int(round(x/4.0))*4)} # Haha, didn't realize this wasn't internal. :p
# Me and my giant stash of scripts. (Well, more or less.)

function AntiAliasRG(clip orig,float "sx", float "sy",string "mode",int "thrl",int "thrc",bool "sharpen",float "sharpstr",bool "sharpmode") {

# Fast antialiasing using RemoveGrain. (It is dull no more!)

# > Requirements: MaskTools2, RemoveGrain(SSE3).dll, Repair(SSE3).dll <- For the sharpening.
# By Bloax, based on the "Antialiasing" script (>now pretty much original), authors of which are; SpikeSpiegel, Didée, mf, scharfis_brain and Soulhunter.

# This script is rather strong, especially when both sharpen and sharpmode are "true". At that point it looks like MSAA, which is quite sharp.
# It's also appliable to sources around 800x600 /before/ downscaling. ^ It's to be noted that it also denoises the video a bit.

# As of now, it seems better than the SangNom ones. Especially with the spooky results from sharpen/sharpmode=true.
# ^ Confirmed stuff, which turned out to be quite positive.

# UpdateUpdate: "sx" And "sy" are now the options to prevent (extreme) text smearing.
# Setting them below 1 works fine, though the speed gains aren't too big. (And the smearing gets much worse.)
# ^ Now, due to the addition of sharpen=true, in combination with sharpmode=true, you can actually get a nice speed boost at sx/sy=0.8 at next to no quality loss!

thrl = Default(thrl, 10) # Luma threshold for the mask(s).
thrc = Default(thrc, 10) # Chroma threshold for the mask(s). (Color, useless?)
sharpen = Default(sharpen, true) # See below.
sharpstr = Default(sharpstr, 0.4) # Added a sharpening filter. Experimental. (Effective though.)
sharpmode = Default(sharpmode, false) # Makes the edge mask use a sharper (custom) kernel. This nullifies the "mode" option though.
# Also degrades the performance a bit because of an extra blur filter. (Because why not.)
ox = orig.width
oy = orig.height
sx = Default(sx, 1.33) # Horizontal supersampling, reducing either to <1 will make text almost unreadable.
sy = Default(sy, 1.33) # Vertical supersampling, ^ unless you don't have sharpen=true, then it's alright to some degree.
mode = Default(mode, "laplace") # Kernel for detecting edges, I prefer this one.
# Possible strings (mode=>"option"<):  roberts (sharp), sobel (too sharp), laplace (pretty soft), prewitt (iseethemedges), hprewitt (prewitt light?)
clp = orig.IsYV12() ? orig : orig.ConvertToYV12() # Is the source in YV12? If not, convert to it. (Shameless ripoff!)

##############

a=clp.RemoveGrain(mode=1) # Subpixel noise removal added.
b=a.BilinearResize(m4(ox*sx),m4(oy*sy))
c=sharpmode ? b.RemoveGrain(mode=5).RemoveGrain(mode=21).RemoveGrain(mode=19).BilinearResize(ox,oy).RemoveGrain(mode=22) : b.RemoveGrain(mode=5).RemoveGrain(mode=21).BilinearResize(ox,oy).RemoveGrain(mode=22)
# RemoveGrain mode 5 is for removing noise, mode 19 is a blurring filter (No longer used), and 21/22 are kind of like SangNom antialiasing.
# sharpmode=true Adds a pass of mode 19, because why not while we're at it.
# Well, except that they're insanely fast. ;)
e=sharpmode ? a.RemoveGrain(mode=0,modeu=-1).Mt_Edge("2 3 3 2 -18 2 3 3 2",thy1=0,thy2=thrl,thc1=0,thc2=thrc).Mt_Inflate().RemoveGrain(mode=9,modeu=-1).Mt_Deflate() : a.RemoveGrain(mode=0,modeu=-1).Mt_Edge(mode=mode,thy1=0,thy2=thrl,thc1=0,thc2=thrc).Mt_Inflate().RemoveGrain(mode=9,modeu=-1).Mt_Deflate()
# Expanding the mask quite a bit, just to make sure the antialiasing gets a more or less smooth area.
# Also making the mask Luma only. (ModeU=-1) > Update: sharpmode=true, as mentioned, uses a custom kernel that's sharper than laplace. (But otherwise pretty identical.)
f=sharpen ? Mt_Merge(a,c,e).Sharpen(sharpstr).Repair(a,mode=9) : Mt_Merge(a,c,e) # Do we require sharpening? If yes, do this. If no, do the same thing without it.
# Why do I even have two Mt_Merge? < To reduce the blurring perhaps?
Mt_Merge(a,f,e).RemoveGrain(mode=22) # Killing the jaggies that got past the mask. ;)
}

This one uses RemoveGrain(SSE3).dll instead of SangNom, however. So it's less prone to make bizarre artifacts from subpixel noise. (Now thanks to prefiltering those out.)
So basically this is a softer antialiasing script (that looks pretty nice in my eyes). I may add an option for additional softening to it though.

Enjoy!

Third (latest) update: Changed the MaskTools stuff to MaskTools2 stuff. Which sped it up by a bit.
Also added sub-pixel aliasing removal by using RemoveGrain(mode=1). Edit: Also added Mt_Deflate() at the end of the edgemask to make the script slightly sharper.
EditEdit: Also added sharpening, at the price of a bit of performance. (140-155 FPS before, 130-140 after.)
These two latest edits should improve cases of overshooting.
^ Edit: Also added sharpmode, which uses a custom kernel to get a sharper mask. With sharpen=true it gets 120-127 FPS though.

Still, whereas I got 60 FPS with my lousy edit of the initial AntiAliasing(), I usually get about 130-140 FPS on the same source with this one.

Second update: Changed the RemoveGrain modes, ended up getting a very nice, and really fast antialiasing filter.
Also made the mask a bit wider and blurrier by using RemoveGrain(mode=19) on it, and by removing the Deflate() at the end.

First update: Cleaned some useless parts up, changed re-resizing to BilinearResize (instead of Gauss), and tweaked the preset settings a bit.
Also tweaked the mask a bit. Overall, it's faster, and a bit prettier. (Thanks to sx/y tweaking.)

P.S: If you want a pretty nice, and absolutely terrifyingly fast antialiasing filter, use this:

RemoveGrain(mode=1).RemoveGrain(mode=5).RemoveGrain(mode=22).RemoveGrain(mode=21)

Mode 1 to kill the smaller noise, mode 5 to kill the bigger noise, and 22 and 21 for the antialiasing. (Which make funky shapes out of the now removed noise.)

Update: Here's the ultimately simple AA script:

RemoveGrain(21,19,19).RemoveGrain(22)

[MenotBumba]

I have tested some filters(TComb, GuavaComb, DeCrawl etc.) and I got the best quality/speed results using TComb(mode=2) (when I put chrfix=true for AnimeIVTC, the encode was too slow - ~1fps). With TComb(mode=2) and chrfix=false I got ~ 3 fps

[naoan]

I like edge detection from lsfmod better than any kernel, a comparison :

laplace : http://i.imgur.com/sIszB.png
lsfmod : http://i.imgur.com/hLjDG.png

[luke83]

Originally Posted by Bloax

function AntiAliasRG(clip orig,float "sx", float "sy",string "mode",int "thrl",int "thrc") {

# Fast antialiasing using RemoveGrain. (It is dull no more!)

# > Requirements: MaskTools2, RemoveGrain(SSE3).dll
# By Bloax, based on the "Antialiasing" script (stealing parts of ;)), authors of which are; SpikeSpiegel, Didée, mf, scharfis_brain and Soulhunter.

# Note: ?Confirm?? This script is pretty weak, so tough edges (huge, curved ones) end up looking plain blurred. < It's rather strong now.
# Though if said source is downscaled, everything looks alright. (More or less better than just downsized.) < Applying it BEFORE downscaling gives superior results, for the price of some speed.

# ?Confirm?? It is to be noted that this script loses to the SangNom ones in slightly aliased cases. (Eg. Warcraft 3: The Frozen Throne title screen.)
# Though for eliminating tons of aliasing in a noisy game (Eg. Doom), this is certainly better than the usual SangNom ones.
# (Well, not so much now.) Due to the fact that it doesn't smoothen out the noise to abstract shapes, and instead eliminates it. (RemoveGrain. :))

# Due to recent changes the above may or may not be absolutely true. :-)
# (Because I'm totally way too lazy to rewrite all of this!)

# UpdateUpdate: "sx" And "sy" are now the options to prevent (extreme) text smearing.
# Setting them below 1 works fine, though the speed gains aren't too big. (And the smearing gets much worse.)

thrl = Default(thrl, 10) # Luma threshold for the mask(s).
thrc = Default(thrc, 10) # Chroma threshold for the mask(s). (Color)
ox = orig.width
oy = orig.height
sx = Default(sx, 1.33) # Horizontal supersampling, reducing either to <1 will make text almost unreadable.
sy = Default(sy, 1.33) # Vertical supersampling, once again, setting both of those results in getting a slower RemoveGrain(mode=21) filter.
mode = Default(mode, "laplace") # Kernel for detecting edges, I prefer this one.
clp = orig.IsYV12() ? orig : orig.ConvertToYV12() # Is the source in YV12? If not, convert to it. (Shameless ripoff!)

##############

a=clp.RemoveGrain(mode=1) # Subpixel noise removal added.
b=a.BilinearResize(m4(ox*sx),m4(oy*sy))
c=b.RemoveGrain(mode=5).RemoveGrain(mode=21).BilinearResize(ox,oy).RemoveGrain(mode=22)
# RemoveGrain mode 5 is for removing noise, mode 19 is a blurring filter (No longer used), and 21/22 are kind of like SangNom antialiasing.
# Well, except that they're insanely fast. ;)
e=a.RemoveGrain(mode=0,modeu=-1).Mt_Edge(thy1=0,thy2=thrl,thc1=0,thc2=thrc,mode=mode).Mt_Inflate().RemoveGrain(mode=9,modeU=-1)
# Expanding the mask quite a bit, just to make sure the antialiasing gets a more or less smooth area.
# Also making the mask Luma only. (ModeU=-1)
# > Switching to MaskTools2 ended up making the smoothening better.
f=Mt_Merge(a,c,e) # I'll probably be messing around with those. < Why do I even have two of these?
Mt_Merge(a,f,e).RemoveGrain(mode=22) # Killing the jaggies that got past the mask. ;)
}

This one uses RemoveGrain(SSE3).dll instead of SangNom, however. So it's less prone to make bizarre artifacts from subpixel noise. (Now thanks to prefiltering those out.)
So basically this is a softer antialiasing script (that looks pretty nice in my eyes). I may add an option for additional softening to it though.

Third (latest) update: Changed the MaskTools stuff to MaskTools2 stuff. Which sped it up by a bit.
Also added sub-pixel aliasing removal by using RemoveGrain(mode=1).

Still, whereas I got 60 FPS with my lousy edit of the initial AntiAliasing(), I usually get about 130-140 FPS on the same source with this one.

Hi. I think your script would be amazing... If I could use it XD

It gives me an error window in witch it says that "m4 function doesn't exist"
In fact I looked for it in the script, but I didn't found it; or well: I found where you use it but I hasn't be declared before. I "loaded" in the avs script mt_masktools 2 and 26, RemoveGrainSSE3 0.9 and 1.0 but It doesn't work

This is my avs script

Where I make a mistake (If there's one)

LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\Anime\RemoveGrain-1.0\RemoveGrainSSE3.dll")
LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\Anime\mt_masktools2.dll")
LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\Decomb.dll")
LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("K:\Temp\Temp Megui\File 1\lupin.d2v", info=3)
LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, threads=0)
# Deinterlacer
AssumeTFF().Telecide(guide=1).Decimate(mode=1,threshold=2.0)
# Cropper
crop( 10, 2, -12, -6)

# Resizer
Spline16Resize(704,528) # Spline16 (Neutral)
# Denoiser and Anti-aliasing
Import("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\Anime\Anti-aliasing\AARG.avs")
AntiAliasRG()

As you can read, I'm using MeGui but the problem is relative to avs script only

So, because you can run your script, one of us has made a mistake: you have forgotten something in you explanation or I'm a newbby

[Didée]

"m4 function" supposedly is this one

function m4(float x) {return(x<16?16:int(round(x/4.0))*4)}

[luke83]

Originally Posted by Didée

"m4 function" supposedly is this one

function m4(float x) {return(x<16?16:int(round(x/4.0))*4)}

Why it isn't in the script?

However, I added before AntiAliasingRG function and now it works

Can I use Spli16Resize instead of Bilinear? This last one makes soft sharp. Spline16 is Neutral. I'm compressing in mkv my Lupin DVDs; the image is not defined as modern anime and the sharpening would be not good

EDIT:
I tried to use Spline16Resize instead of Bilinear but nothing is changed.

Your script is really fast compared with AAA (very slow) but this one (AAA) doesn't smooth as your function. The images are too smoothed.

Script with DeGrainMedian and AAA

LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\Anime\Anti-aliasing\UnFilter.dll")
LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\Anime\Anti-aliasing\SangNom.dll")
LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\Anime\degrainmedian082\degrainmedian.dll")
LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\Decomb.dll")
LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("K:\Temp\Temp Megui\File 1\lupin.d2v", info=3)
LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, threads=0)
# Deinterlacer
AssumeTFF().Telecide(guide=1).Decimate(mode=1,threshold=2.0)
# Cropper
crop( 10, 2, -12, -6)

# Resizer
Spline16Resize(704,528) # Spline16 (Neutral)
# Denoiser
DeGrainMedian()
# Anti-aliasing
Import("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\Anime\Anti-aliasing\AAA.avs")
AAA()

Script with AntiAliasingRG

LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\Anime\RemoveGrain-1.0\RemoveGrainSSE3.dll")
LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\Anime\mt_masktools2.dll")
LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\Decomb.dll")
LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("K:\Temp\Temp Megui\File 1\lupin.d2v", info=3)
LoadPlugin("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, threads=0)
# Deinterlacer
AssumeTFF().Telecide(guide=1).Decimate(mode=1,threshold=2.0)
# Cropper
crop( 10, 2, -12, -6)

# Resizer
Spline16Resize(704,528) # Spline16 (Neutral)
# Denoiser
# Anti-aliasing
Import("K:\Documents and Settings\Luca\Documenti\Software\Video Editing\MeGUI_0.3.5_x86_WithoutInstaller_OfflinePackage\tools\avisynth_plugin\Anime\Anti-aliasing\AARG.avs")
AntiAliasRG()

You can see the effect overall on the faces of the far people

[Bloax]

Originally Posted by luke83

Why isn't it (m4) in the script?

Because I didn't realize it wasn't internal. ;P

Originally Posted by luke83

Can I use Spli16Resize instead of Bilinear? This last one makes soft sharp. Spline16 is Neutral.

Spline16 should actually be a bit sharp, but I went for Bilinear to get a little extra performance. Also because haloed antialiasing is automatically bad.

Originally Posted by luke83

Your script is really fast compared with AAA (very slow)

Glad it's good in terms of performance, I kinda went for that.

Originally Posted by luke83

The images are too smoothed.

That's what happens when you don't have anything but old games to tune for!

But anyway, the only thing I can think of is replacing this line:

e=a.RemoveGrain(mode=0,modeu=-1).Mt_Edge(thy1=0,thy2=thrl,thc1=0,thc2=thrc,mode=mode).Mt_Inflate().RemoveGrain(mode=9,modeU=-1)

With this:

e=a.RemoveGrain(mode=0,modeu=-1).Mt_Edge(thy1=0,thy2=thrl,thc1=0,thc2=thrc,mode=mode).Mt_Inflate().RemoveGrain(mode=9,modeU=-1).Mt_Deflate()

Which should keep the Inflated smoothness of the mask, meanwhile being slightly compacted.

Because as far as I can see, it's the mask being way too wide.
And thus adding quite a bit of the blurry mess to the image.

If that doesn't work, try experimenting with the thrl option, and perhaps pumping sx/sy up to 1.8/2. (Both of which ought to at least slightly reduce said issue.)

And I said "ThrL" because "ThrC" should supposedly do absolutely nothing as of now, since I made the mask operate on a (supposedly) discolored image.

Oh, and by the way - I added the m4 function to the script now.
Thankies to you for it. :P

[luke83]

I GOT IT!!!!
Your mod didn't work so I think to use Unfilter. For my anime (Lupin 2nd Season) after AntialiasRG I put Unfilter(70,70) and the result is the same

EDIT:
Unfilter(60,60) otherwise I have light aliasing on the lips

[Bloax]

Hm, combining "mode="roberts"" with Mt_Deflate() at the end of the mask seems to provide sharper results. (Added that to the main script.)

But yeah, it does need some sharpening. If I can find something appropriately fast, I may add it in as an option.

As for now, that should work.

(I also added a little list of options for mode="blah", just because I felt like it.)

Edit: Also added sharpening now. Do test it. (sharpen=true/false,sharpstr=stuff)
Performance suffers a bit (140-155 FPS Average/Peak, now 130-140 Average/Peak on the same source), but that shouldn't be too much.

And I say "do test it" because I don't have too much to test it on. :-)

EditEdit: Oh boy, format breaking time!

But in other news, added YET ANOTHER sharpening feature, a sharpmode for a sharper kernel for the mask.
That, in addition to sharpen=true makes this look a bit like MSAA. (Oops!)

P.S: Hm, with both of those, sx/sy=0.5 don't really seem to degrade too much. Even 0.8 provide quite a bit of performance for no visible to me quality degradation.
Could someone investigate this one? (Fine wording for test? )


Next day: Confirmed some stuff, which is, updated the random comments inside it that describe it.

[bcn_246]

Originally Posted by Porsche_fan

...Lastly, I have found the original version of SangNom is about 4x faster on my quadcore than the more current versions. Unfortunately it does have a very slightly lower quality but it is hardly perceivable. http://mf.creations.nl/avs/filters/SangNom_old2.dll...

Just wondered if anybody could re-upload this version. I can only find the beta (newest) version of the DLL (date=18/01/2004, crc32=4e79e77e).

TIA

[bcn_246]

Originally Posted by Porsche_fan

...Lastly, I have found the original version of SangNom is about 4x faster on my quadcore than the more current versions. Unfortunately it does have a very slightly lower quality but it is hardly perceivable. http://mf.creations.nl/avs/filters/SangNom_old2.dll...

Originally Posted by bcn_246

Just wondered if anybody could re-upload this version. I can only find the beta (newest) version of the DLL (date=18/01/2004, crc32=4e79e77e).

TIA

Probably shouldn't but *bump*... really am interested in getting hold of the above file...

Sorry guys

[Akai-Shuichi]

I'm curious if anybody could help me out with how to go about anti-aliasing in this anime?

Here is a sample clip: http://www.sendspace.com/file/lozi3h

It is almost as if it slightly wobbles with the aliasing there.

[Didée]

I've found three more versions of sangnom.dll on my HDD, which all do differ by CRC. <click>

Though, I've no clue if any of those is faster or slower or something. You try.

[bcn_246]

Thanks Didée

I'll try them and report back...

[bcn_246]

Okay...

On i7@4.2GHz with source on a ~450MB/s SSD, just a simple:

AVISource("Uncompressed 1920x1080p60 YV12 Input.avi")
SangNom()

SangNom.dll (CRC32: 4E79E77E)
~45.5 FPS

sangnom_old.dll (CRC32: 60BE3D92)
~2.9 FPS

sangnom_old2.dll (CRC32: 232EF465)
~23.9 FPS

sangnom2.dll (CRC32: 725B1C38)
Couldn't get it to load (Access Violation)

[WooWoo]

Hi, I need some help. I am trying to encode to x264 using aaa() loading the latest versions of SangNom and UnFilter and importing AAA.avs (I got all the files today from the avisynth wiki). Everything looks right in the preview window for programs such as VirtualDub and MeGUI. However, when my x264 encode finishes, the height is double the size (width is the same), and the fps is halved. So instead of it being 640x480 @ 23.976 FPS, it is 640x960 @ 11.988 FPS. When I encode to XviD, everything comes out at the correct resolution and FPS using the same script. All I'm doing is loading the source file and plugins/scripts needed for AAA(), and then running AAA() in my avs script.

I then tried out each version of SangNom that Didée posted and bcn_246 tested out. They all resulted in the the same problem except for sangnom_old.dll (CRC32: 60BE3D92). This particular version worked perfectly without changing the resolution or FPS, however it reduced my FPS to a laughable ~2 FPS on my i5 quad core machine. Anybody else having this issue, or am I not loading the correct versions of everything, or what's going on exactly? Why is it changing my resolution and fps?