Anti-aliasing script

[SpikeSpiegel]

function antialiasing(clip orig,int "th_luma",int "th_croma",string "type",int "aath") {

# "th_luma" & "th_croma" are the edge detection thres.: lower values=more edges filtered
# "type" is the matrix used for edge detection: with "sobel" (default) only the 
#     hi-contrast edges, where artefacts are more noticeable, are filtered. If you want
#     to test other matrices, read the MaskTools Guide for more info.
# "aath" = anti-aliasing strenght (default should be fine)

th_luma  = Default(th_luma, 20)
th_croma = Default(th_croma, 20)
type     = Default(type, "sobel")
aath     = Default(aath, 48)
ox = orig.width
oy = orig.height
dx = orig.width * 2
dy = orig.height * 2
clp = orig.IsYV12() ? orig : orig.ConvertToYV12()

a=clp
b=clp.Lanczos4Resize(dx,dy).TurnLeft().SangNom(aa=aath).TurnRight().SangNom(aa=aath) \
.LanczosResize(ox,oy)
c=clp.EdgeMask(th_luma,th_luma,th_croma,th_croma,type)
MaskedMerge(a,b,c)

}

Hi!

This filter is meant for good quality clips with soft antialiasing (like DVs);
thanks to MaskTools' functions, a strong AA-filter (Sangnom) is applied only where it's needed without producing artefacts typical of this filter.

Requires:
-MaskTools
-SangNom

Let me know what do you think about it!

P.S. Thanks to Didée, mf, scharfis_brain and Soulhunter (in alphabetical order) for their help!

EDIT1: Ooops! Now, with Lanczos4Resize instead of PointResize, there's no definition loss nor artefacts.
...and now SangNom's config. is aviable
EDIT2: Credits
EDIT3: Fixed a typo, erased a useless option, more info added.

[Didée]

Did a quick comparison of some methods. I post the script as well, in case I've made an error in the hurry ...



script:

mpeg2source()
crop(much)
orig=last
ox=last.width
oy=last.height
jaggy=orig.separatefields().selecteven()
SangN=jaggy.antialiasing().lanczosresize(ox,oy)
SangNSS=jaggy.lanczosresize(ox,oy).antialiasing()
tom=jaggy.tomsmocomp(1,-1,0)
tomSS=jaggy.lanczosresize(ox,oy).tomsmocomp(1,-1,1).lanczosresize(ox,oy)
trit=orig.tdeint(mode=0,order=1,field=1,mthreshL=-1,mthreshC=-1,type=1)
tritSS=jaggy.pointresize(ox,oy).tdeint(mode=0,order=1,field=1,mthreshL=-1,mthreshC=-1,type=1)
dble=jaggy.pointresize(ox,oy).tdeint(mode=0,order=1,field=1,mthreshL=-1,mthreshC=-1,type=1)
 \                           .tomsmocomp(1,-1,1).lanczosresize(ox,oy)
dbleS=dble.LimitedSharpen(ss_x=2.0,ss_y=1.0,Smode=3,strength=200,Lmode=2,overshoot=1)

# without the dreadful subtitle commands:
stackvertical(
 \ stackhorizontal(jaggy.lanczosresize(ox,oy).subtitle(jaggystr,size=12),orig),
 \ stackhorizontal(SangN,SangNSS),
 \ stackhorizontal(tom,tomSS),
 \ stackhorizontal(trit,tritSS),
 \ stackhorizontal(dble,dbleS) )

return last

edit: better placing of cropped area.

[SpikeSpiegel]

Originally posted by Didée
Did a quick comparison of some methods. I post the script as well, in case I've made an error in the hurry ...

script:

mpeg2source()
crop(much)
orig=last
ox=last.width
oy=last.height
jaggy=orig.separatefields().selecteven()
SangN=jaggy.antialiasing().lanczosresize(ox,oy)
SangNSS=jaggy.lanczosresize(ox,oy).antialiasing()
tom=jaggy.tomsmocomp(1,-1,0)
tomSS=jaggy.lanczosresize(ox,oy).tomsmocomp(1,-1,1).lanczosresize(ox,oy)
trit=orig.tdeint(mode=0,order=1,field=1,mthreshL=-1,mthreshC=-1,type=1)
tritSS=jaggy.pointresize(ox,oy).tdeint(mode=0,order=1,field=1,mthreshL=-1,mthreshC=-1,type=1)
dble=jaggy.pointresize(ox,oy).tdeint(mode=0,order=1,field=1,mthreshL=-1,mthreshC=-1,type=1)
 \                           .tomsmocomp(1,-1,1).lanczosresize(ox,oy)
dbleS=dble.LimitedSharpen(ss_x=2.0,ss_y=1.0,Smode=3,strength=200,Lmode=2,overshoot=1)

# without the dreadful subtitle commands:
stackvertical(
 \ stackhorizontal(jaggy.lanczosresize(ox,oy).subtitle(jaggystr,size=12),orig),
 \ stackhorizontal(SangN,SangNSS),
 \ stackhorizontal(tom,tomSS),
 \ stackhorizontal(trit,tritSS),
 \ stackhorizontal(dble,dbleS) )

return last

edit: better placing of cropped area.

Hi!

I've already done a few tests (always with a deinterlaced DV) with TomsMoComp(x,1,1), but results weren't really encouraging:
with original resolution it removes almost all artefacts, but details are blurred; vice-versa with double height (good definition, but too much alias).

On the contrary, TDeint+doubleheight works quite well: results are similiar to "antialiasing", but there's a little definition loss.

Anyway, give a try to antialias (the last "version") with a hi-medium resolution source: this script is meant for good quality movies, that why it didn't perform well with the WB sample.

[Bulletproof]

Looks promising but it would be really great if someone could figure out how faroujda's DCDI method works to create an avisynth plugin. I personally own 3 devices with the DCDI chip for deinterlacing and it is pretty amazing when it comes to deinterlacing, it really eliminates jaggies good and keeps alot of resolution, here is an example:



It's not that great of a sample but if you've seen DCDI in action you will see that there is nothing that can beat it right now (I think so anyway). More detailed explination here: http://www.av-outlet.com/en-us/dept_335.html

[Didée]

Originally posted by SpikeSpiegel
Anyway, give a try to antialias (the last "version") with a hi-medium resolution source: this script is meant for good quality movies, that why it didn't perform well with the WB sample.

Erh, sorry. If the quality of that sample is not sufficient for your script, then your script is useless. The given sample is perfectly valid (It's the original PAL trailer of HPII, in full resolution. I just cropped down to an 'interesting' area).
In fact, most sources that do need antialiasing will be of not-so-impressing quality to start with - most probably they'll be of lower quality than the sample I used. And just discarding one field from a progressive sequence, as I did here, is a good simulation of common aliasing problems (capturing only one field, dumb bobbing, one field useless due to fieldblending, etc. pp.)


However, your script has a couple of issues that could be improved:


1.
You're offering the aa threshold for sangnom, but it is not used in the script. Your call of SangNom should read

SangNom(order,aath)


2.
Chroma probably is just passed through (it IS on that P-III I'm sitting at right now).
Better would be:

c=edgemask(...).FitY2UV()

I've found that in many cases, it's better to use the luma edges for chroma also, insted of creating edges on chroma directly.

If you prefer to use "native" chroma edges, then you must still write

c=edgemask(... , Y=3,U=3,V=3)

Else, you won't get any edges on the chroma planes at all (as in your posted script).

Finally, you should write

MaskedMerge(a,b,c, Y=3,U=3,V=3)

AFAIK, MaskedMerge's default is to leave chroma alone, unless one specifies U|V=3.


3.
As it is typical for EDI resamplers, SangNom too delivers an output that is spatially shifted from the input. This would not be a big issue ... if you would create the edgemask on the shifted frame. But you are creating it on the not-shifted input, and apply it later on the shifted output. That's not ideal. So, the declaration of "c" better should read

c=b.edgemask( .. ) ..


Below script addresses abovementioned issues, and includes these additional changes:

- added "HQedge" [bool]. It's slower, reckognizes edges better, is more robust against noise ... and perhaps is not suited at all for this application. So, it's just for testing & playing.
- normal edgemask is created on the original sized frame. HQedgemask is created on the (for sangnom) double-sized frame.

Without much testing:

function antialiasing( clip orig, int "th_luma", int "th_croma", string "type", int "order", int "aath", bool "HQedge") 
{

# "th_luma" and "th_croma" affect directly the edge detection: higher values = more edges filtered 
# set "order = 0" for Top Field First; order = 1 --> Bottom Field First
# aath = anti-aliasing strenght (default should be fine)


th_luma  = Default(th_luma, 20)
th_croma = Default(th_croma, 20)
type     = Default(type, "sobel")
order    = Default(order, 1)
aath     = Default(aath, 48)
HQedge   = Default(HQedge, false)
ox = orig.width
oy = orig.height
dx = orig.width * 2
dy = orig.height * 2

orig.convertToYV12()
a=last
b=lanczos4Resize(dx, dy).TurnLeft().SangNom(order,aath).TurnRight().SangNom(order,aath)

# native chroma edges:
#c=b.lanczosresize(ox,oy).EdgeMask(0, th_luma, 0, th_croma, type,Y=3,U=3,V=3)

# use luma edges for chroma:
c=b.lanczosresize(ox,oy).EdgeMask(0, th_luma, 0, th_croma, type,Y=3,U=1,V=1).FitY2UV()

d= logic( b.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=1,V=1)
 \       ,b.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=1,V=1)
 \       ,"max").greyscale.levels(0,0.8,128,0,255,false).FitY2UV().lanczosresize(ox,oy)
b=b.lanczosresize(ox,oy)

HQedge ? MaskedMerge(a,b,d,Y=3,U=3,V=3) : MaskedMerge(a,b,c,Y=3,U=3,V=3)
}

Just some suggestions.

[SpikeSpiegel]

@ Didée:

Thanks for your help, as you may have noticed (the 1st error was quite exemplifying ) I didn't pay too much attention on the creation of the script itself (that needs MUCH improving), rather I wanted to give an idea, i.e.:

For aliasing is present only on edges, why we should filter the entire frame?

And the main reason of filtering only what needs it (edges), in this case, is not speed gain (...and you should know why ), but to keep max definition.
Obviously, using a bad-quality source (where there's something more than simple alias), there's no need of this, that's why I wrote that it's more suiteg for medim-good quality stuff (even if this reduces the possible applications).

Anyway the EdgeDetection is tuned for sharp vids, then, if you are testing with mpeg2 samples, you should increase the threshold.

Now I don't have much free time, but asap I'll test HQedge and the other settings!


P.S. I don't see any difference between:

c=edgemask( .. )

and

c=b.edgemask( .. ).FitY2UV()

[Soulhunter]

Hmm...

Anti-aliasing via SangNom seems to work !!!

Source



SangNom



Think the edge-detection is the culprit ???

-> Range is not wide enough...


Btw, I thought the purpose is simple aliasing reduction !?!

But it seems you wanna use it to reduce interlace-leavings...


Bye

[Soulhunter]

And it works very good @ anime/toon sources !!!

Sample with bad aliasing...

Source



SangNom




Bye

[SpikeSpiegel]

Originally posted by Soulhunter
Anti-aliasing via SangNom seems to work !!!

Hi!

Did you use only SangNom or the anti-alias script?

Originally posted by Soulhunter
Btw, I thought the purpose is "simple" anti-aliasing !?!

But it seems you wanna use it for interlace-leavings...

No, absolutely not, as you can see in the Didée's test, it doesn't work properly with interlace-leavings; anyway, some adaptive deinterlace filters may produce aliasing, that can be easily erased with a script like this, but only if there aren't other kinds of imperfections.

[Soulhunter]

Originally posted by SpikeSpiegel

Did you use only SangNom or the anti-alias script?

For the samples above Ive simply used the stuff mf told me...

LanczosResize(X*2,Y*2).TurnLeft().SangNom().TurnRight().SangNom().LanczosResize(X,Y)


Bye

[scharfis_brain]

try

pointResize(X*2,Y*2).TurnLeft().SangNom().TurnRight().SangNom().LanczosResize(X,Y)

[Soulhunter]

Originally posted by scharfis_brain
try

pointResize(X*2,Y*2).TurnLeft().SangNom().TurnRight().SangNom().LanczosResize(X,Y)

Uhm, actually this is what mf told me...

before = last
PointResize(width*2, height*2)
ConvertToYUY2()
TurnLeft()
ConvertToYV12()
SangNom()
ConvertToYUY2()
TurnRight()
ConvertToYV12()
SangNom()
LanczosResize(width/2, height/2)
MergeChroma(before)

Sorry n' Bye

[Soulhunter]

Ok, a small update...

Source



Lanczos->SangNom->Lanczos



PointResize->SangNom->Lanczos



Lanczos->SangNom->Bilinear



Lanczos->SangNom->Bilinear->Unfilter(50,50)




Imo the PointResize sample is sharp but shows still some aliasing as the Lanczos sample...

The Bilinear/Unfilter sample looks same sharp but has near like no aliasing left !!!

#################################################################

X = 320 # Resolution width
Y = 240 # Resolution height

A = 50 	# Unfilter value A
B = 50 	# Unfilter value B

#################################################################

AviSource("C:\MovieBackups\Source.avi").ConverttoYV12()

#################################################################

Before = Last

LanczosResize(X*2,Y*2)

ConvertToYUY2()

TurnLeft()

ConvertToYV12()

SangNom()

ConvertToYUY2()

TurnRight()

ConvertToYV12()

SangNom()

BilinearResize(X,Y)

Unfilter(A,B)

MergeChroma(Before)

#################################################################

But I dunno how to make a function out of this...

AnimeAntiAliasing: AAA(X,Y,A,B)


Bye

[Didée]

What's that, Soulhunter?

A brigade of inbetween "ConvertTo...ConvertTo...ConvertTo...", and in the end, the unprocessed original chroma is copied back. Perhaps one should throw some lsresize()'s in the mix, to make those NOPs even more valuable

However, making a function out of that is pretty simple:

You just have to /*whisper...whisper...whisper*/, that's all

[Soulhunter]

# _______________________________________________________
#
# SAA | Simple Antialiasing
# _______________________________________________________
#					
# Simplicity by Soulhunter... \m\ ^_^ /m/
# _______________________________________________________
#
#
# Requirements:
#
# SangNom.dll
# 100GHz CPU
#
#

Function SAA(Clip Clp,Int"SS",Bool"CP")
{

OX = Clp.Width
OY = Clp.Height
SS = Default(SS,2)
CP = Default(CP,True)


Clp = Clp.IsYV12() ? Clp : Clp.ConvertToYV12()

Clp.PointResize(OX*SS,OY*SS).SangNom().TurnRight()
\  .SangNom().TurnLeft().BilinearResize(OX,OY)

CP ? Last : MergeChroma(Clp)

Return(Last)

}

Bye

[SpikeSpiegel]

Originally posted by Didée
As it is typical for EDI resamplers, SangNom too delivers an output that is spatially shifted from the input. This would not be a big issue ... if you would create the edgemask on the shifted frame. But you are creating it on the not-shifted input, and apply it later on the shifted output. That's not ideal. So, the declaration of "c" better should read

c=b.edgemask( .. ) ..

There's still a great problem:
this way, the base clip is still in the original position, then, overlaying base & overlay clip (shifted), I get artefacts as well (=alias on one side of edges).

If the SangNom's "shift vector" is (1,1) (as it seems to be), I should use something like:


orig = clip.ConvertToRGB()
clp = orig.crop(0,1,-1,0).addborders(0,1,1,0).ConvertToYV12()
shift = orig.crop(1,0,0,-1).addborders(0,1,1,0).ConvertToYV12()

...

a= clp
b= shift.Lanczos4Resize(dx, dy).TurnLeft()...
c= clp.EdgeMask(0, th_luma, 0, th_croma, ....)


But, if I wrote "There's still a great problem", there must be a reason :
actually the "shift vector" shouldn't have integer components (it's more like (1/2,1/2) ), then...I don't see any solution!

EDIT1: fixed crop&addborders

[scharfis_brain]

I don't see any solution!

upsize using with something better than pointresize to 2x & 2y.

then shift by the desired vector and to a point-downsize to the original resolution.

[SpikeSpiegel]

Originally posted by scharfis_brain
I don't see any solution!

upsize using with something better than pointresize to 2x & 2y.

then shift by the desired vector and to a point-downsize to the original resolution.

I've already tried something like this with (1/2,1/2--> too much) and then with (1/4,1/4--> closer):

ox = clip.width
oy = clip.height
qx = clip.width * 4
qy = clip.height * 4
orig = clip.ConvertToRGB()
clp = orig.Lanczos4Resize(qx, qy).crop(0,1,-1,0).addborders(0,1,1,0).ConvertToYV12().LanczosResize(ox, oy)
shift = orig.Lanczos4Resize(qx, qy).crop(1,0,0,-1).addborders(0,1,1,0).ConvertToYV12().LanczosResize(ox, oy)

...but still I can't find the right vector (that maybe has different components, like (x,y) ) and the speed loss is already excessive.
I suppose that a fast "shiftVector()" plugin doesn't already exist, isn't it?

[Didée]

Originally posted by SpikeSpiegel
I suppose that a fast "shiftVector()" plugin doesn't already exist, isn't it?

No, alas not.

But you can take either GeneralConvolution() or MaskTool's DEdgeMask(), together with pen & paper, and produce the frame shifts yourself through an operation on a 3x3 kernel:

  0  0  1
  0  2  0
  0  0  0

with normalization 3 gives a shift of 1/3 to the upper right.
Since this is simple bilinear sampling, you'll loose sharpness as well. Perhaps upsample first to 2x with lanczos, shift by 2x of the initial target vector, and downsize back.

[SpikeSpiegel]

Originally posted by Didée
No, alas not.

But you can take either GeneralConvolution() or MaskTool's DEdgeMask(), together with pen & paper, and produce the frame shifts yourself through an operation on a 3x3 kernel:

  0  0  1
  0  2  0
  0  0  0

with normalization 3 gives a shift of 1/3 to the upper right.
Since this is simple bilinear sampling, you'll loose sharpness as well. Perhaps upsample first to 2x with lanczos, shift by 2x of the initial target vector, and downsize back.

Good idea, but there is another solution:
throw away PointResize.

The "culprit" wasn't SangNom: using lanczos4resize to upsize, there's no shift at all.

Thanks anyway, this method is really faster and I'm sure it will be useful in other situations.