AntiAliasHQ - Oh snap!

[Bloax]

Oh snap, it's me!

And it's another antialiasing script, argh!!
:stupid:

Anyways, this is a new (whoah!) script by (silly) me. ;)
It's made to kill jaggies, using both the old SangNom and the RemoveGrain mode 21.

And since it uses nothing else, it doesn't really smooth much.
It does include a no-edge sharpening, though it gets outside edges a lot at thr<5

It's kinda fast (read: kinda), too. Because a good additional script is a fast additional script. :)

Feel free to try it out, also brownie points to redfordxx for the awesome plugin. (I love messing with differences.)

Have fun!

function AntiAliasHQ(clip clp,int "thr",int "aa",bool "sharpen",float "sharp") {
# Another antialiasing script by, you guessed it - Bloax.
# Extra cookies to redfordxx for the masking simplification idea. (Now there are two masks, instead of two masks and two inversions!)

# I doubt this one's really usable for much, but I'll just go ahead and post it -
# Since it 'bypasses' a SangNom issue hackily, and combines it with RemoveGrain mode 21.

# It's also pretty fast. No supersampling for you though!
# But overall, it doesn't really "smooth" much, it does kill jaggies quite nicely though. (Game footage, anyone?)

# Requirements: RemoveGrain, RedAverage.dll (You can google it), SangNom, MaskTools2

# "thr" Is the threshold for detecting edges, higher values INCREASE the amount of edges found.
# It's operating in Pow2, so be careful about the values. (6 is 64, while 8 is >256< :P) [/basicmath]
# > Seems like lower values are softer. Lower values sharpen more though, whilst higher sharpen less.
# Due to the fact that when more edges are "detected", there's less overall space in the inverse edge mask.
# Which is what the sharpening is limited to.

# "aa" Is a silly little value for SangNom, I have no idea about what it does above 48-64.
# "sharpen" Controls whenever there's internal (non-edge) sharpening, and "sharp" controls the strength.

# Enjoy!

thr = Default(thr, 5)
aa = Default(aa, 1024)
sharpen = Default(sharpen, true)
sharp = Default(sharp, 0.33)

thrval = Pow(2,thr)

clp2 = IsYV12(clp) ? clp : clp.ConvertToYV12() # Don't we love to copy this? :-)

edgemsk=RAverageW(clp2,-thrval,clp2.RemoveGrain(19,-1),thrval,mode=4,u=0,v=0,bias=256)
aa1=clp2.SangNom(0,aa)
msk1=RAverageW(clp2,-64,aa1,64,mode=4,u=0,v=0,bias=-64)
aa2=clp2.SangNom(1,aa)
merge=RMerge(aa2,aa1,msk1.Mt_Expand().Mt_Deflate().Mt_Inflate(),mode=255)
clean=sharpen ? RMerge(merge,clp2.Sharpen(sharp),edgemsk.Mt_Deflate(),mode=255) : RMerge(clp2,merge,edgemsk.Mt_Deflate(),mode=255)
RMerge(clean.RemoveGrain(21),clean,edgemsk,mode=255)
}

P.S: Yes, I do mask PROPERLY here. :devil:
..Or do I!?

[Previous version] (http://pastebin.com/4eWHBedP)

[redfordxx]

Some speedup?edgemsk=RAverageW(clp2,-thrval,clp2.RemoveGrain(19,-1),thrval,mode=4,u=0,v=0,bias=255+128)
RMerge(clean.RemoveGrain(21),clean,edgemsk,mode=255)
same with invmsk1 maybe?

[Bloax]

Hm, is doing the whole averaging faster than a simple reversion?

[redfordxx]

Hm, is doing the whole averaging faster than a simple reversion?The thing is, you are adding the processing time of reversion to the processing time of averaging which happens anyway. And it can all be done in the averaging
And by the way, the filter works on two clips at once so the processing time of even and odd number of clips should be almost the same, provided you don't have bottleneck in the data transfer.
I could not mesure the exact speed coz I have bottle neck in data transfer (10GB/s = ~1000FPSFullHD and the filter is faster;))

[Bloax]

I'm actually talking about running RemoveGrain(19,-1) again to get the edges in reverse, instead of just reversing the existing mask. :)

[Gavino]

He's not running RemoveGrain() again -you also remove the assignment to 'edges' which is no longer used.

[Didée]

P.S: Yes, I do mask PROPERLY here. :devil:

No, you don't. The masks are missing 50% of area-of-interest, because you're cutting the darker half.

[Bloax]

Hm, are you guys talking about instead of doing RMerge(last,stuff,edgeshere) & RMerge(previousstuff,newstuff,invertedges) Then just do RMerge(stuff,last,invertedges) & RMerge(morestuff,previousstuff,invertedges) ?
If so, then I finally see your point. It's just that I was in a hurry when I wrote the previous post.

I'll mess with it some time later (maybe today), since I'm kinda busy on wednesdays. ;)

No, you don't. The masks are missing 50% of area-of-interest, because you're cutting the darker half.
D: !

Are we talking about the bias here?
If so, then it's for conservative edge-y reasons. Because the way I'm getting the edges is horribly hacky, and thus unbiased averaging results in a vast overkill of a detection.

What I'm talking about; Bias 0 (http://img812.imageshack.us/img812/9617/mask1e.png) - Bias -128 (http://img843.imageshack.us/img843/8143/mask2.png)

Now to find out whenever Mt_Edge is faster or not (LOL!), but I certainly find the amount of edges found appropriate.
As said, I'm a silly, silly guy. :stupid:

[redfordxx]

I don't get why are you talking about removegrainedges=RAverageW(clp2,thrval,clp2.RemoveGrain(19,-1),thrval*-1,mode=4,u=0,v=0,bias=-128)
edgemsk=RAverageW(edges,-1,mode=8,u=0,v=0,bias=255) # Inverse edge mask, for removing SangNom weirdness outside edges.

edges=thrval*x+thrval*-1*y-128
edgemsk=255-1*edges
==========
edgemsk=255-(thrval*x+thrval*-1*y-128)=255+128-thrval*x+thrval*y

You replace two lines of your code with the one I posted in the beginning, but you must also switch the sides in RMerge()

I believe invmsk1 is the same case.

[Bloax]

I did finally get it now, it's just that I was in a hurry when I wrote that post. (And thus didn't get the point.)

Now to try it out.

Alright, it seems a bit faster now. Cookies are to be handed out.

Edit: edge=RAverageW(last,32,last.RemoveGrain(19,-1),-32,u=0,v=0,bias=-64,mode=4)
aa1=RemoveGrain(15)
msk=RAverageW(last,32,aa1,-32,u=0,v=0,mode=4)
RMerge(last,RMerge(RemoveGrain(16),aa1,msk.Mt_Inflate()),edge)
Seems like deinterlacers can be used for antialiasing in general. Hah!

[Bloax]

Hm, during my rather short observance, I noted that
(Deinterlancer).TurnRight().(Deinterlacer).TurnLeft()
Provides really sub-par results.

A better solution (for antialiasing at least), is to do this:
Merge(Deinterlacer(TopField),Deinterlacer(BottomField),0.5)

Sense? Dunno.
Good results? You bet.

[Didée]

Sense? Dunno.
Good results? You bet.
Sure so!

It made sense and gave good results five years ago (http://forum.doom9.org/showthread.php?p=907915#post907915), did ever since, still does, and will continue to do. ;)

[dansrfe]

^ and that's the "oh snap" lol

[Bloax]

I wonder why you didn't scream in their face that they're doing something really stupid then. ;P
Since honestly, Merge(SangNom(0,255),SangNom(1,255),0.5) looks way better than SangNom(0,255).TurnRight().SangNom(0,255).TurnLeft()

[cretindesalpes]

They do different things. The former method generally erases any 1-pixel-high vertical detail. It is suited to post-process deinterlacing or IVTC problems (horizontal aliasing, residual combing, etc). It's the DAA() (http://avisynth.org/mediawiki/DAA) way.

You generally use the latter method on 2x-upscaled data to smooth a uniform aliasing (horizontal and vertical) caused by a bad digital conversion. It preserves pixel-sized details correctly. Of course you can upscale, DAA() and turn, but it involves twice more deinterlacing operations.

[Bloax]

You sure are right, but for Ye Olde Antialiasing (edge smoothening), the former is much better.

Though this:
PointResize(Width*2,Height*2)
SangNom(0,255).TurnLeft().SangNom(0,255).TurnRight()
Merge(SangNom(0,255),SangNom(1,255),0.5)

Looks pretty funky. (And the latter is indeed useful for 2x upscaled material.)
Though for antialiasing, it causes even worse jaggies, as it discards an entire field, and thus the existing jaggies get much wider. (And thus noticeable.)

[Didée]

@cretindesalpes - Yepp, just that the 2*upscaling part was actually missing in the scripts of this thread. The used deinterlacer throws away half of the pixels, although they (should) contain "good" data in game footage.

How many AA filters are available for Avisynth - a dozen? Two or three dozens if you count the "variants" of them? All code that makes sense is already out there (the LSharpAAF collection (forum.doom9.org/showthread.php?t=153835) comes to mind). You only need to realize what your source is about, then choose an appropriate method.

Presuming that all pixels are valid (no interlacing or former de-interlacing) in game footage, a theoretically-sufficient filter would be

NNEDI3_rpow2(2,cshift="bicubicresize",fwidth=width,fheight=height)

A bit problematic could be if there are very "hard" or very "high contrast" contours - in such areas, NNEDI3 every now&then doesn't want to connect. (Other deinterlacers may show a similar problem.)
In this case, processing twice can help, maybe with a blur-bypass in between.

[Bloax]

I have messed around with NNEDI3, but honestly, it's just too slow to be useful.
Antialiasing is supposed to be add-on filtering, not the heavyweight script that sets the FPS to <20

Talking about that, this seems to provide the highest quality:
Merge(SangNom(0,255),SangNom(1,255),0.5)
While this one is slightly faster (~25% on the source I test on):
Merge(SangNom(0,255),Yadif(2,1),0.5)
But provides slightly less smooth results. (Which may be preferable, actually.)

[Didée]

You surely didn't mean to say "the best", because really, it isn't.

As said, the "general" antialiasing methods are out there already.

For your specific case of game footage, advice is not possible without having seen something. In this thread you're randomly mixing one-axis processing with two-axis-processing (horizontal-only versus horizontal+vertical) - is the game using some asymetric AA from the start? It's not even clear what type of game footage it is - some modern FPS shooter, or Mario or Donkey Kong on an emulator? It's all different footage, with different footage having different problems, thus requiring different solutions.

Discussing in the blue, then saying "this script da best" is a bit ... nebulous.

[Bloax]

I'm not saying that "This is the best", oops.
I actually mean "The best deinterlacer for the job", I tried NNEDI3 but it's inferior to SangNom in that aspect. (Also: Speed.)
YADIF is a pretty nice solution too, and fast. Though a bit too sharp for my tastes, it's still very useful.

Here's the clip (http://www.mediafire.com/?d92ed0cs3ukfdqk), it's kind of a mixed bag (both big and small aliasing), which is why I use it for testing.
I don't exactly have much material to work with, since I don't really have a lot of (modern) games.

[Didée]

avisource("Heroes5comp.avi") #.converttoyv12()
o=last

mode = 1 # 1 = fast, but more frequently errors on certain repetitive textures
# 2 = slower, but more safe for those textures
# 3 = faster than "4" would be. :-p

pointresize(width*mode,height*2).sangnom(0,256).spline16resize(width,height,0,0.5,width*mode,height*2)
turnleft()
pointresize(width*mode,height*2).sangnom(0,256).spline16resize(width,height,0,0.5,width*mode,height*2)
turnright()

#interleave(o,last)

return(last)

That's using sangnom "the appropriate way for this case",
seems to look good,
and surely does already exist as some abcAAxyz() function, whatever the exact name.


Edit - wait, Sangnom did have this ickle texture problem, didn't it?

=> added "mode" parameter.

[Bloax]

That does look great.

Here's what that script gives (http://img59.imageshack.us/img59/6686/slow0r.png) (~30 FPS)
Here's what Merge(SangNom(0,255),SangNom(1,255),0.5) gives (http://img850.imageshack.us/img850/9949/fast0.png) (~60 FPS)
And here's what Merge(Yadif(2,0),SangNom(1,255),0.5) gives (http://img853.imageshack.us/img853/6581/faster0.png) (~84 FPS)

As for that ickle texture problem, it's present in the usual implementation of that kind, but not in yours.
But yeah, it's a matter of speed really. Upscaling and downscaling provides a little boost too, but at a rather big price.