3D Game Anti-Aliasing (FXAA, SMAA, MLAA)

I've seen a posts about this from previous years, but none with a definitive answer so I thought I'd ask the experts.

When you play a 3D game there are generally two categories of Anti-Aliasing. Traditional Anti-Aliasing, and post-Anti-Aliasing.

Traditional Anti-Aliasing methods include supersampling (SSAA, FSAA) and multi-sampling (MSAA).

Post Anti Aliasing methods include FXAA, SMAA and MLAA.

My understand is that the traditional category is done by the GPU and cannot be changed once the scene is rendered as it relies on the scene or polygons to be rendered at a higher resolution and then shrunk down. The "Post AA" category is different however. It relies on the scene to be rendered and then applies the effect on top of the image as a filter.

Like many others, my card (AMD R9-285) can force different types of AA. If I force MLAA in game, it gives a slightly blurred, but anti-aliased image (as I would expect). If I capture footage from the game with a program such as fraps and then play it back, the effect is not there. Consequently, if I force Media Player Classic to use MLAA, when I play back the footage in it, I get the exact same effect.

My point is, a lot of the time I see this question answered with reference to traditional anti-aliasing and comments regarding the fact you'd need the game itself to do this sort of effect. I however get the same results applying the effect in game as I do applying the effect to a video file recorded without the effect.

Are there any sort of filters or plugins that can achieve this sort of effect? I'd be looking to polish some videos of games that don't support any sort of anti-aliasing and figured here was the place to ask!

Thank you for your time, and sorry about the big post!

[feisty2]

Upscale with eedi3 (sclip=nnedi3()) and then scale it back
Repeat it for a few times

Thank you for your fast reply. It works, although the results aren't too nice. It takes quite a few times before the edges see any sort of effect and by then the image is quite distorted. The trade off for distortion vs. visible aliased edges of polygons remaining is almost not worth it.

[vivan]

http://avisynth.nl/index.php/Externa...#Anti-aliasing
Try daa, it's based on this approach but is a bit more sophisticated. There're also few other filters to try.

[raffriff42]

I love QTGMC for AA on game footage - it does some kind of temporal filtering, in addition to normal AA, to soften the twinkling/shimmering that I find annoying and hard to compress. There is an "EEDI3+NNEDI3" mode, although with my material I don't notice an improvement.

/* PLUGIN REQUIREMENTS (YMMV)
## mvtools2.dll == AutoLoad, in Plugins folder
## libflowsse.dll == AutoLoad, in Plugins folder
LoadPlugin("mt_masktools-26.dll")
LoadPlugin("RemoveGrainSSE2.dll")
LoadPlugin("nnedi3.dll")
LoadPlugin("eedi3.dll")
LoadPlugin("FFT3DFilter.dll")
*/
#QTGMC(preset="Very Slow", FPSDivisor=2, EdiMode="EEDI3+NNEDI3")
QTGMC(preset="Slower", FPSDivisor=2) ## good enough

Here is a screenshot I took playing Mario Party through an emulator. It was the quickest decent example I could find. I used ImageSource to load it up from a png file and apply the anti aliasing. The outcome is similar, although with less noticeable distortion and a sharper look within the characters, though the outsides are noticeably aliased still. Daa was definitely faster.

Original:

Daa, five times:

EEDI3, five times:


Hopefully these screenshots will help a bit too.

Edit: I only saw the QTGMC post after this. I just tried it and found that the results provided the least distortion but also the least correction in relation to the above methods. It's definitely the best visually so far.

[feisty2]

u sure u did it correctly?

ImageSource("w7mzqg.png")

interleave (showred ("yv12"),showgreen ("yv12"),showblue("yv12"))
eedi3 (dh=true,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (dh=true, nsize=3, nns=4, qual=2, etype=1))
turnleft ()
eedi3 (dh=true,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (dh=true, nsize=3, nns=4, qual=2, etype=1))
turnright ()
bicubicresize (640, 240, b=-1, c=0)

eedi3 (dh=true,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (dh=true, nsize=3, nns=4, qual=2, etype=1))
turnleft ()
eedi3 (dh=true,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (dh=true, nsize=3, nns=4, qual=2, etype=1))
turnright ()
bicubicresize (640, 240, b=-1, c=0)

mergergb (selectevery (3,0),selectevery (3,1),selectevery (3,2))

Originally Posted by feisty2

u sure u did it correctly?

I thought I did, but I guess not! I didn't do any rotation or interleaving so that's probably where I went wrong. That result looks beautiful.

[feisty2]

Originally Posted by thecoreyburton

I thought I did, but I guess not! I didn't do any rotation or interleaving so that's probably where I went wrong. That result looks beautiful.

yeah, I said "upscale it", not "deinterlace it"

[wonkey_monkey]

FXAA sounds pretty simple to implement, but I can't quite get a full grasp of it from the documents I've found. Is there any interest in it, or is multi-NNEDI3 the preferred (if presumably slower) approach?

[feisty2]

Originally Posted by davidhorman

Is there any interest in it

Quality is the one and only thing I care about, if the quality is lower than the script at #7, then no, for me

[Desbreko]

feisty2's script is a good method of antialiasing, provided you don't mind the slow speed of eedi3, but the center shift should be corrected. It can also be sped up by not processing the unused chroma channels, and using FTurn instead of the internal turn functions will speed it up some more unless you're using AviSynth+.

ImageSource("w7mzqg.png")

interleave (showred ("yv12"),showgreen ("yv12"),showblue("yv12"))
eedi3 (field=1,dh=true,U=false,V=false,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (field=1, dh=true, U=false, V=false, nsize=3, nns=4, qual=2, etype=1))
turnleft ()
eedi3 (field=1,dh=true,U=false,V=false,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (field=1, dh=true, U=false, V=false, nsize=3, nns=4, qual=2, etype=1))
turnright ()
bicubicresize (640, 240, b=-1, c=0, src_left=0.5, src_top=-0.5)

eedi3 (field=1,dh=true,U=false,V=false,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (field=1, dh=true, U=false, V=false, nsize=3, nns=4, qual=2, etype=1))
turnleft ()
eedi3 (field=1,dh=true,U=false,V=false,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (field=1, dh=true, U=false, V=false, nsize=3, nns=4, qual=2, etype=1))
turnright ()
bicubicresize (640, 240, b=-1, c=0, src_left=0.5, src_top=-0.5)

mergergb (selectevery (3,0),selectevery (3,1),selectevery (3,2))

[MysteryX]

Originally Posted by feisty2

mergergb (selectevery (3,0),selectevery (3,1),selectevery (3,2))

I was wondering what this line was about.

Temporal anti-aliasing. Smart.

[feisty2]

Originally Posted by MysteryX

I was wondering what this line was about.

Temporal anti-aliasing. Smart.

Sadly, it's 100% spatial

[pbristow]

Originally Posted by davidhorman

FXAA sounds pretty simple to implement, but I can't quite get a full grasp of it from the documents I've found. Is there any interest in it, or is multi-NNEDI3 the preferred (if presumably slower) approach?

The key to it seems to be the depth-aware edge detection. How do you do that, if you don't have a 3D model to get the depth info from? And is the technique even relevant, if you don't have a distinction between edges that need de-aliasing (because of how they were created in the game) and ones that don't (because they're from stored textures)?

If your source is a 2D image, then you've either got to try to infer depth from features of the image (lots of processing, and probably very error-prone); or else just process all detected edges equally, which means you're not really doing FXAA.

[wonkey_monkey]

The key to it seems to be the depth-aware edge detection.

FXAA isn't depth-aware.

[pbristow]

Originally Posted by davidhorman

FXAA isn't depth-aware.

Hmmm...? [GOES GOOGLING]

Ah! It seems I have been led astray by a Wikipedia entry ( https://en.wikipedia.org/wiki/Fast_a..._anti-aliasing ). The original nVidia App Note ( http://developer.download.nvidia.com...WhitePaper.pdf ) doesn't mentioning anything about depth awareness, as you say, so I wonder what the Wikipedia author was on about. Time to submit a correction, methinks...

[EDITED TO ADD:] I've put something in the talk section about it. Didn't want to edit the article directly, as I'm not at all sure now what it *should* say, and don't really have time to write a new one from scratch... =:o\

[raffriff42]

@feisty2 (Post #7) & Desbreko (Post #12)
Beautiful results, but my puny i3 spins for ~15 seconds per frame @ 1080p!
I had to look for something based on your ideas, but faster (this one is about 15x faster for me)

(EDIT now with RGB support)

#Import("QTGMC-3.32.avsi")

ImageSource("E:\Data\Downloads\w7mzqg.png") ## Post #6, "Original"
#ConvertToYV12(matrix="Rec601", chromaresample="spline16") 

return pow2Qt

##################################
### Antialiasing (AA) for game footage (YV12 only)
### (blending nnedi3_rpow2 & QTGMC) 
##
## @ sharpen - post-process Sharpen to mitigate QTGMC smoothing
##             (on double-size image, so apparent effect is 1/2)
##             (0.2-0.3 best in initial tests; default 0.3)
## @ cpoint  - upscale pass2 chroma with PointResize (sharper)
## @ doublesize - 2x enlargement, but AA less effective
##
## v0.42.02 - support RGB32, RGB24, Y8 (and YV12 as before)
##
function pow2Qt(clip C, float "sharpen", bool "doublesize", bool "cpoint")
{
    Assert(C.IsRGB || C.IsYV12 || C.IsY8, 
    \   "pow2Qt: source must be RGB, YV12 or Y8")

    shp = Default(sharpen, 0.0)
    dbl = Default(doublesize, false)
    cpt = Default(cpoint, false)

    return 
    \   (C.IsYV12) ? C._pow2QtYV12(sharpen, doublesize, cpoint)
    \ : (C.IsY8)   ? C._pow2QtY8  (sharpen, doublesize)
    \ :              C._pow2QtRGB (sharpen, doublesize)
    ## Y8   : 25 fps
    ## YV12 : 13 fps
    ## RGB24:  8 fps
    ## RGB32:  7 fps
}
##################################
function _pow2QtYV12(clip C, float "sharpen", bool "doublesize", bool "cpoint")
{
    Assert(C.IsYV12, "_pow2QtYV12: source must be YV12")

    shp = Default(sharpen, 0.3)
    cpt = Default(cpoint, false)
    dbl = Default(doublesize, false)

    C
    wid = Width
    hgt = Height

    ## pass 1: nnedi3 double size, scale back

    Y  = nnedi3_rpow2(rfactor=2, nsize=0, nns=3, qual=2, 
    \                 cshift="Spline64Resize")    
    UV = (cpt) 
    \  ? PointResize(2*wid, 2*hgt, src_top=0.25)
    \  : Spline64Resize(2*wid, 2*hgt, src_left=+0.5, src_top=+0.5)
    YToUV(UV.UToY, UV.VToY, Y.ConvertToY8)

    BicubicResize(wid, hgt, b=-1, c=0, src_left=-0.5, src_top=+0.5)

    ## pass 2: nnedi3 double size, QTGMC de-shimmer, sharpen, scale back

    Y = nnedi3_rpow2(rfactor=2, nsize=0, nns=3, qual=2, 
    \                 cshift="Spline64Resize") 
    UV = (cpt) 
    \  ? PointResize(2*wid, 2*hgt, src_top=0.25)
    \  : Spline64Resize(2*wid, 2*hgt, src_left=-0.5, src_top=-0.5)
    YToUV(UV.UToY, UV.VToY, Y.ConvertToY8)
    
    QTGMC(preset="Medium", InputType=1)
    Sharpen(shp)

    return (dbl) 
    \  ? Last 
    \  : BicubicResize(wid, hgt, b=-1, c=0, src_left=0.5, src_top=-0.5)
}
##################################
function _pow2QtY8(clip C, float "sharpen", bool "doublesize")
{
    Assert(C.IsY8, "_pow2QtY8: source must be Y8")

    shp = Default(sharpen, 0.3)
    dbl = Default(doublesize, false)

    C
    wid = Width
    hgt = Height

    ## pass 1: nnedi3 double size, scale back 

    ConvertToYV12
    ### {
        nnedi3_rpow2(rfactor=2, nsize=0, nns=3, qual=2, 
        \            cshift="Spline64Resize")    

        BicubicResize(wid, hgt, b=-1, c=0) #, src_left=-0.5, src_top=+0.5)

        ## pass 2: nnedi3 double size, QTGMC de-shimmer, sharpen, scale back

        nnedi3_rpow2(rfactor=2, nsize=0, nns=3, qual=2, 
        \             cshift="Spline64Resize") 

        QTGMC(preset="Medium", InputType=1, RepChroma=false)
    ### }
    ConvertToY8
    Sharpen(shp)

    return (dbl) 
    \ ? Last 
    \ : BicubicResize(wid, hgt, b=-1, c=0) #, src_left=0.5, src_top=-0.5)
}
##################################
function _pow2QtRGB(clip C, float "sharpen", bool "doublesize")
{
    Assert(C.IsRGB, "_pow2QtRGB: source must be RGB")

    shp = Default(sharpen, 0.3)
    dbl = Default(doublesize, false)

    return (C.IsRGB32) 
    \ ? MergeARGB(
    \       C.ShowAlpha("Y8")._pow2QtY8(sharpen, doublesize),
    \       C.ShowRed  ("Y8")._pow2QtY8(sharpen, doublesize),
    \       C.ShowGreen("Y8")._pow2QtY8(sharpen, doublesize),
    \       C.ShowBlue ("Y8")._pow2QtY8(sharpen, doublesize)
    \   )
    \ : MergeRGB(
    \       C.ShowRed  ("Y8")._pow2QtY8(sharpen, doublesize),
    \       C.ShowGreen("Y8")._pow2QtY8(sharpen, doublesize),
    \       C.ShowBlue ("Y8")._pow2QtY8(sharpen, doublesize),
    \       pixel_type="RGB24"
    \   )
}

[feisty2]

Cretindesalpes' eedi3 works pretty fast, give it a try

[Bloax]

function AA_SangNom2(clip in, int "aggression", float "sharpness")
{
    sharp = Default(sharpness, 0.25)
    str = Default(aggression, 48)
    a = in
    Upright = Merge(Sangnom2(in,0,str),Sangnom2(in,1,str))
    Left = in.FTurnLeft()
    Rotated = Merge(Sangnom2(left,0,str),Sangnom2(left,1,str)).FTurnRight()
    Merge(Upright,Rotated)
    return Merge(a,sharp)
}

If you want it quick then sangnom2 or yadif is the way to go, but of course - they're completely spatial.

ImageSource("a.png")
ConvertToYV24()
AA_Sangnom2(64,0.1)

Even faster would be only applying sangnom2 to the Y plane, and I know for a fact that this is not the first example of sangnom being used for exactly this purpose.