3D Game Anti-Aliasing (FXAA, SMAA, MLAA)

[thecoreyburton]

I've seen a posts about this from previous years, but none with a definitive answer so I thought I'd ask the experts.

When you play a 3D game there are generally two categories of Anti-Aliasing. Traditional Anti-Aliasing, and post-Anti-Aliasing.

Traditional Anti-Aliasing methods include supersampling (SSAA, FSAA) and multi-sampling (MSAA).

Post Anti Aliasing methods include FXAA, SMAA and MLAA.

My understand is that the traditional category is done by the GPU and cannot be changed once the scene is rendered as it relies on the scene or polygons to be rendered at a higher resolution and then shrunk down. The "Post AA" category is different however. It relies on the scene to be rendered and then applies the effect on top of the image as a filter.

Like many others, my card (AMD R9-285) can force different types of AA. If I force MLAA in game, it gives a slightly blurred, but anti-aliased image (as I would expect). If I capture footage from the game with a program such as fraps and then play it back, the effect is not there. Consequently, if I force Media Player Classic to use MLAA, when I play back the footage in it, I get the exact same effect.

My point is, a lot of the time I see this question answered with reference to traditional anti-aliasing and comments regarding the fact you'd need the game itself to do this sort of effect. I however get the same results applying the effect in game as I do applying the effect to a video file recorded without the effect.

Are there any sort of filters or plugins that can achieve this sort of effect? I'd be looking to polish some videos of games that don't support any sort of anti-aliasing and figured here was the place to ask!

Thank you for your time, and sorry about the big post! :D

[feisty2]

Upscale with eedi3 (sclip=nnedi3()) and then scale it back
Repeat it for a few times

[thecoreyburton]

Thank you for your fast reply. It works, although the results aren't too nice. It takes quite a few times before the edges see any sort of effect and by then the image is quite distorted. The trade off for distortion vs. visible aliased edges of polygons remaining is almost not worth it.

[vivan]

http://avisynth.nl/index.php/External_filters#Anti-aliasing
Try daa, it's based on this approach but is a bit more sophisticated. There're also few other filters to try.

[raffriff42]

I love QTGMC (http://avisynth.nl/index.php/QTGMC)for AA on game footage - it does some kind of temporal filtering, in addition to normal AA, to soften the twinkling/shimmering that I find annoying and hard to compress. There is an "EEDI3+NNEDI3" mode, although with my material I don't notice an improvement./* PLUGIN REQUIREMENTS (YMMV)
## mvtools2.dll == AutoLoad, in Plugins folder
## libflowsse.dll == AutoLoad, in Plugins folder
LoadPlugin("mt_masktools-26.dll")
LoadPlugin("RemoveGrainSSE2.dll")
LoadPlugin("nnedi3.dll")
LoadPlugin("eedi3.dll")
LoadPlugin("FFT3DFilter.dll")
*/
#QTGMC(preset="Very Slow", FPSDivisor=2, EdiMode="EEDI3+NNEDI3")
QTGMC(preset="Slower", FPSDivisor=2) ## good enough

[thecoreyburton]

Here is a screenshot I took playing Mario Party through an emulator. It was the quickest decent example I could find. I used ImageSource to load it up from a png file and apply the anti aliasing. The outcome is similar, although with less noticeable distortion and a sharper look within the characters, though the outsides are noticeably aliased still. Daa was definitely faster.

Original:
http://i57.tinypic.com/w7mzqg.png
Daa, five times:
http://i58.tinypic.com/2cs8kmu.png
EEDI3, five times:
http://i62.tinypic.com/wb4cpu.png

Hopefully these screenshots will help a bit too.

Edit: I only saw the QTGMC post after this. I just tried it and found that the results provided the least distortion but also the least correction in relation to the above methods. It's definitely the best visually so far.

[feisty2]

u sure u did it correctly?

http://i.imgur.com/ay7WXnU.png


ImageSource("w7mzqg.png")

interleave (showred ("yv12"),showgreen ("yv12"),showblue("yv12"))
eedi3 (dh=true,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (dh=true, nsize=3, nns=4, qual=2, etype=1))
turnleft ()
eedi3 (dh=true,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (dh=true, nsize=3, nns=4, qual=2, etype=1))
turnright ()
bicubicresize (640, 240, b=-1, c=0)

eedi3 (dh=true,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (dh=true, nsize=3, nns=4, qual=2, etype=1))
turnleft ()
eedi3 (dh=true,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (dh=true, nsize=3, nns=4, qual=2, etype=1))
turnright ()
bicubicresize (640, 240, b=-1, c=0)

mergergb (selectevery (3,0),selectevery (3,1),selectevery (3,2))

[thecoreyburton]

u sure u did it correctly?

I thought I did, but I guess not! I didn't do any rotation or interleaving so that's probably where I went wrong. That result looks beautiful.

[feisty2]

I thought I did, but I guess not! I didn't do any rotation or interleaving so that's probably where I went wrong. That result looks beautiful.

yeah, I said "upscale it", not "deinterlace it"

[wonkey_monkey]

FXAA sounds pretty simple to implement, but I can't quite get a full grasp of it from the documents I've found. Is there any interest in it, or is multi-NNEDI3 the preferred (if presumably slower) approach?

[feisty2]

Is there any interest in it

Quality is the one and only thing I care about, if the quality is lower than the script at #7, then no, for me

[Desbreko]

feisty2's script is a good method of antialiasing, provided you don't mind the slow speed of eedi3, but the center shift should be corrected. It can also be sped up by not processing the unused chroma channels, and using FTurn instead of the internal turn functions will speed it up some more unless you're using AviSynth+.

ImageSource("w7mzqg.png")

interleave (showred ("yv12"),showgreen ("yv12"),showblue("yv12"))
eedi3 (field=1,dh=true,U=false,V=false,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (field=1, dh=true, U=false, V=false, nsize=3, nns=4, qual=2, etype=1))
turnleft ()
eedi3 (field=1,dh=true,U=false,V=false,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (field=1, dh=true, U=false, V=false, nsize=3, nns=4, qual=2, etype=1))
turnright ()
bicubicresize (640, 240, b=-1, c=0, src_left=0.5, src_top=-0.5)

eedi3 (field=1,dh=true,U=false,V=false,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (field=1, dh=true, U=false, V=false, nsize=3, nns=4, qual=2, etype=1))
turnleft ()
eedi3 (field=1,dh=true,U=false,V=false,alpha=1,beta=0,gamma=0,nrad=3,mdis=40,sclip=nnedi3 (field=1, dh=true, U=false, V=false, nsize=3, nns=4, qual=2, etype=1))
turnright ()
bicubicresize (640, 240, b=-1, c=0, src_left=0.5, src_top=-0.5)

mergergb (selectevery (3,0),selectevery (3,1),selectevery (3,2))

[MysteryX]

mergergb (selectevery (3,0),selectevery (3,1),selectevery (3,2))
I was wondering what this line was about.

Temporal anti-aliasing. Smart.

[feisty2]

I was wondering what this line was about.

Temporal anti-aliasing. Smart.

Sadly, it's 100% spatial

[pbristow]

FXAA sounds pretty simple to implement, but I can't quite get a full grasp of it from the documents I've found. Is there any interest in it, or is multi-NNEDI3 the preferred (if presumably slower) approach?

The key to it seems to be the depth-aware edge detection. How do you do that, if you don't have a 3D model to get the depth info from? And is the technique even relevant, if you don't have a distinction between edges that need de-aliasing (because of how they were created in the game) and ones that don't (because they're from stored textures)?

If your source is a 2D image, then you've either got to try to infer depth from features of the image (lots of processing, and probably very error-prone); or else just process all detected edges equally, which means you're not really doing FXAA.

[wonkey_monkey]

The key to it seems to be the depth-aware edge detection.

FXAA isn't depth-aware.

[pbristow]

FXAA isn't depth-aware.

Hmmm...? [GOES GOOGLING]

Ah! It seems I have been led astray by a Wikipedia entry ( https://en.wikipedia.org/wiki/Fast_approximate_anti-aliasing ). The original nVidia App Note ( http://developer.download.nvidia.com/assets/gamedev/files/sdk/11/FXAA_WhitePaper.pdf ) doesn't mentioning anything about depth awareness, as you say, so I wonder what the Wikipedia author was on about. Time to submit a correction, methinks...

[EDITED TO ADD:] I've put something in the talk section about it. Didn't want to edit the article directly, as I'm not at all sure now what it *should* say, and don't really have time to write a new one from scratch... =:o\

[raffriff42]

@feisty2 (Post #7) & Desbreko (Post #12)
Beautiful results, but my puny i3 spins for ~15 seconds per frame @ 1080p!
I had to look for something based on your ideas, but faster (this one is about 15x faster for me)

(EDIT now with RGB support)
https://www.dropbox.com/s/kurutluhsomng4q/pow2Qt-02-RGB.png?raw=1

#Import("QTGMC-3.32.avsi")

ImageSource("E:\Data\Downloads\w7mzqg.png") ## Post #6, "Original"
#ConvertToYV12(matrix="Rec601", chromaresample="spline16")

return pow2Qt

##################################
### Antialiasing (AA) for game footage (YV12 only)
### (blending nnedi3_rpow2 & QTGMC)
##
## @ sharpen - post-process Sharpen to mitigate QTGMC smoothing
## (on double-size image, so apparent effect is 1/2)
## (0.2-0.3 best in initial tests; default 0.3)
## @ cpoint - upscale pass2 chroma with PointResize (sharper)
## @ doublesize - 2x enlargement, but AA less effective
##
## v0.42.02 - support RGB32, RGB24, Y8 (and YV12 as before)
##
function pow2Qt(clip C, float "sharpen", bool "doublesize", bool "cpoint")
{
Assert(C.IsRGB || C.IsYV12 || C.IsY8,
\ "pow2Qt: source must be RGB, YV12 or Y8")

shp = Default(sharpen, 0.0)
dbl = Default(doublesize, false)
cpt = Default(cpoint, false)

return
\ (C.IsYV12) ? C._pow2QtYV12(sharpen, doublesize, cpoint)
\ : (C.IsY8) ? C._pow2QtY8 (sharpen, doublesize)
\ : C._pow2QtRGB (sharpen, doublesize)
## Y8 : 25 fps
## YV12 : 13 fps
## RGB24: 8 fps
## RGB32: 7 fps
}
##################################
function _pow2QtYV12(clip C, float "sharpen", bool "doublesize", bool "cpoint")
{
Assert(C.IsYV12, "_pow2QtYV12: source must be YV12")

shp = Default(sharpen, 0.3)
cpt = Default(cpoint, false)
dbl = Default(doublesize, false)

C
wid = Width
hgt = Height

## pass 1: nnedi3 double size, scale back

Y = nnedi3_rpow2(rfactor=2, nsize=0, nns=3, qual=2,
\ cshift="Spline64Resize")
UV = (cpt)
\ ? PointResize(2*wid, 2*hgt, src_top=0.25)
\ : Spline64Resize(2*wid, 2*hgt, src_left=+0.5, src_top=+0.5)
YToUV(UV.UToY, UV.VToY, Y.ConvertToY8)

BicubicResize(wid, hgt, b=-1, c=0, src_left=-0.5, src_top=+0.5)

## pass 2: nnedi3 double size, QTGMC de-shimmer, sharpen, scale back

Y = nnedi3_rpow2(rfactor=2, nsize=0, nns=3, qual=2,
\ cshift="Spline64Resize")
UV = (cpt)
\ ? PointResize(2*wid, 2*hgt, src_top=0.25)
\ : Spline64Resize(2*wid, 2*hgt, src_left=-0.5, src_top=-0.5)
YToUV(UV.UToY, UV.VToY, Y.ConvertToY8)

QTGMC(preset="Medium", InputType=1)
Sharpen(shp)

return (dbl)
\ ? Last
\ : BicubicResize(wid, hgt, b=-1, c=0, src_left=0.5, src_top=-0.5)
}
##################################
function _pow2QtY8(clip C, float "sharpen", bool "doublesize")
{
Assert(C.IsY8, "_pow2QtY8: source must be Y8")

shp = Default(sharpen, 0.3)
dbl = Default(doublesize, false)

C
wid = Width
hgt = Height

## pass 1: nnedi3 double size, scale back

ConvertToYV12
### {
nnedi3_rpow2(rfactor=2, nsize=0, nns=3, qual=2,
\ cshift="Spline64Resize")

BicubicResize(wid, hgt, b=-1, c=0) #, src_left=-0.5, src_top=+0.5)

## pass 2: nnedi3 double size, QTGMC de-shimmer, sharpen, scale back

nnedi3_rpow2(rfactor=2, nsize=0, nns=3, qual=2,
\ cshift="Spline64Resize")

QTGMC(preset="Medium", InputType=1, RepChroma=false)
### }
ConvertToY8
Sharpen(shp)

return (dbl)
\ ? Last
\ : BicubicResize(wid, hgt, b=-1, c=0) #, src_left=0.5, src_top=-0.5)
}
##################################
function _pow2QtRGB(clip C, float "sharpen", bool "doublesize")
{
Assert(C.IsRGB, "_pow2QtRGB: source must be RGB")

shp = Default(sharpen, 0.3)
dbl = Default(doublesize, false)

return (C.IsRGB32)
\ ? MergeARGB(
\ C.ShowAlpha("Y8")._pow2QtY8(sharpen, doublesize),
\ C.ShowRed ("Y8")._pow2QtY8(sharpen, doublesize),
\ C.ShowGreen("Y8")._pow2QtY8(sharpen, doublesize),
\ C.ShowBlue ("Y8")._pow2QtY8(sharpen, doublesize)
\ )
\ : MergeRGB(
\ C.ShowRed ("Y8")._pow2QtY8(sharpen, doublesize),
\ C.ShowGreen("Y8")._pow2QtY8(sharpen, doublesize),
\ C.ShowBlue ("Y8")._pow2QtY8(sharpen, doublesize),
\ pixel_type="RGB24"
\ )
}

[feisty2]

Cretindesalpes' eedi3 works pretty fast, give it a try

[Bloax]

function AA_SangNom2(clip in, int "aggression", float "sharpness")
{
sharp = Default(sharpness, 0.25)
str = Default(aggression, 48)
a = in
Upright = Merge(Sangnom2(in,0,str),Sangnom2(in,1,str))
Left = in.FTurnLeft()
Rotated = Merge(Sangnom2(left,0,str),Sangnom2(left,1,str)).FTurnRight()
Merge(Upright,Rotated)
return Merge(a,sharp)
}
If you want it quick then sangnom2 or yadif is the way to go, but of course - they're completely spatial.
ImageSource("a.png")
ConvertToYV24()
AA_Sangnom2(64,0.1)

https://dl.dropboxusercontent.com/u/63152810/ayy0.png

Even faster would be only applying sangnom2 to the Y plane, and I know for a fact that this is not the first example of sangnom being used for exactly this purpose.

[thecoreyburton]

Thank you all for the help and ideas so far. My footage is looking amazing!

In some actual footage tests the results have been consistent with the results from the single frame shot we've been using, with @feisty2's method being the slowest (although slightly faster with @Desbreko's tweaks) and the most visually appealing.

Though for the speed / effect trade off sangnom2 does produce decent results. The only drawback I can see is that it smooths some edges better than others (Donkey Kong's back vs Princess Peach's back).

Does anyone know how well these work with footage that has the unsightly "jitter" found in games with fxaa? To describe it better, it's the sort of flickering you get when a thin line of pixels moves slightly, causing very different results. I don't have any example footage at the moment, but I'll go hunting in a few days time if it's needed.

I know there were some temporal methods (like SMAA T2x) that were made to combat this, but I have a feeling the way the first method works might avoid it as the jitter might be a side effect of how fast fxaa works.

[raffriff42]

... it's the sort of flickering you get when a thin line of pixels moves slightly...That's exactly why I retained a call to QTGMC in my new script (http://forum.doom9.org/showthread.php?p=1737201#post1737201).

[chilledinsanity]

FXAA sounds pretty simple to implement, but I can't quite get a full grasp of it from the documents I've found. Is there any interest in it Yes, I've been wanting this for years (http://forum.doom9.org/showthread.php?t=169636).

[Video Nerd]

As I'm looking for such a feature by myself, I have tried and experimented with feisty2's approach.
But as it was deeeead slow, I've looked closely as to how nnedi/eedi interpolates and see if I can optimize it:

I found that nnedi3 is better than eedi3 when it comes to building a gradient between the edges in the lines above and below.

So we have what we want: A nice gradient in the new lines. This is what we want to achieve!
The old lines still have this very abrupt ending. This is what we want to get rid off!

Earlier attempts always tried to mix/merge both lines with bicubic or spline resizing, but the result is rather unsatisfying.
As I already pointed it out just before: DROP THEM !!!

You can do it this way:
nnedi3 (field=1,dh=true, nsize = 3, pscrn = 4)
separatefields().SelectEven()
turnleft()

nnedi3 (field=1,dh=true, nsize = 3, pscrn = 4)
separatefields().SelectEven()
turnright()


It's recommended to tune nnedi to the max with nsize=3 and pscrn=4 to avoid step artefacts in very low slopes

or simplified:

w=width()
h=height()

nnedi3 (field=1,dh=true, nsize = 3, pscrn = 4)
turnleft()
nnedi3 (field=1,dh=true, nsize = 3, pscrn = 4)
turnright()

pointresize (w, h, src_left=0, src_top=0)

If you'd first turn right and then left, you'd need to crop by one pixel from left (i think).
Edit: Above is tested to be correct.

or condensed into this:


w=width()
h=height()

nnedi3_rpow2 (rfactor=2, nsize=3, pscrn = 4, cshift="pointresize", fwidth=w,fheight=h)

(cshift commands to shifts half a pixel, put as pointresize can only accept whole numbers it does no shifting at all)

Result: old hard edges are gone, completely replaced by gradiented edges almost exactly like fxaa,smaa,mlaa etc.
And runs at least a fifty times faster too. Okay, my first test have shown that the speed depends on the amount of antialiasing to be generated, esp. when nnedi3 is turned so high. Still it is apx. 5-10 times faster at worst case in comparison to feisty2's script

[Video Nerd]

Forget to show how the above test image gets processed with my filter approach:

[raffriff42]

Another one. I call it 'FRAANQ,' pronounced like 'Frank.'

To the previous filters' Nnedi3 and QTGMC, FRAANQ adds FrameRateConverter (https://forum.doom9.org/showthread.php?t=174793), which acts not as a framerate converter but as a very-low-artifact interframe interpolator. Interpolation adds a directional quality to seemingly random texture moire, shimmering & twittering. It helps FRAANQ (sometimes) extract plausible detail from apparent chaos. FFT3DFilter (https://forum.doom9.org/showthread.php?t=174347)is added also; with an edge mask, it smooths some stubborn jaggies that the other filters miss -- but may possibly overdo it, so this step is optional.

[Still shots don't really show it - click for video.] (https://www.youtube.com/watch?v=C_qOBBO3j_c)
The video shows a collection of clips with a variety of aliasing problems and what FRAANQ does with them. It doesn't fix all aliasing problems, only some; but neither does it seem to add new problems.

Here's the code. There's nothing to the script itself, the magic is done by the components mentioned. It could use more performance tuning though! ##################################
### smooth certain types of aliasing artifacts (slow!)
### with [[FrameRateConverter]] + [[Nnedi3]] + [[QTGMC]] + [[FFT3DFilter]]
##
## @ smoother - if true (default), add extra FFT3DFilter edge-filtering
## @ faster - if true, use faster, lower quality settings (default false)
## @ motionblur - if true, add slight motion blur (default false)
##
## @ version 0.42.01 28-Sep-2017
## @ version 0.42.04 add FFT3DFilter
##
function FRAANQ(clip C, bool "smoother", bool "faster", bool "motionblur")
{
Assert(C.IsYV24 || C.IsYV12,
\ "FRAANQ: source must be YV24 or YV12")
smoo = Default(smoother, true)
fast = Default(faster, false)
blur = Default(motionblur, false)

C
## FRC presets: (slower|slow|normal|fast)
FrameRateConverter(Preset=(fast ? "fast" : "normal"))
SelectOdd

Y = nnedi3_rpow2(2, nsize=0, nns=4, qual=2,
\ cshift="Spline64Resize",
\ fwidth=C.Width, fheight=C.Height)
UV = Spline64Resize(C.Width, C.Height, src_left=-0.5, src_top=-0.5)
YToUV(UV.UToY, UV.VToY, Y)

## QTGMC presets: (slower|slow|medium|fast|faster)
QTGMC(
\ InputType = 1,
\ Preset = (fast ? "fast" : "medium"),
\ ShutterBlur = (blur ? 3 : 0),
\ ShutterAngleSrc = (blur ? 120 : 180),
\ ShutterAngleOut = (blur ? 270 : 180)
\ )

S = FFT3DFilter(sigma=20, sigma4=20, bw=24, bh=24,
\ bt=3, ow=8, oh=8, sharpen=0.5, ncpu=2)
E = mt_edge("laplace", thY1=16, thY2=16, chroma="-128")
\ .Levels(0, 1.0, 128, 0, 255)
\ .TemporalSoften(2, 255, 255, 255, 2)

return (smoo) ? mt_merge(Last, S, E) : Last
}