A mini codec shootout: animation encoding

[Dark Shikari]

Encoder comparisons are a dime a dozen these days, but there’s one thing I’ve almost never seen tested: cartoon sources. Animated material has totally different characteristics from film and presents a whole separate set of encoding challenges.

First, we’ll start with what makes such video easy to compress. Animation is mostly static; backgrounds are completely static with characters placed in front of them. The characters themselves are mostly static as well; modern animation is usually at a significantly lower framerate than the actual video. Furthermore, characters may stand still with only their mouths moving, dramatically reducing complexity. Finally, animation is usually very clean, without any film grain. All of this combines to seemingly make animation compression a very simple task.

But don’t let the above reasons fool you...

Read more (http://x264dev.multimedia.cx/?p=102)

(OK, so ~17 encoders is not exactly small.)

[Selur]

Nice to see how the codecs behave at 250kBit/s even it that bitrate is far to low for my preferences, even for web content I prefer at leas 512kBit/s (but to be frank I normally don't encode cartoon content). ;)

I was surprised to see:
- snow so far to the top (would have expected worse of snow, but happy to see it where it is ;))
- x264 fastest behind Xvid (thought that x264 would always outperform Xvid ;))
- Badaboom even behind Xvid
- VP7 so far to the top (but that would probably change on higher bitrates)
- how bad wmv performed

all in all a nice test, thanks for taking the time to post it.

Cu Selur

Ps.: Would be nice to see how these codecs compare for 1024x768 windows/linux screen captures.

[Dark Shikari]

- x264 fastest behind Xvid (thought that x264 would always outperform Xvid ;))I'm actually surprised it was as high as it was; ultrafast disables all subpel among other things... :)

WMV is probably crippled by the fact that it only allows 8x8dct in intra mode.

[iwod]

All of a sudden, the questions pop up in my head, if x264 is free, and apple are already paying license fees for including an CRAPY encoder, why they dont just include an version of x264?

Anyway my though was with Macroblock Tree Ratecontrol, nothing come close to x264. In the case of Anime, it is that good.

[nakTT]

I'm actually surprised it was as high as it was; ultrafast disables all subpel among other things... :)

WMV is probably crippled by the fact that it only allows 8x8dct in intra mode.
So far after looking at video files that I can play, I can see that x264 Medium is awesome (is it because of MBtree?). Can't wait to see SNOW as its is said to be better right?

[LoRd_MuldeR]

I was surprised to see:
- x264 fastest behind Xvid (thought that x264 would always outperform Xvid ;))

x264 should always outperform Xvid - at the same speed!

But if you tune x264 for maximum speed (--preset ultrafast), while Xvid still uses "quality" settings, the result is not surprising at all.

I was surprised to see:
- Badaboom even behind Xvid

I'm not surprised at all. Not after I tested Badaboom once :rolleyes:

I can see that x264 Medium is awesome (is it because of MBtree?). Can't wait to see SNOW as its is said to be better right?

Uhm, look at the graph! In this test Snow is slightly ahead of Xvid, but slightly behind WMV. And all of them are far behind all the (proper) H.264 encoders.

Nothing did perform better than "x264 Medium" in this test, except for x264 itself :p

[nakTT]

I'm not surprised at all. Not after I tested Badaboom once :rolleyes:
I can see that apple implementation is even worst and ffmpeg_mpeg2.mpg is even more worst. Perhaps the lowest quality?

Uhm, look at the graph! In this test Snow is slightly ahead of Xvid, but slightly behind WMV. And all of them are far behind all the (proper) H.264 encoders.

Nothing did perform better than "x264 Medium" in this test, except for x264 itself :p
Ooops, I guess I misinterpret it. Then x264 Medium IS THE BEST of all.

Its a shame that "CS MSU Graphics&Media Lab" done their test in May, not after MBtree been released.

[shon3i]

I'm not surprised at all. Not after I tested Badaboom once Well this is metric test ;), Badaboom visualy for sure has the lead over Xvid in any way

[nm]

Did you watch the whole clip through? At times, Badaboom's output gets totally borked and it takes a moment to get back to normal. Apparently a ratecontrol issue. Although the Xvid encode is blocky, at least it's fairly consistent and therefore more pleasing to watch.

[benwaggoner]

This is a good excuse as any to do some test clips with the Expression Encoder H.264 encoder as well.

Main Profile only in v3 (it's a somewhat souped-up version of the device transcoder from Windows 7 with CABAC, B-frames, and higher complexity modes).

If nothing else, I'm sure the devs will find DS's inevitable scathing teardown illuminating reading :).

[Dark Shikari]

This is a good excuse as any to do some test clips with the Expression Encoder H.264 encoder as well.

Main Profile only in v3 (it's a somewhat souped-up version of the device transcoder from Windows 7 with CABAC, B-frames, and higher complexity modes).

If nothing else, I'm sure the devs will find DS's inevitable scathing teardown illuminating reading :).I don't have it; want to submit an encode for me to SSIM-test? ;)

[RNiK]

Thusnelda (August 7th ffmpeg2theora nightly)
Video format: Theora
Settings: Two-pass (no other quality tweaks are available)
UH? :confused:

Is really that poor configurable Thusnelda?

[ricardo.santos]

Your mistake is that you forgot to add ConvertToYV12() at the end of your script.

i did add that to the avs script after just to see if it was that that was causing the error but it isnt, also i see a lot of people saying the same thing by doing a google search.

thanks anyway, its not a big deal just wanted to reproduce your test here, see the results instead of just reading the results

[thewebchat]

UH? :confused:

Is really that poor configurable Thusnelda?

There used to also be an "optimize" and a "sharpness" setting, but they are now deprecated.

[benwaggoner]

I don't have it; want to submit an encode for me to SSIM-test? ;)
Sure. What's the best way to get it to you?

I guess it's a tiny file @250 Kbps. I'll try to stick up a .mp4 and .wmv up on SkyDrive later.

This test may have sussed out a little bug with the SAD/Hadamard motion match setting...

[Dark Shikari]

Sure. What's the best way to get it to you?

I guess it's a tiny file @250 Kbps. I'll try to stick up a .mp4 and .wmv up on SkyDrive later.Mediafire is fine...

[IgorC]

Interesting. Thank you for test, DS.

Any chance to see Divx encoders? MC H.264 encoder?
I can't get any decent results from demo version of last MC encoder. I don't know what kind of special tuned MC encoder was in last MSU. It's unavailable.
And what decoders and its settings (of decoders) did you use for Xvid and ffmpeg MPEG4 ASP?

x264 Baseline is better than any other H.264 encoder. Great.

[Dark Shikari]

Interesting. Thank you for test, DS.

Any chance to see Divx encoders? MC H.264 encoder?
I can't get any decent results from demo version of last MC encoder. I don't know what kind of special tuned MC encoder was in last MSU. It's unavailable.
And what decoders and its settings (of decoders) did you use for Xvid and ffmpeg MPEG4 ASP?

x264 Baseline is better than any other H.264 encoder. Great.I used ffvideosource() for all decoding (at least when possible).

DivX encoder, due to a bug, doesn't work with keyframe intervals over 96 yet, so I can't test it fairly.

I don't have a full version of a recent Mainconcept encoder.

[benwaggoner]

Mediafire is fine...
Well, I used SkyDrive :).

This includes the .mp4, the Settings file I use, and a couple .wmv variants. Well, nominal variants at least; it appears that the SAD/Hadamard motion match method selection isn't working in this case, as both modes produce identical output in identical time.

No idea if this is related to the ASP>VC-1 issue, but I've passed the info on to the Expression Encoder team.

Anyway, the H.264 looks pretty decent for Main Profile. Frame type selection is primative to the point of theoretical in this build, but it's got single-slice CABAC, multiple reference frames, and a promising new RD mode. Should beat QuickTime by a good margin at least.

I'm teaching my compression class at Stanford this week, and everyone in the class that's doing any MP4/MOV/F4V encoding is using Apple's H.264 encoder. So I'm adding your chart to my PowerPoint :).

I spend 40 hours doing an intensive seminar on hands-on video compression, and sometimes it seems like 80% of the improvement I help them with is just from:

Deinterlace or inverse telecine if needed
Crop letterboxing and blanking
Correct for your aspect ratio
Don't use any delivery codec from Apple


Apple's professional codec team has done fine work for years, but it's remarkable that in nearly two decades there's never been a single competitive video codec for delivery codec from Apple, from Road Pizza ("Apple Video" - 'rpza') to their MPEG-2, MPEG-4 Part 2, and H.264 implementations.

[Dark Shikari]

Well, I used SkyDrive :).Did someone forget the link? ;)

[benwaggoner]

Did someone forget the link? ;)

Too much talking and typing, not enough sleeping :)...

http://cid-bee3c9ac9541c85b.skydrive.live.com/browse.aspx/.Public/Maikaze

[Dark Shikari]

Too much talking and typing, not enough sleeping :)...

http://cid-bee3c9ac9541c85b.skydrive.live.com/browse.aspx/.Public/MaikazeThis is the first encoder I've seen other than the reference and x264 that uses sub-8x8 partitions :p

And even in B-frames; not even x264 bothers with that.

Edit: er... your encode has dropped frames, and thus gets out of sync, netting you an SSIM of 0.65303.

[benwaggoner]

This is the first encoder I've seen other than the reference and x264 that uses sub-8x8 partitions :p

And even in B-frames; not even x264 bothers with that.
Well, it's not on by default :).

This is kind of an odd encoder, being basically the Win7 device transcoder plus some extra modes the developer added to try some other scenarios. The library included eve more esoteric stuff that didn't get into the interface, but 4x4 seemed interesting enough I advocated for keeping it in. Attaching a screen shot if you're curious about its parameters.

I haven't done enough production work with it myself to stack rank it against other H.264 implementations; I'm quite curious to find out where its score lands.

..and how my WMV ranks as well. Visually the I-frame DQuant helped quite a bit in places.

[benwaggoner]

Edit: er... your encode has dropped frames, and thus gets out of sync, netting you an SSIM of 0.65303.
Well, that's not good.

Are the dropped frames additional instances of identical frames? This implementation may extend the duration of the prior frame when it gets a frame that doesn't otherwise get encoded, as opposed to inserting an explicit skip frame. That's how VC-1 used to do it (although we've changed the default behavior in Expression Encoder to use skip frames with VC-1 Advanced Profile).

Or is it actually dropping real unique frames(and thus has a quality bug)?

[Dark Shikari]

Well, that's not good.

Are the dropped frames additional instances of identical frames? This implementation may extend the duration of the prior frame when it gets a frame that doesn't otherwise get encoded, as opposed to inserting an explicit skip frame. That's how VC-1 used to do it (although we've changed the default behavior in Expression Encoder to use skip frames with VC-1 Advanced Profile).

Or is it actually dropping real unique frames(and thus has a quality bug)?Elecard Streameye says your output file has 4846 frames. When using FFVideoSource, the frames don't match up with the original decoded frames temporally.

DirectShowSource() with FPS forced to 23.976 works slightly better but still gets out of sync.

[benwaggoner]

Elecard Streameye says your output file has 4846 frames. When using FFVideoSource, the frames don't match up with the original decoded frames temporally.
Do you see any dropped frames visually?

Hmmm. I did use an .avs to convert from your .mkv source to a Lagarith .avi before encoding. I suppose I could have messed something up there with frame rate, although I can't think of how. The WMV was encoded from the same .avi file, so if it also shows the same dropped frame pattern, it's probably operator error.

EEv3 can read the .mkv just fine via ffdshow, but I wanted to make scrubbing faster and make sure that I was getting YV12 source.

[Dark Shikari]

Do you see any dropped frames visually?

Hmmm. I did use an .avs to convert from your .mkv source to a Lagarith .avi before encoding. I suppose I could have messed something up there with frame rate, although I can't think of how. The WMV was encoded from the same .avi file, so if it also shows the same dropped frame pattern, it's probably operator error.

EEv3 can read the .mkv just fine via ffdshow, but I wanted to make scrubbing faster and make sure that I was getting YV12 source.Remember, there's ~5000 frames in the input file; if you have any less, you've dropped some.

I suspect it could be that most of the tools available don't react very well to VFR MP4s like yours, especially when attempting to extract specific frames to compare with an original file.

[benwaggoner]

Remember, there's ~5000 frames in the input file; if you have any less, you've dropped some.
Yeah, but I didn't see anything missing visually, which is why I'm guessing VFR.

Other than analysis tools and broadcast applications, is there a strong reason not to use VFR for web/device scenarios?

I suspect it could be that most of the tools available don't react very well to VFR MP4s like yours, especially when attempting to extract specific frames to compare with an original file.
Likely enough. There's no way in the UI to turn off VFR, so I'm not sure how to help here. Could you decode to lossess using a tool that would insert skip frames correctly?

[Dark Shikari]

Yeah, but I didn't see anything missing visually, which is why I'm guessing VFR.

Other than analysis tools and broadcast applications, is there a strong reason not to use VFR for web/device scenarios?


Likely enough. There's no way in the UI to turn off VFR, so I'm not sure how to help here. Could you decode to lossess using a tool that would insert skip frames correctly?ffmpeg doesn't do it, DirectShowSource doesn't do it, ffvideosource doesn't do it.. :confused:

[benwaggoner]

ffmpeg doesn't do it, DirectShowSource doesn't do it, ffvideosource doesn't do it.. :confused:
AVISynth to VirtualDub to Lagarith?

Since AVI has a fixed frame rate, tools may know how to deal with its "VFR" better than formats where you really can have arbitrary durations for each frame.

[Dark Shikari]

AVISynth to VirtualDub to Lagarith?As I said, I can't get it working in Avisynth, so there's no point in trying to move it from there.

[Dark Shikari]

Using "-vf nailfps" from this hacky patch by Xiphmont (http://web.mit.edu/xiphmont/Public/mplayer-fork-20090813.patch), I was able to get the frames synced up.

And...

...

...

...

Microsoft H.264 ties Ateme (per the original graph). Of course, per the original comparison, this isn't the latest Ateme encoder.

But x264 still trashes it, even in veryfast mode, of course :p

[benwaggoner]

Using "-vf nailfps" from this hacky patch by Xiphmont (http://web.mit.edu/xiphmont/Public/mplayer-fork-20090813.patch), I was able to get the frames synced up.
Cool, thanks for sticking with it.

Microsoft H.264 ties Ateme (per the original graph). Of course, per the original comparison, this isn't the latest Ateme encoder.

But x264 still trashes it, even in veryfast mode, of course :p
That's actually a pretty good result! Ateme's a nice encoder, and certainly has seen a whole lot more refinement than the EEv3 implementation. Plus my sample was Main Profile only.

Get a chance to check my newer WMV?

[roozhou]

@DS
Why do you always encode your samples without correct aspect ratio?

[Dark Shikari]

@DS
Why do you always encode your samples without correct aspect ratio?Because SAR doesn't affect encoding quality and I'm lazy.

[juGGaKNot]

how about a counter-strike test ?

720p60

[Boolsheet]

how about a counter-strike test ?

720p60
Is this some kind of test where you look for a source that gives the complete opposite of the last result? ... Oh, you mean the game. ;)

Is there a specific part you had in mind or just computer graphics in general? I'm sure x264 dominates the field here too, but there's just so much motion and flashes in a first person shooter, I don't think the encoders can handle that and give meaningful results. It gets even worse with newer games, now they start using "film grain". *sigh* The good old days (http://www.mediafire.com/download.php?z3qtnztnz2d) were better (or not, that game is hard).


Thanks for the comparison Dark Shikari, hope you do something like that again with new features of x264.

[Dark Shikari]

Is this some kind of test where you look for a source that gives the complete opposite of the last result? ... Oh, you mean the game. ;)

Is there a specific part you had in mind or just computer graphics in general? I'm sure x264 dominates the field here too, but there's just so much motion and flashes in a first person shooter, I don't think the encoders can handle that and give meaningful results. It gets even worse with newer games, now they start using "film grain". *sigh* The good old days (http://www.mediafire.com/download.php?z3qtnztnz2d) were better (or not, that game is hard).The good old days are still here (http://mirror05.x264.nl/Dark/Flash/extra.html) ;)

[benwaggoner]

Is there a specific part you had in mind or just computer graphics in general? I'm sure x264 dominates the field here too, but there's just so much motion and flashes in a first person shooter, I don't think the encoders can handle that and give meaningful results.
Not at 250 Kbps maybe, but with multiple reference frames or BI frames, flashes are a solved problem with H.264 and VC-1. And a good adaptive motion search range helps with rapid motion. I did some CounterStrike tests with the current VC-1 a few months ago, and it was able to do 1024x576 (IIRC) reasonably nicely well below 1 Mbps.

It gets even worse with newer games, now they start using "film grain". *sigh*
Mass Effect was much more pleasureable for me once I turned off a all the "film look" stuff. Watching all that noise and lack of contrast reminded me way too much of my day job :).

[juGGaKNot]

I can't get qp under 22 for less than 6mb at 720p and with mbtree the killtext + dark maps look really bad

not to mention the fades, really really bad.

http://www.sourceradio.com/modules.php?name=Vault&page=watch&id=1775

800p35 at 4 mb, the best i have seen so far

Format profile : High@L3.2
Duration : 4mn 8s
Bit rate mode : Variable
Bit rate : 4 000 Kbps
Maximum bit rate : 12.5 Mbps
Width : 1 280 pixels

Writing library : x264 core 67 r1153M 7b6ce6a
Encoding settings : cabac=1 / ref=5 / deblock=1:-1:-1 / analyse=0x3:0x133 / me=umh / subme=7 / psy_rd=1.0:0.0 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-2 / threads=3 / nr=0 / decimate=1 / mbaff=0 / bframes=3 / b_pyramid=1 / b_adapt=2 / b_bias=0 / direct=3 / wpredb=1 / keyint=250 / keyint_min=25 / scenecut=40 / rc=2pass / bitrate=4000 / ratetol=1.0 / qcomp=0.60 / qpmin=10 / qpmax=45 / qpstep=5 / cplxblur=20.0 / qblur=0.5 / ip_ratio=1.60 / pb_ratio=1.20 / aq=1:1.00

the movie is free to view/download, no warez.

What i'm working on now :

http://www.filefront.com/14275521/Movie_X264_2D.mp4/

Format profile : High@L4.0
Duration : 1mn 11s
Bit rate mode : Variable
Bit rate : 3 973 Kbps
Nominal bit rate : 4 000 Kbps
Maximum bit rate : 7 227 Kbps
Width : 1 184 pixels
Height : 656 pixels
Writing library : x264 core 71 r1210 42d6b17
Encoding settings : cabac=1 / ref=5 / deblock=1:-3:-3 / analyse=0x3:0x133 / me=umh / subme=9 / psy=1 / psy_rd=1.0:0.0 / mixed_ref=1 / me_range=32 / chroma_me=1 / trellis=0 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-2 / threads=3 / nr=0 / decimate=1 / mbaff=0 / bframes=5 / b_pyramid=1 / b_adapt=2 / b_bias=0 / direct=3 / wpredb=1 / keyint=250 / keyint_min=25 / scenecut=40 / rc=2pass / mbtree=0 / bitrate=4000 / ratetol=1.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / vbv_maxrate=20000 / vbv_bufsize=20000 / ip_ratio=1.10 / pb_ratio=1.10 / aq=1:1.00

[froggy1]

@juGGaKNot

try increasing qcomp a bit which lowers mbtree. I agree that mbtree does really bad on fades. Hopefully this'll be improved soon and weighted p is coming soon too.

[CruNcher]

The problem is no one will ever find out how far Ateme is by now (also because they don't do any official comparisons anymore with MSU they see no reason being broadcast optimized) it must be 2 Generations of improvements by now :)
though Dark the v 2 core is used in Nero Recode :) 1.11 seems very old they should be somewhere @ 3.5 or even 4 by now (2 Generations further then they provided Nero as their consumer space partner with) :)
Not sure but i think they wouldn't even provide a Bitstream to compare with but maybe Manao or Bobololo could make their own Graphs from the latest core and plot it given the source ;)

[Dark Shikari]

The problem is no one will ever find out how far Ateme is by now (also because they don't do any official comparisons anymore with MSU they see no reason being broadcast optimized) it must be 2 Generations of improvements by now :)
though Dark the v 2 core is used in Nero Recode :) 1.11 seems very old they should be somewhere @ 3.5 or even 4 by nowIncorrect. 1.11 is the version in their 2007 encoder; 2.x is their latest. Nero Recode has a much older version.

[CruNcher]

isn't the Core number written in the bitstream or muxed MP4 as EAVC (same as their cli encoder name encavc) and reads 2 for Nero Recode ?
Maybe 1.11 is the revision of the professional core application you tested with ?
The Cli Encoder outputs the Core version of every component (psy lib,muxer,main core,rc)
http://www.ateme.com/press_releases_archives_detail.php5?id=66 <- Core 3
your test if it's newer then Nero Recode seems to be based somewhere @ Core 2.x
maybe you meant with 2.0 the Kyrion File Encoder revision :) that's Core 3 now

x264 caught up their Speed Control (since some time now Available from Avail), Automated Profile Decision,Preset System, Better B-adapt, HRD (soon to be finished), Lookahead (in test), weighted-p (soon), open gop (soon) and introduced many of it's own nice visual enhancements (Psy-RD,new AQ,better chroma) (and of course all the speed improvements) :)

Efficiency,Visually and Speed wise it's would be a great comparison but stability wise i would count on Ateme (optimized VBV,HRD, no DPB problems, slices work, a lot of RC work) :)

[shon3i]

AFAIK Ateme 1.11 use eavc engine v.1.4.4.4 and Nero recode use something like v.1.4.5.1

wise i would count on Ateme (optimized VBV,HRD, no DPB problems, slices work, a lot of RC work) Agree, and if include lastest Mainconcept SDK for even better comparison :)

[CruNcher]

Ah yes Core 1.x actually i asked if Nero would license 2.x though now with DivX Plus HD as competition they can go straight to license Core 3.x ;)