MPEG-4 AVC/H.264 video codecs comparison

[Dyomich]

Dear doom9 experts,

Moscow State University Graphics & Multimedia Laboratory has finished 6-th H.264 codecs comparison.
It is intended for practical researchers and developers in the field of high-end video compression.

http://compression.ru/video/codec_comparison/h264_2010

We have tested newest implementations of MPEG-4 AVC/H.264 video codecs and compare with XviD (MPEG-4 ASP) and Theora encoders.
One of the main targets for this comparison was to test H.264 encoders for transcoding tasks for Movies and HDTV video content.

Codec that were tested:

DivX H.264
Elecard H.264
Intel® MediaSDK AVC/H.264
MainConcept H.264
Microsoft Expression Encoder
Theora
x264
XviD (MPEG-4 ASP codec)
VP8


Summary report topics:

Objective measurements (SSIM, PSNR, Average Advantage and etc.)
Encoding speed
Analysis of averaged objective results
Leaders in different areas (Movies, HDTV)
Options analysis for codecs
Additional subjective analysis for video codes

For psycho-visual enhancement in codecs analysis
For fade processing analysis
For animation movie compression analysis

Codecs encoding quality progress over years


Enhancements in comparison to Previous H.264/AVC Comparison:

Subjective comparison
New codecs
New sequences
New type of special analysis for codecs
Using natural sequences' special modifications
Using synthetic sequences
Not only H.264 Codecs (but also XviD, Theora) were tested


Some examples from the comparison report:

http://compression.ru/video/codec_comparison/h264_2010/figures/rdmv_normal_ice.png
This figure depicts RD-curve for bitrate/quality. Higher curve corresponds better encoding quality. This graph shows quality drop for Theora encoder at 1000kbps.

http://compression.ru/video/codec_comparison/h264_2010/figures/bhmv_normal_up.png
This figure depicts bitrate handling graph – encoders with good bitarte handling methods has horizontal lines close to 1.0 value. This graph shows strange bitrate handling methods for MS Expression encoder. The more information for it could be found by other graphs analysis.

http://compression.ru/video/codec_comparison/h264_2010/figures/x264_over_years_2010_sq_Y-SSIM.png
This figure depicts the progress of the x264 encoder over several years. Y-axis shows encoding quality – encoders with its mark higher than other have better quality. X-axis shows encoding time – encoders with its marks placed to left are faster than other. Therefore encoders in upper-left corner are best – faster and have higher quality than competitors.

More detailed analysis could be found at next page

http://compression.ru/video/codec_comparison/h264_2010 (http://compression.ru/video/codec_comparison/h264_2010)

Best regards,
Dr. Dmitriy Kulikov,
Head of Video Codec Testing team,
Graphics&Media Lab,
Moscow State University

[mbohupa]

Wow! Great article!!

[Blue_MiSfit]

Thanks for posting this as always, Dr. Kulikov. MSU's comparisons are always thorough and well written. I personally think it would have been interesting to compare some extreme cases with much lower speed requirements, so x264's extra special sauce could be used.

Still, I'm totally unsurprised by x264's dominance. Thank you again.

Derek

[Shevach]

Dr. Kulikov

According to your report only encoders were compared, i.e. Rate Control and Choose Mode.
My question concerns the assessment of decoders, namely the robustness of decoders under bit-stream errors.
Have you any methods to compare H.264 decoders?

[Keiyakusha]

Have you any methods to compare H.264 decoders?
There is nothing to compare except speed. And thats pretty much clear, just read some CoreAVC, DiAVC threads.

[nm]

There is nothing to compare except speed. And thats pretty much clear, just read some CoreAVC, DiAVC threads.

Robust handling of corrupted streams is quite important for digital television reception, for example. That's why many people prefer lavc for MPEG-2 decoding (http://compression.ru/video/codec_comparison/mpeg-2_2006_en.html).

[PhrostByte]

Nice article, great job!

[Dyomich]

Dr. Kulikov

According to your report only encoders were compared, i.e. Rate Control and Choose Mode.
My question concerns the assessment of decoders, namely the robustness of decoders under bit-stream errors.
Have you any methods to compare H.264 decoders?

Yes, we have a methodlogy to comapre and analyze H.264 decoders, also we had a project on private MPEG-2 and H.264 decoders comparison with codec developer company.

[Dyomich]

Robust handling of corrupted streams is quite important for digital television reception, for example. That's why many people prefer lavc for MPEG-2 decoding (http://compression.ru/video/codec_comparison/mpeg-2_2006_en.html).
Yes, you are right. Error concelament algorithm is very improtant point of any decoder.
You have pointed to one of our previous decoders comparison. Now we have better methodology for decoder comparison.

[Dyomich]

Nice article, great job!
Thank you :)

[Raptus]

The leader in this comparison is x264—its quality difference (according to the SSIM metric) could be explained by the special encoding option ("tune-SSIM"). Interestingly, using the PSNR metric for MainConcept yielded results comparable with or better than those of x264. This means that no encoder can achieve the best results for both SSIM and PSNR when using the same parameters.
Of course. The requirements for optimizing for each of those metrics (or visual quality for that matter) are not orthogonal. Why omit that x264 can also be tuned for PSNR?

I'm also missing a downloadable PDF version of the free report.

[dapperdan]

Why omit that x264 can also be tuned for PSNR?

I read that quote as meaning that when both encoders were tuned for PSNR, MainConcept won (though it not's clear that's what it actually means).

It seems the for-pay versions have the PSNR graphs included, so if anyone has access to those they could confirm one way or the other.

[Dyomich]

I'm also missing a downloadable PDF version of the free report.
We plan to include downloadable version of pdf in few days.

[bob0r]

The moment you posted the results, they are already outdated and x264 has already moved further ahead! :D

[CruNcher]

also a more important contender entered the stage after the call was made :( VP8

[Astrophizz]

They plan to add VP8 in an appendix.

[Dyomich]

VP8 has recently attracted a lot of interest after it was owned by
Google. As you know, on May 2010, the WebM Project was launched,
featuring contributions from "Mozilla, Opera, Google and more than
forty other publishers, software and hardware vendors" in a major
effort to use VP8 as the codec for HTML5.

As one of appendixes to the annual H.264 comparison report
an additional VP8 encoder vs. x264 encoder comparison was presented.

We have tested VP8 encoder and compare its encoding quality and
speed with x264.

The final report contains all RD-curves, bitrate handling analysis and
speed/quality graphs. Six different VP8 presets were tested which
were chosen with the help of VP8 developers (so those were VP8
developers guided settings).

http://www.compression.ru/video/codec_comparison/h264_2010/vp8_vs_h264.html

[Dark Shikari]

You really shouldn't allow misleading comments by developers to be posted like that. They claim that artifacts from MPEG compression "benefit H.264 and MPEG" and bias against VP8, but VP8 has almost the exact same transform scheme as H.264, and the same transform size. The "bias" against VP8 should be practically the same as the bias against any other H.264 encoder.

This kind of lie has been spread constantly when it comes to any non-MPEG video format; the same was repeated over and over with Theora, despite the fact that Theora's transform is practically identical to MPEG-1/2/4.

[fields_g]

Exactly what I was thinking when I read that statement. I'm glad someone with a bit more technical knowledge spoke up.

[Dyomich]

You really shouldn't allow misleading comments by developers to be posted like that. They claim that artifacts from MPEG compression "benefit H.264 and MPEG" and bias against VP8, but VP8 has almost the exact same transform scheme as H.264, and the same transform size. The "bias" against VP8 should be practically the same as the bias against any other H.264 encoder.

This kind of lie has been spread constantly when it comes to any non-MPEG video format; the same was repeated over and over with Theora, despite the fact that Theora's transform is practically identical to MPEG-1/2/4.

Yes, you are probably right, but our comparison rules permit developers to post their comments to comparison.