MPEG-4 AVC/H.264 video codecs comparison

[Dyomich]

Dear doom9 experts,

Moscow State University Graphics & Multimedia Laboratory has finished 6-th H.264 codecs comparison.
It is intended for practical researchers and developers in the field of high-end video compression.

http://compression.ru/video/codec_comparison/h264_2010

We have tested newest implementations of MPEG-4 AVC/H.264 video codecs and compare with XviD (MPEG-4 ASP) and Theora encoders.
One of the main targets for this comparison was to test H.264 encoders for transcoding tasks for Movies and HDTV video content.

Codec that were tested:

• DivX H.264
• Elecard H.264
• Intel® MediaSDK AVC/H.264
• MainConcept H.264
• Microsoft Expression Encoder
• Theora
• x264
• XviD (MPEG-4 ASP codec)
• VP8

Summary report topics:

• Objective measurements (SSIM, PSNR, Average Advantage and etc.)
• Encoding speed
• Analysis of averaged objective results
• Leaders in different areas (Movies, HDTV)
• Options analysis for codecs
• Additional subjective analysis for video codes

  • For psycho-visual enhancement in codecs analysis
  • For fade processing analysis
  • For animation movie compression analysis

• Codecs encoding quality progress over years

Enhancements in comparison to Previous H.264/AVC Comparison:

• Subjective comparison
• New codecs
• New sequences
• New type of special analysis for codecs
• Using natural sequences' special modifications
• Using synthetic sequences
• Not only H.264 Codecs (but also XviD, Theora) were tested

Some examples from the comparison report:

This figure depicts RD-curve for bitrate/quality. Higher curve corresponds better encoding quality. This graph shows quality drop for Theora encoder at 1000kbps.

This figure depicts bitrate handling graph – encoders with good bitarte handling methods has horizontal lines close to 1.0 value. This graph shows strange bitrate handling methods for MS Expression encoder. The more information for it could be found by other graphs analysis.

This figure depicts the progress of the x264 encoder over several years. Y-axis shows encoding quality – encoders with its mark higher than other have better quality. X-axis shows encoding time – encoders with its marks placed to left are faster than other. Therefore encoders in upper-left corner are best – faster and have higher quality than competitors.

More detailed analysis could be found at next page

http://compression.ru/video/codec_comparison/h264_2010

Best regards,
Dr. Dmitriy Kulikov,
Head of Video Codec Testing team,
Graphics&Media Lab,
Moscow State University

[mbohupa]

Wow! Great article!!

[Blue_MiSfit]

Thanks for posting this as always, Dr. Kulikov. MSU's comparisons are always thorough and well written. I personally think it would have been interesting to compare some extreme cases with much lower speed requirements, so x264's extra special sauce could be used.

Still, I'm totally unsurprised by x264's dominance. Thank you again.

Derek

[Shevach]

Dr. Kulikov

According to your report only encoders were compared, i.e. Rate Control and Choose Mode.
My question concerns the assessment of decoders, namely the robustness of decoders under bit-stream errors.
Have you any methods to compare H.264 decoders?

[Keiyakusha]

Originally Posted by Shevach

Have you any methods to compare H.264 decoders?

There is nothing to compare except speed. And thats pretty much clear, just read some CoreAVC, DiAVC threads.

[nm]

Originally Posted by Keiyakusha

There is nothing to compare except speed. And thats pretty much clear, just read some CoreAVC, DiAVC threads.

Robust handling of corrupted streams is quite important for digital television reception, for example. That's why many people prefer lavc for MPEG-2 decoding.

[PhrostByte]

Nice article, great job!

[Dyomich]

Originally Posted by Shevach

Dr. Kulikov

According to your report only encoders were compared, i.e. Rate Control and Choose Mode.
My question concerns the assessment of decoders, namely the robustness of decoders under bit-stream errors.
Have you any methods to compare H.264 decoders?

Yes, we have a methodlogy to comapre and analyze H.264 decoders, also we had a project on private MPEG-2 and H.264 decoders comparison with codec developer company.

[Dyomich]

Originally Posted by nm

Robust handling of corrupted streams is quite important for digital television reception, for example. That's why many people prefer lavc for MPEG-2 decoding.

Yes, you are right. Error concelament algorithm is very improtant point of any decoder.
You have pointed to one of our previous decoders comparison. Now we have better methodology for decoder comparison.

[Dyomich]

Originally Posted by PhrostByte

Nice article, great job!

Thank you

[Raptus]

The leader in this comparison is x264—its quality difference (according to the SSIM metric) could be explained by the special encoding option ("tune-SSIM"). Interestingly, using the PSNR metric for MainConcept yielded results comparable with or better than those of x264. This means that no encoder can achieve the best results for both SSIM and PSNR when using the same parameters.

Of course. The requirements for optimizing for each of those metrics (or visual quality for that matter) are not orthogonal. Why omit that x264 can also be tuned for PSNR?

I'm also missing a downloadable PDF version of the free report.

[dapperdan]

Originally Posted by Raptus

Why omit that x264 can also be tuned for PSNR?

I read that quote as meaning that when both encoders were tuned for PSNR, MainConcept won (though it not's clear that's what it actually means).

It seems the for-pay versions have the PSNR graphs included, so if anyone has access to those they could confirm one way or the other.

[Dyomich]

Originally Posted by Raptus

I'm also missing a downloadable PDF version of the free report.

We plan to include downloadable version of pdf in few days.

[bob0r]

The moment you posted the results, they are already outdated and x264 has already moved further ahead!

[CruNcher]

also a more important contender entered the stage after the call was made VP8

[Astrophizz]

They plan to add VP8 in an appendix.

[Dyomich]

VP8 has recently attracted a lot of interest after it was owned by
Google. As you know, on May 2010, the WebM Project was launched,
featuring contributions from "Mozilla, Opera, Google and more than
forty other publishers, software and hardware vendors" in a major
effort to use VP8 as the codec for HTML5.

As one of appendixes to the annual H.264 comparison report
an additional VP8 encoder vs. x264 encoder comparison was presented.

We have tested VP8 encoder and compare its encoding quality and
speed with x264.

The final report contains all RD-curves, bitrate handling analysis and
speed/quality graphs. Six different VP8 presets were tested which
were chosen with the help of VP8 developers (so those were VP8
developers guided settings).

http://www.compression.ru/video/code...8_vs_h264.html

[Dark Shikari]

You really shouldn't allow misleading comments by developers to be posted like that. They claim that artifacts from MPEG compression "benefit H.264 and MPEG" and bias against VP8, but VP8 has almost the exact same transform scheme as H.264, and the same transform size. The "bias" against VP8 should be practically the same as the bias against any other H.264 encoder.

This kind of lie has been spread constantly when it comes to any non-MPEG video format; the same was repeated over and over with Theora, despite the fact that Theora's transform is practically identical to MPEG-1/2/4.

[fields_g]

Exactly what I was thinking when I read that statement. I'm glad someone with a bit more technical knowledge spoke up.

[Dyomich]

Originally Posted by Dark Shikari

You really shouldn't allow misleading comments by developers to be posted like that. They claim that artifacts from MPEG compression "benefit H.264 and MPEG" and bias against VP8, but VP8 has almost the exact same transform scheme as H.264, and the same transform size. The "bias" against VP8 should be practically the same as the bias against any other H.264 encoder.

This kind of lie has been spread constantly when it comes to any non-MPEG video format; the same was repeated over and over with Theora, despite the fact that Theora's transform is practically identical to MPEG-1/2/4.

Yes, you are probably right, but our comparison rules permit developers to post their comments to comparison.