Third Annual MSU MPEG4-AVC/H.264 codecs comparison (published!)

[DmitriyV2]

LAST NEWS: Comparison was published, it's URL is:
http://www.compression.ru/video/codec_comparison/mpeg-4_avc_h264_2006_en.html
---------------------------------------------------------------------

Please, forward this information to H.264 codecs developers!

MSU Graphics&Media Lab Video Group announce third Annual H.264 video codec comparison:

===========================================================================
CALL FOR MPEG4-AVC/H.264 CODECS
Third Annual H.264 video codec comparison
For people, who make real research in field of high-end video compression
===========================================================================

Scope of test
-------------

* Encoding and decoding time
* Objective measurements (PSNR, SSIM, VQM, Average Advantage and etc)
* Analysis of averaged objective results
* Leaders in different areas
* Cross-compatibility test
See previous comparison (http://www.compression.ru/video/codec_comparison/mpeg-4_avc_h264_2005_en.html) for detailes

Important Dates
---------------

* August, 1 - deadline for receipt of a H.264 codec with required presets
* August, 15 - deadline for settling technical problems with codec's functioning


The Facts about the Previous H.264 Video Codecs Comparison
----------------------------------------------------------

* There were more than 50.000 downloads of previous H.264 video codec comparison results
* Many codec's bugs were found and reported to developers
* Following codecs was granted for second annual comparison by
developers last year (in alphabetic order):
- ArcSoft H.264
- Ateme H.264
- ATI H.264
- Elecard H.264
- Fraunhofer IIS H.264
- VSS H.264
- x264
* Intel H.264 will going to take part in comparison this year


Enhancements from Previous H.264/AVC Comparison
-----------------------------------------------

* Better sequences set (several subsets with independent encoding presets
and results calculation)
* Decompression will be performed by JM reference software
* Comparison of original decoders and JM reference decoder (postfilters
testing) will be added
* Cross-compatibility test will be added
* Decoders time measurement will be added
* 2-pass encoding will be added and tested


Developers Deliverables
-----------------------

Following deliverables should be provided by each developer:
* Codec files (CLI executable file is preferable)
* Short description of codec parameters
* Codec's presets with pointing of used H.264/AVC profile


Useful Links to Previous Comparisons
-----------------------------------

* Second Annual MSU MPEG-4 AVC/H.264 Video Codec Comparison
http://www.compression.ru/video/codec_comparison/mpeg-4_avc_h264_2005_en.html
* MPEG-2 Video Decoders "Crash-Test" Comparison
http://www.compression.ru/video/codec_comparison/mpeg-2_2006_en.html
* MSU MPEG-4 SP/ASP Codecs Comparison
http://www.compression.ru/video/codec_comparison/mpeg-4_en.html
* MSU Lossless Video Codecs Comparison
http://www.compression.ru/video/codec_comparison/lossless_codecs_en.html

See full text of CALL FOR MPEG4-AVC/H.264 CODECS on
http://www.compression.ru/video/codec_comparison/call_for_codecs_06.html

----------------------------------------------
Our current roadmap (codecs testing area):
1) Third Annual H.264 video codec comparison (extended in comparison with previous year).
2) MPEG-4/MPEG-4 AVC Decoders "Crash-test"

Enjoy! :)

[Golgot13]

No comparison between Main Profil and High Profil ??

All hardware chipsets accept at maximum level 4.1, I suggest to limit
the level at 4.1 or 4.0.

Golgot13

[Revgen]

I'm glad you're doing 2-pass this time.

But like Golgot13 said, we would also like to see more HP tests.

Good Luck

[DmitriyV2]

No comparison between Main Profil and High Profil?? All hardware chipsets accept at maximum level 4.1, I suggest to limit the level at 4.1 or 4.0.
This year we will ask developers to disclose codec profile also we are thinking now about profile limitation for videoconferences stream.

[DmitriyV2]

I'm glad you're doing 2-pass this time.
But like Golgot13 said, we would also like to see more HP tests.
We expect enough number of HP codecs this year. If company has no high profile - there is no limitations, but there is no reason to use BP or MP if codec has HP for best quality presets. And again - if HP is not so tuned - company can use the best from there point of view.

[akupenguin]

Developers Deliverables
* Short description of codec parameters
* Codec's presets with pointing of used H.264/AVC profile


Does this mean you want, (a) a commandline you should use, and a description of what is used by default if you don't set the options
or (b) a description of what each option does, and a commandline you should use ?

[DmitriyV2]

Does this mean you want, (a) a commandline you should use, and a description of what is used by default if you don't set the options
or (b) a description of what each option does, and a commandline you should use ?
Last year there was resonable questions - what parameters was used for codecs. So this year we ask for short description to codec parameters (where it's not clear what was used).

x264 has good documentation, so no additional information about parameters it's necessary. For other codecs (without public command line version) such infrmation is required.

[DmitriyV2]

Short update: we test received codecs now and help companies to select parameters with approximatly the same compression time. Number of codecs will be close to previous year. Also we think about tests (with selected presets) of codecs not only from developers (for some interesting codecs).

And I repeat the differencies from last year comparison:
* Better sequences set (several subsets with independent encoding presets and results calculation)
* Decompression will be performed by JM reference software
* Comparison of original decoders and JM reference decoder (postfilters testing) will be added
* Cross-compatibility test will be added
* Decoders time measurement will be added
* 2-pass encoding will be added and tested

See Call for codecs page: http://www.compression.ru/video/codec_comparison/call_for_codecs_06.html

[fields_g]

I'm curious how your comparision is going. Do you have an estimate on when you might present your findings?

[DmitriyV2]

I'm curious how your comparision is going. Do you have an estimate on when you might present your findings?
We have several commercial projects started from September, so we try to collect codecs and make measurements in August (as clever guys). But really guys from ALL (!) companies did not prepare there codecs in time. August was too hot month for exact planning :). So...

So comparison is moving forward, but as background project (at some evenings after commercial projects, sometimes at nights). I'l post news periodically (you right this is good idea to keep people informed).

[DmitriyV2]

Situation update:

We measure more than 100Mb of digits in csv files.
To measure all we want we need bigger time than reasonable. So we remove decoders performance/time comparison (and cross-compatibility test, unfortunately). This was bad news, good news: we add "cross years" comparison where we present results of 15 versions of several codecs during 3 years of measurements.
2 weeks ago measurement results was sent to developers for verification.
One week ago short report draft was sent to developers for verification.

So we are on homestretch of comparison.

[DmitriyV2]

Final draft is ready. If anyone here want to be a reviewer please write us on
videocodec_testing on the graphics.cs.msu.ru
(http://www.compression.ru/video/codec_comparison/images/videocodec_testing_graphics_cs_msu_ru.gif ,)
we will send draft to you.

[DmitriyV2]

Third annual H.264 comparison published!

Tested codecs:
* MainConcept H.264
* Intel H.264 (as was promised, Intel provide there codec this year and will provide better version next year, thay already have it)
* VSS H.264
* x264
* Apple H.264 (partially tested)
* Sorenson H.264 (partially tested)

Most interesting features:
1. Using of integral characterictics.
2. Cross-years comparison of codecs for 2004-2005-2006.

Comparison URL: http://www.compression.ru/video/codec_comparison/mpeg-4_avc_h264_2006_en.html

And big thanks to all comparison reviewers!

Enjoy! :)

[bond]

:thanks: a lot. seems mainconcept/elecard gets closer and closer to x264

[Audionut]

Your download link says " Second Annual MSU H.264 Video Codec Comparison".

Thanks for your work.

[DmitriyV2]

:thanks: a lot. seems mainconcept/elecard gets closer and closer to x264
Yes. Originally they are optimazed codec for satellite broadcasters primarily (this is really another requarements and this is reason why Ateme take off their codec from test), and now mainconcept/elecard have time for another optimization.

[DmitriyV2]

Your download link says " Second Annual MSU H.264 Video Codec Comparison".
What link, sorry?

Thanks for your work.
Welcome!

[Audionut]

http://img222.imageshack.us/img222/21/20070101010546dk0.png

[DmitriyV2]

Thanks! I was in trip after New Year and now everithing was fixed.
Thanks for picture (I did not found where is error after first mails with it's bugreport)!