Fourth Annual H.264 video codecs comparison published!

[DmitriyV2]

Comparison was published now!

Comparison page is: http://www.compression.ru/video/codec_comparison/mpeg-4_avc_h264_2007_en.html
Direct link to PDF:
http://www.compression.ru/video/codec_comparison/pdf/msu_mpeg_4_avc_h264_codec_comparison_2007_eng.pdf

Enjoy! :)

D.
------------------------------------------------------------------
Please forward this information to codec developers!

===========================================================================
CALL FOR MPEG4-AVC/H.264 CODECS
Fourth Annual H.264 video codec comparison
For practical researchers and developers in the field of high-end video compression
===========================================================================

Scope of Test
-------------

* Encoding time, speed/quality analysis
* Objective quality measurements (PSNR, SSIM, Average Advantage, etc.)
* Analysis of averaged objective results
* Leaders in different areas
* Decoder tests (including cross-compatibility tests)
* Special analysis of codec parts

Important Dates
---------------

August, 26 – Deadline for receipt of a H.264 codec with required presets
September, 10 – Deadline for settling technical problems with codec’s functioning
November, 7 - Draft of the report that will be sent to all participants
November, 14 - Deadline for reception of comments on the draft
December, 15 - Comparison report release

Enhancements in comparison to Previous H.264/AVC Comparison
-----------------------------------------------------------

* Cross-compatibility test
* New sequences
* New type of special analysis for codecs
* Using natural sequences' special modifications
* Using synthetic sequences
* Separate analysis of codecs main subsystems

Developer Deliverables
----------------------

The following deliverables should be provided by each developer:
* Codec files (CLI executable file is preferable)
* Short description of codec parameters
* Codec's presets with mentioning what H.264/AVC profiles are used

The full text of Call for Codecs is available at
http://www.compression.ru/video/codec_comparison/call_for_codecs_07.html

Variants of Participation
-------------------------

There are two variants for companies to participate in our comparison:

1. Participation for free. All results of your codec will be published,
except special cases of measurements problems due to codec
instability.

2. Private participation. A special report will be prepared only for your
company. This report contains:
* Your codec results and all material from the free version
* Special additional analysis of your codec

If you are interested in the private participation, please contact us
for details.

Useful Links
------------

* Third Annual MSU MPEG-4 AVC/H.264 Video Codec Comparison
http://compression.ru/video/codec_comparison/mpeg-4_avc_h264_2006_en.html
* MSU Lossless Video Codecs Comparison'2007
http://compression.ru/video/codec_comparison/lossless_codecs_2007_en.html
* Subjective Comparison of Modern Video Codecs
http://compression.ru/video/codec_comparison/subjective_codecs_comparison_en.html

[Dark Shikari]

"Minimum 1 fps for "High Quality" preset"

What computer is used for the encoding? If its pretty fast, it sounds like I should add a patch to x264 to add a Hadamard-based rather than SAD-based motion search, since that's the only way you'll get close to 1 fps without doing something stupid like using 16 reference frames, which is pointless for quality anyways. Might as well get the absolute most out of the encoding possible.

[akupenguin]

Same CPU as last year: Athlon64 3000+ in 32bit mode.
x264 has no problem getting down to 1.5fps at 720p, but that does leave a little room for improvement.

[Dark Shikari]

Same CPU as last year: Athlon64 3000+ in 32bit mode.
x264 has no problem getting down to 1.5fps at 720p, but that does leave a little room for improvement.
Heh, that means one can add 50% more instructions without going over the limit. Not bad...

[DmitriyV2]

Heh, that means one can add 50% more instructions without going over the limit. Not bad...
Sure. We received presets for x264 now.

Also this year we change MPEG-4 reference codec from DivX to XviD. We have problems with official answer from DivX and presets for all other codecs was selected by developers, so we prefer to use codec, with presets provided by developers and developers comparison verification (we sent to developers raw measured data for sure). Xvid preset's tuning for comparison (time equalization) is in progress now.

Another interesting news - AMD H.264 will be a participant. Also we will add JM (by many requests from developers, that use JM as starting and reference point for commercial codecs).

[DmitriyV2]

Our fourth comparison is close to publication.

In this report we changed MPEG-4 ASP codec (used as "reference point for new H.264") from DivX to Xvid (due to lack of communication with DivX guys). And frankly speaking results of Xvid is very good, but also during our extensive tests we found some problems, that slightly decrease sometimes quality of Xvid (we will provide all info to developers). So next year results of Xvid can be better.

Other details coming soon.

[Dark Shikari]

Our fourth comparison is close to publication. Heh, so unfortunate that all the latest improvements to x264 came after the testing date.

So I guess we'll have to take these results with a grain of salt ;)

[Sagekilla]

Heh, so unfortunate that all the latest improvements to x264 came after the testing date.

So I guess we'll have to take these results with a grain of salt ;)

A shame indeed.. I'm still waiting for someone to compile a version of x264 with all these goodies soon :) Unless I somehow figure out how to compile x264 (Which looks like quite a bit far off from happening at this point)

[DmitriyV2]

Heh, so unfortunate that all the latest improvements to x264 came after the testing date.
Codec developers of all codecs regulary tell us similar things every years, that's why we organize comparison every year at the same time :).

Also it's very gaily to see such good progress every year.

[fields_g]

Well.... When we see x264 sweep it all :D (hopefully not too optimistic), we can just smile knowing the large improvements that have happened since ~r670. Just as long as the competitors haven't made similar advances. Don't worry... we'll always have next year! I'm looking forward to this report!

[patrick_]

I think these comparisons are interesting, but to me, it would be way more interesting to see a comparison using max compression settings.
That is x264 is better than MainConcept because it allows higher compression settings. Maybe I'm different from everybody else, but I don't care about the time it takes to compress a video, I only care about quality and filesize. BTW I also don't care about seek-times or CPU requirements.
Probable x264 is also better than VC-1 because of the flexible configuration. Maybe I haven't looked enough, but it seems that about no program allows a full configuration of codecs.
-----
I use StaxRip to convert with x264. But I wouldn't mind using anything else (I use Vegas to add effects).

[Sagekilla]

I think these comparisons are interesting, but to me, it would be way more interesting to see a comparison using max compression settings.
That is x264 is better than MainConcept because it allows higher compression settings. Maybe I'm different from everybody else, but I don't care about the time it takes to compress a video, I only care about quality and filesize. BTW I also don't care about seek-times or CPU requirements.
Probable x264 is also better than VC-1 because of the flexible configuration. Maybe I haven't looked enough, but it seems that about no program allows a full configuration of codecs.
-----
I use StaxRip to convert with x264. But I wouldn't mind using anything else (I use Vegas to add effects).

The thing with that is.. While yes, it does give an idea of how well two codecs compress using all their tricks, it doesn't represent what most people will actually do, since max compression can take maybe 20 to 30 hours instead of only 6, while only offering a 2 - 3% file size decrease. Those aren't exact numbers, I'm just saying that pumping compression to max really slows thing to a halt for a very minimal quality increase.

[patrick_]

Sure it's not the way most people compress at this time, but while processing power is increasing, people will use a more advanced configuration. So it will give an impression about the future of the codecs.

[DmitriyV2]

Probable x264 is also better than VC-1 because of the flexible configuration.
Initially Microsoft with VC-1 going to participate in this year comparison and we have long discussion. But next they changed one's mind. Maybe they done some internal tests.

[Sharktooth]

They dont like unbiased opinions...

1st rule of marketing: your product is always the best. dont let anyone prove it isnt.

[DmitriyV2]

They dont like unbiased opinions...
1st rule of marketing: your product is always the best. dont let anyone prove it isnt.
This is true. But I want draw attention only to one thing - some guys in MS try to "think different". This is not so easy, I suppose.

[bond]

you can see it that way: you dont need to waste time running the comparison also on vc-1 to realise that vc-1 is worse.
m$ told you that it is worse by not taking part in the comparison ;)

[Sulik]

It's almost December 15th... Any idea when we can expect the results of the 2007 H.264 codec comparison ?

It would be good to also include low-res baseline profile results in the next comparison (baseline is still quite useful for iPod and other portable devices)

[akupenguin]

Given that the drafts were mailed on Dec 5 and not the planned Nov 7, I don't have much hope for the final publication being on time.

[IgorC]

No even one word about participants until final publication. Is it any secret information? KGB style.
"The results will be lately". Too much vodka, guys.

[DmitriyV2]

It's almost December 15th... Any idea when we can expect the results of the 2007 H.264 codec comparison?
Now final fixes and remarks are included taking into consideration codec developers feedback. Sorry for delay!

It would be good to also include low-res baseline profile results in the next comparison (baseline is still quite useful for iPod and other portable devices)
Hm. Good idea!

[Nil Einne]

Microsoft don't want to participate with VC1 but how about Xiph.org with Theora? With it reaching v1.0 soon if it's not in this years hopefully it'll be in next years.

Edit: Although then again, after reading comments for people who seem to know what they're talking about it seems Theora can't even really compared to XviD yet so I guess there's no point

[DmitriyV2]

Microsoft don't want to participate with VC1 but how about Xiph.org with Theora? With it reaching v1.0 soon if it's not in this years hopefully it'll be in next years.

Edit: Although then again, after reading comments for people who seem to know what they're talking about it seems Theora can't even really compared to XviD yet so I guess there's no point
Ok! When they will be ready, we will be glad to include them. We invite second open source codec to comparison this year and there is no problem to add third. Of course this is not H.264 and not so resonable reference point, but we can measure it carefully in appendix, no problem.

[omf]

Dmitriy, when do you expect the results to be available?

[fatso485]

when do expect to see the results. is it the 15th yet.


also is there another recent codec comparison that i can look into

[CruNcher]

Ok! When they will be ready, we will be glad to include them. We invite second open source codec to comparison this year and there is no problem to add third. Of course this is not H.264 and not so resonable reference point, but we can measure it carefully in appendix, no problem.

Is this second Open Source Codec XviD 2.0 AVC :) ?

[pc2099]

Is the report out yet? Also given mainconcept was the winner in 2006 and is now owned by DivX...will DivX AVC be the leader in quality/speed in 2008?

Interesting times indeed

[Alex-Kid]

Date: 2 January, 2008 (btw, Happy New Year to everyone :))

Fourth AVC Video Codec Comparison still missing.

Does someone know something about Ateme codec for this comparison? I missed it last year comparison, because it was the winner in 2005.

Keeping in touch.

Saludos, by ALEX-KID

[foxyshadis]

Hey, come on, no need to pile onto the thread, one query is plenty.

[CruNcher]

Date: 2 January, 2008 (btw, Happy New Year to everyone :))

Fourth AVC Video Codec Comparison still missing.

Does someone know something about Ateme codec for this comparison? I missed it last year comparison, because it was the winner in 2005.

Keeping in touch.

Saludos, by ALEX-KID
A Happy New Year to you and everyone else too

I think Ateme is not going to take place in the Test again, as the results of it seem not to be important for them their Market @ the moment is another one (Broadcast), now with the buyout of Mainconcept to DivX Inc im not sure if that stays this way ;)
It does really nice especialy with many problems that other Codecs still suffer from very special HVS stuff that rarely occours in pre rendering, but more in Live Scenarios ofcourse it can also happen in non Live low bitrate conditions and they doing really some magical HVS tricks in such situations it's Amazing.

[DmitriyV2]

Dmitriy, when do you expect the results to be available?
Now preliminary final report is ready and we plan to publish it 14 January.

Sorry for delay - I did not want to announce exact date without exact information.

[omf]

Now preliminary final report is ready and we plan to publish it 14 January.

Sorry for delay - I did not want to announce exact date without exact information.

No problem, and thanks for the update.

[Sulik]

Dmitriy: Any updates ?

[DmitriyV2]

Specially for you :)

Please find results is here:
http://www.compression.ru/video/codec_comparison/mpeg-4_avc_h264_2007_en.html
Direct link to PDF:
http://www.compression.ru/video/codec_comparison/pdf/msu_mpeg_4_avc_h264_codec_comparison_2007_eng.pdf

Thanks for patience!

[Dark Shikari]

Quite an old version of x264 used there... :p

[Sulik]

Excellent. Too bad there isn't more RD curves or image comparisons this year (I guess we have to pay for it now - too high for my budget :)

If I had some suggestions for next year, it would be the following:

1. Use bitrates proportional to the resolutions (measured in bits per pixel). Realistically, regardless of the encoder used, if the PSNR is in the mid-to-low 30's, the quality is crap.

2. Include a separate section for Baseline profile (CAVLC, no B-frames), since this is a very commonly used scenario for H.264 encoding (iPod & other devices) -> this might be a bit more fair to encoders like AMD, since it's difficult to tell if the differences come mostly from Baseline vs High Profile or from encoder implementation (Or at least include a standard Baseline reference such as JM in constant quant mode).

3. (Related to #1) Go a bit higher on the bitrate scale, instead of only looking at the section of RD curves where the encoders start breaking down (though this might reduce SNR differences between encoders).

4. Include a wider range of content complexity, possibly using standard sequences (sequences such as Mobile & Calendar comes to mind)

5. I'd rather seem more RD graphs and screenshots for different sequences than basically the same data displayed in 4 different forms

[DeathTheSheep]

Only 9 revisions behind my own. :)

[CruNcher]

Reading the results i can be quiet sure now that my Visual Interpretation of Atemes Codec is not wrong and it seems still to be the most Advanced :)
Im not makeing this statement from a pure Scientific Point of View, more from constantly analysing the Look and Feel of X264, Mainconcept and Ateme over the years.
This results now confirm that X264 and Mainconcept are almost identical (i can confirm that is the case also Look and Feel wise) compared to Atemes Implementation wich imho is the best i saw so far for real footage. Also it's sad that XviD 2.0 AVC was not in the test @ all, im suprised that it's still not here yet :(
Who are those Artemis guys and what the Hell they are deoptimizing x264 for (looks like somekind of decoding complexity reduction optimization to me) that I frame "pulse" looks strange :D

[IgorC]

I'm totally agree about Ateme encoder.
x264 and MC are quite close but there is lack of psy optimizations. And Ateme has it.


DmitriyV2 Thanks for result.
For me AMD and Intel encoders were totally out of interest.
Maybe just pretest and then discard them.
Instead of it give a try to Apple and Ateme (or Nero). imo

[Dark Shikari]

I'm totally agree about Ateme encoder.
x264 and MC are quite close but there is lack of psy optimizations. And Ateme has it.Try the new AQ... I don't think a 22% bitrate-adjusted SSIM boost (with appropriately correlated increased visual quality) is something to scoff at ;)

[DeathTheSheep]

I had a feeling you'd eventually come in and say that, DS. I was biding my time with bated breath. :)

(Now if only the that 3% absolute quality boost in me-prepass can live again... A quality gain roughly equivalent to 1/3 of that derived from using CABAC is contained within it, after all.)

[CruNcher]

Jep Dark indeed your AQ boosts the Percepted Visual Quality tremendously for some hard scenes (no doub't in that) and i have to reevaluate with it now, but overal the look and feel based on the momentarily used deadzone system introduced by Alex stays the same (for most other scenes) it seems :)

[shon3i]

@CruNcher, IgorC, i am agree with you about ateme, realy can produce best quailty for given bitrate and comparing to MC/Elecard and x264 , always beat by hair. But sometimes from my testing Elecard's psy optimisations can done better job, than ateme. Complexity algo with negative values can produce magical to some footage. I think elecard converter studio 2.0 now have all options that one AVC encoder need for very quality encode and aslo fast, and most important to me is compactibility with hardware acceleration decoders to i can watch, only miss CQM :(

[Wilbert]

I'm a bit baffled by the conclusions. Statements like
The main reason of AMD encoder low quality is very high speed of the encoder.
are clearly false. The low quality is not the consequence of the 'very high speed' (more precisely the 'high speed preset' which was used in the encoding process). Both are the consequence of the fact that only features of the codec are used in the encoding process (or implemented from the standard ) which are computationally fast, but don't result in good quality.

I don't know some of the codecs, but reading your article it seems that all three can be the reason of the bad quality, depend on the codec under consideration.

Also, what do you mean with the following statement:
The XviD codec demonstrates difficulties with bitrate handling algorithms, so does the AMD encoder as well.


Last question. Is Artemis x264 also GPL (since you claim it is based on x264)?

[Ranguvar]

By the Xvid comment, I think he was referring to the 2-pass rate control in it.

[DmitriyV2]

I started to write FAQ for this comparison - see draft below:

----------------------------------------------------
Q. What was main goal of the comparison?

A. main idea of this annual comparison - official commercial companies
involving and so increasing number of participants. We are already
involved Intel, AMD, ATI, Apple (partially) and smaller Fraunhofer,
Elecard, Ateme, VSS, MainConcept, Sorenson, Artemis etc.

Q. Why official? You can get codec from a developer's site or (if
unavailable) from P2P networks and compare them successfully!

A. This is not true. For example, key feature of our comparisons is
comparison in several application areas (videoconferences, SD,
HDTV). Different areas require different codec settings. The
simplest strategy -- use of default settings -- definitely works,
but it is as much definitely nonoptimal (in terms of
quality-bitrate balance). So we received more optimal settings from
developers to test codec with higher accuracy. Also every year we
change sequences in an unpredictable way to prevent tuning to
specific sequences by developers.

Another problem is speed. Default settings of one codec can made it
10 time slower than another one and all your quality measurements
will not be so correct. Of course we can select suitable parameters
with close speed, but everybody can say "you select wrong
parameters for our codec, so you received wrong results", that's
why we do not use this approach in the annual H.264 comparison. Our
current solution is not sophisticated: We just asked companies for
optimal settings (as they think of them) for the predefined
application area and predefined time limits. The first step of
codec qualification to the comparison is time measurement: we
report time, measured on our system to developers with comments
like "you have 50% overtime, please make your preset faster", or
"your time reserve is still 40%, you can make your preset slower,
but with better quality/rate results".

Another advantage of official contacts: we typically receive the
newest internal versions, that are planned to be published not very
soon (it was up to half a year for one of our participants).


Q. Fine. But why you have so long timeout between starting till
publication?

Several reasons:

1. As we told previously - we have slow speed acceptance procedure,
that require correspondence with companies.

2. Next - if it's possible - we try to get companies chance to fix
clear bug. We receive fresh research version, so some bugs inside
is possible (especially this was actual for second annual
comparison). This can be very simple problem like "unlucky
build" with fast fixing. So if we have time reserve, we stop
measurement, report bug and get company chance to fix it. Of
course we wait very few time (2-3 days) with clear final
deadline, but this procedure also add additional delay.

3. Also we have measurements approval procedure. After we done all
measurements and after internal checking we provide measured
results of company's codec and test sequences to correspondent
companies, so companies are able to check our measurements with
there codec. We published our measurement tool (MSU Video Quality
Metric) and provide our measurement results in it's format,
so all results can be verified.

4. Next step is reports preparation (short, long versions and
versions for private comparison). Report size can be up to 200
pages, we prepare several reports, so this is also not so fast
step.

5. And final step is reports verification and reviewing. We provide
report draft to all participants for verification. They report
defect and commonly good suggestions. If we think, that some
suggestion is very important we can add appendix to comparison
with additional measurements or some clarifications. This also
require time. And final step is independent reviewing by very
good independent experts in H.264 codecs.

Q. Ok. Situation with time is clear. Is it possible for a company to
remove some results if it ask for this?

A. Now we avoid such situation by using a so-called private participation.

The motivation is pretty simple: at the start, without knowing its
real position against competitors, a developer did want to
participate in the public comparison. But after receiving results,
if this is not the first place or so, they did not want to see
these results published :). So, if we allow to remove, there will
be only one codec in the comparison. Maybe two. :) (somebody and
x264). So we restrict such removing now, but allow private
participation.

Some companies did not want to participate due to not so good
current results which they know beforehand about. In private
participation a company pay us and receive its personal report.
This company codec is not included in the public comparison at all.
So, if company pay us, we prevent any conflicts of interests. A
company can understand its place and does further works on codec
improvement, but do not get its current results published and
well-known (the annual comparisons are very popular - we have more
than 200000 total downloads now).

Q. And the main question: what codec is the best?

A. We really have maybe the maximum information about the best codecs in
the world and we really do not know. :) At first, please specify
usage area, speed and other requirements, etc, etc, etc... We can
answer what codec is better for some common usage samples.

Our another objective is to help codec developers to make there
codecs better and better and better. Commonly developers achieve
some really good results and believe that this is the absolute
limit, but we help them to see new limits.

---------------------------------------------------------

And again - any updates, fixes and suggestions are welcome!

Also I'll answer all your questions soon (thanks for good ideas!)

[CruNcher]

You should maybe better write.
"So, if companies pay us for the private participation, we avoid any conflicts of interests."
Else in context to the rest of the text it sounds a little corrupt, like hey if X264 devs would pay us they would stay ontop ;)
Im not sure how much effect this test actually had now for the Quality of X264 did you found any severe bugs in the progress and reported problems to the devs, for example the problems with that LOTR Sample seem to be a Quality problem?

[audyovydeo]

And again - any updates, fixes and suggestions are welcome!


I know I'l going to get tons of flak for this but so be it.
I think you got your report distribution model wrong.
My numbers are guestimates of course :

2008 price customers revenue
full_report 600$ 200 120000
half_report 0 200000 0

2009 price customers revenue
full_report 2$ 200000 400000

personally I'd spend 2$ and read the full report rather than have a free but wishy-washy one. This year's report was much anticipated but, if you noticed, it didn't spark a rain of forum messages. And I'm not sure you get thousands of companies buying it at full price.

Oh I have another question that I know you will not answer : did Apple decline to take the test this year, or was it excluded because of of its lack of High profile support ?

cheers
audyovydeo

[bob0r]

CruNcher why would x264 ever pay? LOOL
x264 is the only encoder widely used, and most "groups" use even such horrible settings, quality is something they dont care about anyway, sadly but true :/

[Dark Shikari]

CruNcher why would x264 ever pay? LOOL
x264 is the only encoder widely used, and most "groups" use even such horrible settings, quality is something they dont care about anyway, sadly but true :/Oh how true (http://i32.tinypic.com/f258k4.jpg).