MSU MPEG-4 AVC/ H.264 codec comparison test

[DmitriyV2]

MSU MPEG-4 AVC/ H.264 codec comparison test released!

Main features:
* 6 H.264 codecs was compared with last DivX.
* 3 codecs was recieved from codec developers directly for test.
(We would like to thank Moonlight Cordless LTD, Fraunhofer Institute for Integrated Circuits IIS and Ateme for kindly providing us their codecs for this test.)
* We measure the most used PSNR, Bitrate handling and visual comparison.

This comparison was first is series.

In next version (will be in 2 month):
* New codecs will be added
* We plan to add new measures
* 2 presets, recieved from codec developers will be measured:
"tuned" - maximum quality,
"fast" - maximum (optimum) speed
* Rules for uniformal comparison will be more formal. :)

http://compression.ru/video/codec_comparison/mpeg-4_avc_h264_en.html

Enjoy! ;)

[akupenguin]

Errors:
The graph labelled "On this picture Y-PSNR values are shown" is titled "V-PSNR". Which is it?
One of the entries in the graphs is labelled "AVC". All of the codecs (except DivX of course) are AVC; I assume you mean Mpegable?
You calculate PSNR as an arithmetic mean of per-frame PSNR. This is completely bogus. (Think: what does a single pure-black frame do to the average? Hint: infinity. And that's only an extreme case, anything involving an average of logarithms is bogus even if you don't actually have any perfect frames.) In a constant-quality encode it might be good enough, but this was CBR, so quality goes all over the place. The correct measure of global PSNR is log(mean(error)), not mean(log(error)).
... and there's no x264 :(. But I'm not particularly impressed by our CBR ratecontrol, so maybe that's a good thing.

[SeeMoreDigital]

I reckon this thread should be moved over to the "New A/V Formats - Codecs" section...


Cheers

[CruNcher]

DmitriyV2 i hope in the second round you'll compare with XviD too :)

[lazyn00b]

A minor quibble: "Ateme" should probably be changed to "Nero", unless you got this version of the codec directly from Ateme. My assumption (correct me if I'm wrong) is that Nero paid for this codec to be developed and for the right to distribute it under their name. As far as I know, there is no "Ateme" codec that can purchased or evaluated by the regular consumers reading your test.

[Sharktooth]

He tested the commandline encoder from Ateme and i suppose (but i cant really tell) he tested the high profile.

[Manao]

There are some strange things :
* foreman is 30 fps, not 15
* the black square helps codecs with a logo, and the bigger the logo, the bigger the help (!)
* there's no way for divx's psnr curve to be higher than ateme's, even at high bitrates, and certainly not on foreman ( and since you give per frame psnr for high bitrate on foreman, you can check that something is wrong ).

[lazyn00b]

Originally posted by Sharktooth
He tested the commandline encoder from Ateme and i suppose (but i cant really tell) he tested the high profile.

Thanks, I stand corrected, then. Please ignore my post above.

lazyn00b

[Manao]

i suppose (but i cant really tell) he tested the high profile.Hard to tell : they don't give the settings for each codec, nor the encoding speed.

[bobololo]

Originally posted by Manao
Hard to tell : they don't give the settings for each codec, nor the encoding speed.

They encoded using the default settings, so they only used main profile encoding.

[DmitriyV2]

Originally posted by lazyn00b
A minor quibble: "Ateme" should probably be changed to "Nero", unless you got this version of the codec directly from Ateme. As far as I know, there is no "Ateme" codec that can purchased or evaluated by the regular consumers reading your test.
Sure! This codec does no available for customers now (unfortunatelly). We got codec directly from Ateme.
Please see screenshot at the beginning of comparison! ;)
(We began to make screenshots after the same discuaaion with our previous test).

[DmitriyV2]

Originally posted by bobololo
They encoded using the default settings, so they only used main profile encoding.
Sure! We plan to use 2 presets provided by codec developers from next comparison.

[DmitriyV2]

Originally posted by akupenguin
... and there's no x264 :(. But I'm not particularly impressed by our CBR ratecontrol, so maybe that's a good thing. [/B]
Thanks for verification! We'l check it! Metric from next comparison will be published for free usage, hope you will test it as more, as possible! :)

x264 will be added to next comparison, so you have about 2 month to tune rate control. Hope you will improve it fundamentally! :)

[DmitriyV2]

Originally posted by CruNcher
DmitriyV2 i hope in the second round you'll compare with XviD too :)
Not to this (H.264) comparison.

We plan MPEG-4 codecs comparison with all version of DivX (from early to modern) and with XviD of cource! :)

[DmitriyV2]

Originally posted by Manao
There are some strange things :
* foreman is 30 fps, not 15
We have 15 fps foreman only. If you have really 30 fps (not 2 times faster moved) - please put it somewhere for me!
* the black square helps codecs with a logo, and the bigger the logo, the bigger the help (!)
Yes! It's not justly, that bigger logo a little increase metric. :)
We console oneself only that bigger logo _essentially_ decrease number of codec users! So this damage, I think, is more serious. ;)

And do not worry - our main course - work with codec developers directly, so we hope to clear away comparison from such codecs.

* there's no way for divx's psnr curve to be higher than ateme's, even at high bitrates, and certainly not on foreman ( and since you give per frame psnr for high bitrate on foreman, you can check that something is wrong).
We will put measure program for free for
Really we found more such observations about codecs optimization. :)

[CruNcher]

Maybe you missunderstood me i meant that
Main features:
* 6 H.264 codecs was compared with last DivX.
why did you compared vs DivX instead of XviD im not sure that this is really fair ?

[Sergey A. Sablin]

Originally posted by DmitriyV2
We have 15 fps foreman only. If you have really 30 fps (not 2 times faster moved) - please put it somewhere for me!
Hi Dmitriy!
If you try to play back yours foreman with 30 fps you will see that in this case it is more closely to reality than playing it with 15 fps ;)

[LordRPI]

Originally posted by CruNcher
Maybe you missunderstood me i meant that
Main features:
* 6 H.264 codecs was compared with last DivX.
why did you compared vs DivX instead of XviD im not sure that this is really fair ?

Not to mention the study used DivX 5.1.1, not DivX Fusion.

[stephanV]

DivX wasnt used as DivX as such, but as a MPEG4 ASP codec to compare with some H264 codecs, to see an improvement of H264 over MPEG4 ASP. DivX would suffice for that. It's not directly related to fairness.

If I interpret the conclusions of the test correctly, DivX ended up 2nd after Ateme so I doubt if using XviD would have really changed the results. Also, I hope that the possible difference between a H264 codec and a MPEG4 ASP codec is bigger than the difference between XviD and DivX right now, otherwise you could throw the whole standard in the trashcan right now IMO.

[peteag]

They said, that H.264 codecs are comparable to a DivX 2.0-status (when looking at 5.2.1) and must be improved by the years, they're not exhausted. But how's the ASP-standard exhausted? Is it realy full-featured like the reference-code or are there even tons of features to be implemented?

[Sergey A. Sablin]

Originally posted by peteag
They said, that H.264 codecs are comparable to a DivX 2.0-status (when looking at 5.2.1) and must be improved by the years, they're not exhausted. But how's the ASP-standard exhausted? Is it realy full-featured like the reference-code or are there even tons of features to be implemented?
There is no answer for this question cause there isn't even features with which encoders were tested and of course there isn't description of features each encoder has, so there is nothing to compare unfortunaly...

[Manao]

DivX doesn't implement GMC 3-points, and XviD doesn't implement interlaced motion compensation. That's the only two features missing for MPEG-4 ASP codecs. But quality isn't only about implemented features. VHQ in XviD doesn't add features, but brings a lot quality wise. In this regards, both codecs aren't yet finished ( and never will ).

For AVC codecs, some features are still missing ( weighted prediction / interlaced support in x264, 4x4 inter blocks in nero / ateme ), and using correctly all the features will take a lot of time, especially since encoding speed with h264 soon becomes an issue and that a lot of features means a slow encoding speed.

[DmitriyV2]

Originally posted by CruNcher
why did you compared vs DivX instead of XviD im not sure that this is really fair?
Again: DivX is more popular and good reference line for MPEG-4 AVC. We will test MPEG-4 ASP and SP codecs now. Please be ready! :)

[DmitriyV2]

Originally posted by Sergey A. Sablin
If you try to play back yours foreman with 30 fps you will see that in this case it is more closely to reality than playing it with 15 fps ;)
Nice to meet you here, Sergey! ;)

We received several equal notes about foreman, check sequence and will change 15 to 30 in next PDF!

[DmitriyV2]

Originally posted by stephanV
DivX wasnt used as DivX as such, but as a MPEG4 ASP codec to compare with some H264 codecs, to see an improvement of H264 over MPEG4 ASP. DivX would suffice for that. It's not directly related to fairness.
Absolutly!

If I interpret the conclusions of the test correctly, DivX ended up 2nd after Ateme so I doubt if using XviD would have really changed the results. Also, I hope that the possible difference between a H264 codec and a MPEG4 ASP codec is bigger than the difference between XviD and DivX right now, otherwise you could throw the whole standard in the trashcan right now IMO.
In next comparison slow "tuned" profile will be tested. It's clean, that H264 codecs does not reach extremum of codeing efficiency with it's format.