Call for MPEG4-AVC/H.264 codecs!

[DmitriyV2]

We will make new formal comparison of H.264/AVC with more metrics and more options. Looks like for last year from our previous comparison situation changed seriously, so we will measure this changes and current stage.

Please transmit this information to codec developers!

---------------------------------

Call for MPEG4-AVC/H.264 codecs
Second Annual H.264 video codec comparison
For people, who make real research in field of high-end video compression


Important Dates

# September, 20 – deadline of codec receipt with required presets
# September, 25 – notification of codecs acceptance for tests notification

Enhancements from Previous H.264/AVC Comparison

# Measuring will be performed with extended metrics set: PSNR, SSIM VQM
# Two codec presets will be compared:“Maximum quality” & "Maximum speed"
# For each preset both speed and quality will be measured

Codec Requirements

# Codec should be H.264/AVC Main Profile codec
# Codec should allow to set arbitrary bitrate of resulted sequence
# 3 variants of codec interface are possible:

Console codec version (with batch processing support — bitrate and file names must be assigned from command line
Video for Windows Codec with correct state saving (batch processing support)
Direct Show filter. In this case software for batch processing should be provided

# Codec should open and save *.yuv or *.avi (YUV colorspace) files
# Result video sequences should be opened with standard methods

Comparison Rules


All measurements will be produced by MSU Video Quality Measurement Tool (http://www.compression.ru/video/quality_measure/video_measurement_tool_en.html)
Codec options for “Maximum quality” and “Maximum speed” presets should be provided by codec authors. If these options won’tnot be provided, default options will be used
Verification of comparison results is possible for codec authors before comparison publication
Codec authors can delete all information about their codec from public comparison document.

See Previous H.264/AVC comparison for other rules and details.


Useful Links


Previous H.264/AVC comparison (http://www.compression.ru/video/codec_comparison/mpeg-4_avc_h264_en.html)
MPEG-4 SP/ASP comparison (http://www.compression.ru/video/codec_comparison/mpeg-4_en.html)
Lossless codecs comparison (http://www.compression.ru/video/codec_comparison/lossless_codecs_en.html)

[bond]

first of all great that you do this! :)

but why do you require the main profile? this defacto punishes better codecs supporting the high profile already, thats not fair

[DmitriyV2]

first of all great that you do this! :)
Thank you! :)

but why do you require the main profile? this defacto punishes better codecs supporting the high profile already, thats not fair
Most of codecs are main profile only, so we use it for correct comparison results. Maybe it's good idea to compare high profile additionaly to main profile comparison.

[Revgen]

...Codec should be H.264/AVC Main Profile codec...


You're not going to test High Profile codecs?

[shlezman]

Why Main profile, I think a minor category of baseline might be a good idea. There's a great intrest in the baseline profile especially from the R/T industry.

[DmitriyV2]

Why Main profile, I think a minor category of baseline might be a good idea. There's a great intrest in the baseline profile especially from the R/T industry.
More measurements - more time for tesing, more pages in report. There is no sponsorship of this tests, so we measure what is interesting for our laboratory video group first of all. ;) But we plan to make this tests annual (this is second one) to track changes in AVC codecs area with the same procedure, and this year we extend numped of tests. So it's great likelihood, that we will measure Base profile also later.

[bond]

actually i would say that limiting to main profile punishes defacto x264, as its the only publically available avc high profile encoder (except the reference). i think removing the limitation might get some codec devs, like ateme, who work on high profile to let their high profile encoder pretested which could get very interesting

the codecs should be tested according to the maximum quality they can reach and not according to some arbitrary profile...

if you would do the same for mpeg-4 part2 you would need to test mpeg-4 simple profile, as its far more widely supported than asp (+ maybe b-frames), defacto punishing the good codecs...

[DmitriyV2]

actually i would say that limiting to main profile punishes defacto x264, as its the only publically available avc high profile encoder (except the reference). i think removing the limitation might get some codec devs, like ateme, who work on high profile to let their high profile encoder pretested which could get very interesting
Ok, if x264 developers will help us, I think - it's possible.

Also we try to get from JVT 2 sets of options for recent JM reference software, but it looks difficult... Who is enouch familiar with it here to create such files (at least initial for discussion)?

the codecs should be tested according to the maximum quality they can reach and not according to some arbitrary profile...
It's resonable to test codecs in one category.

if you would do the same for mpeg-4 part2 you would need to test mpeg-4 simple profile, as its far more widely supported than asp (+ maybe b-frames), defacto punishing the good codecs...
See our "MPEG-4 SP/ASP video codecs comparison tests" on http://www.compression.ru/video/codec_comparison/mpeg-4_en.html
For MPEG we did not set up 2 categories (not so actual now, modern hardware support ASP as well as SP).

[bond]

Ok, if x264 developers will help us, I think - it's possible.well its pretty easy, simply enable --8x8dct and i8x8 ;)

Also we try to get from JVT 2 sets of options for recent JM reference software, but it looks difficult... Who is enouch familiar with it here to create such files (at least initial for discussion)? its pretty easy to enable b-frames, cabac, loop and the other important things, what exactly do you need to know?

[DmitriyV2]

well its pretty easy, simply enable --8x8dct and i8x8 ;)
its pretty easy to enable b-frames, cabac, loop and the other important things, what exactly do you need to know?
With high profile enaibling in x264 - ok! :) But other options need to be approved.
With JM - we want the same that we want for all codecs - we want to receive option set (of course - we can select options, whatever the case the situation when we get options from developers it's better). This is more correct position, especially for speed optimization testing.

[bond]

its indeed the best to ask the devs

[DmitriyV2]

its indeed the best to ask the devsSure! We start very useful discussion with x264 developer, who will support this comparison.

[superdump]

Sure! We start very useful discussion with x264 developer, who will support this comparison.With whom are you discussing?

[IgorC]

@Dmitriy
Your comparison of ASP codecs. It was dated by 1 march of 2005. In conclusion of that comparison it says Divx5 is clear winner.

1. It's not clear winner http://multimediacom.free.fr/HTM/32_Test_Codec_Video.htm
2. During time when you was comparing Xvid 1.0.3 with others ASP codecs. Xvid 1.1 beta 1 already was enable. Your comparison went out too late.
3. It's not clear what settings you used for each codec(it's TOO esential.). What purpose of comparison : best quality or trade on quality/speed.

Counting all these issues of your comparison I can't say that it was a professional job or there wasn't enough time to do it.

Nothing personal but it will be nice if all these problems will be avoid in AVC comparison. :)

[Sagittaire]

Benchmarks
DARC performed a compression test for the DivX Pro Fusion codec against a suite of 7 clips and plotted the average bit spend against PSNR for several fixed-quantizer encodings. In the chart below, the bitrate for each encoding mode is plotted as a percentage referenced against XVID (Motion 6 VHQ 4 Trellis Off), an independently developed open-source codec. Lower lines represent better compression.

http://labs.divx.com/archives/TESTFusion.gif


DivX Networks say "XviD (stable version I think) is very better than DivX 5.2.1". DivX Networks say "DivX Fusion (alias DivX6) is very very better than DivX 5.2.1" ...

[DmitriyV2]

@superdump

Alex Izvorski

@IgorC

1. We did not love "average" mark. On some sequences one codec is better, on some - another, on low bitrates on one sequence - one, on high - another and etc. So our mission - provide measurement results for those, who can interpret PSNR charts. :)
And that's why professionals send us good reviews (http://compression.ru/video/codec_comparison/h_264_comments_en.html).
Also see answer on question: (http://compression.ru/video/codec_comparison/codecs_faq_en.html)
Q: It's all clear to me, you've described all the codecs in details, thanks a lot! But which codec is the best after all?

2. Date of comaprison - is date of report preparation. New version was released later than we make measurements.
In current 264 comparison some codecs again will be ever unreleased. For example new codecs:
* ATI H.264 unavailable now
* VSS Pro H.264 also unavailable now (only ordinary version)

3. We use default (after installation on clean system) codec setting in MPEG-4 comparison and for AVC/H.264 will use settings provided by codec developers.

[bond]

@superdump

Alex Izvorski well he isnt really a main dev of x264

you should definitely ask akupenguin aka pengvado aka loren meritt, who is THE main x264 dev

[DmitriyV2]

Current list of codecs, declared to comparison:

1. VSS H.264
2. Frankhofer H.264
3. Ateme H.264
4. ATI H.264
5. Elecard H.264
6. x264
7* DivX 6

Please transfer this information to codecs developers!

[Kostarum Rex Persia]

DmitriyV2,please,can you tell me version of each codec.Do you use newest release of the codecs,or not.

About VSS 3.0 Pro is unavailable for the mortals,at the moment.But,on site www.vsofts.com/h264 you can ask their manager to borow you a professional version of 2.2 codec,rather than use 2.3 consumer version.

Then comes Ateme.What version,do you mean Beta 2,or older version.

DmitriyV2,please,use newest revision 293 of x264 codec in your tests.

[DmitriyV2]

2Kostarum Rex Persia

ALL 264 codecs received from there developers, so we will measure version thet they prefer with parameters, they think better.

Thank you for your care!

[stephanV]

Eeerrr? Why DivX6? as comparison with MPEG4 ASP?

[DmitriyV2]

Why DivX6? as comparison with MPEG4 ASP?
Absolutely!

[ggab]

Important Dates
# September, 20 – deadline of codec receipt with required presets
# September, 25 – notification of codecs acceptance for tests notification
could i ask: How is it going?

thanks

[DmitriyV2]

could i ask: How is it going?
Well, thanks! 7 codecs, Ateme, ATI and others.
Public results will be at the end of October.

[Revgen]

Well, thanks! 7 codecs, Ateme, ATI and others.
Public results will be at the end of October.

Wow! :eek:

It's only 7 codecs. You must be doing a lot of tests.

[DmitriyV2]

Wow! :eek: It's only 7 codecs. You must be doing a lot of tests.
Approximatly 3 times more tests, then during last year testing. And longer (mostly due to speed tests).

[bobololo]

Approximatly 3 times more tests, then during last year testing. And longer (mostly due to speed tests).

And I'm afraid that the speed test in its current approach won't provide significant results. As pengvado said, with your rules, a blank coder would win since quality isn't taken into account in the results even if some informative metrics are given.

Beside, the rules are getting quite annoying to us. You prevent 2 passes encoding which is the mode that provides for every codec the best result (except for those who don't have it, but this is their loss). And also, while you will use both objective and subjective metrics (which are conceptually opposed), you prevent us from providing a specific setting for each of them whereas our encoder features psychovisual enhanchements.

I don't know how other codecs devs reacted about your rules, but we really feel like deprived of our best features :(

[Kostarum Rex Persia]

Yes,bobololo,you are apsolutely right.His rules is rather strange.

[LRN]

DmitriyV2 , so, is there any progress? Уже почти ноябрь :)

[DmitriyV2]

Yes,bobololo,you are apsolutely right.His rules is rather strange.
You are welcome with your own comparison. :)

[DmitriyV2]

DmitriyV2 , so, is there any progress? Уже почти ноябрь :)
:) Do not worry! (Все под контролем! ;))

Due to trips I did not report last situation.

Main news: start testing with 8 codecs. 7 H.264/AVC codecs and there
presets was received from codecs manufactures. List of testing
codecs:
0. DivX 6.0 as MPEG-4 ASP reference (very good PSNR optimization
in last version with compatible format - we check this fact)
1. x264 revision 293
2. ATI H.264 3.1.2
3. Arcsoft H.264 (current dev. version)
4. VSS H.264 3.0.2.7
5. Elecard H.264 (current dev. version)
6. Ateme H.264 1.2.1.6
7. Franhofer H.264 build 20.09.2005

Tested profile: main
Number on sequences: 7 (foreman, susi, bbc and others)

Tested presets:
1. Maximum speed
2. Maximum quality
Both presets - one pass. Also on one sequence we compare one pass and
2 pass modes with "Maximum quality".

Measured metrics: Y-PSNR, U-PSNR, V-PSNR, L-PSNR, R-PSNR, G-PSNR,
B-PSNR, Y-SSIM, Y-VQM, Y-BLUR, U-BLUR, V-BLUR, Y-BLOCK, U-BLOCK,
V-BLOCK (all metrics are from publicly available MSU Video Quality
Measurement Tool)

Tested bitrates:
1. 100
2. 225
3. 340
4. 460
5. 700
6. 938
7. 1140
8. 1340
9. 1840
10. 2340

Due to big number of sequences (7), codecs (8), presets (2) and
bitrates (10) current total pure compression time - 208 hours (8.5
days). Also we use up to 10 repeated measurements to measure time.

A little bit smaller time is necessary for decompression.

And approximately the same time is necessary for final metrics
calculation (due to big number of metrics).

So total pure calculation time is about 24 days(!) on P4-2400.

Real time is bigger. For example twice we found serious bugs in
codecs, report bugs, received new version and repeated all
measurements (compression, decompression, measure, all bitrates,
presets, sequences) for updated codecs.

We use 3 fully equal computers (processor, motherboard, memory, disk)
to speed up measurements process.

In a few days (we wait for one developer answer) we finalize all test
and will prepare report (initially for developers).

Public report will be posted in 1-2 months approximately (previous our H.264 comparison was measured in October and officially published in
January).

We can send draft report early (before publication) during
verification period in exchange to qualitative feedback (please send personal
requests).

[bond]

are you only testing these small clips?
do i understand it right that you are mainly testing 1pass?
you still have the limitation to main profile?

if this is true, sorry, imho your test setup totally punishes good codecs

[baer999]

Great, I am very interested in the developing of the new video codec era...

[Revgen]

are you only testing these small clips?
do i understand it right that you are mainly testing 1pass?
you still have the limitation to main profile?

if this is true, sorry, imho your test setup totally punishes good codecs

As far as I know (correct me if I'm wrong), but at the time codecs were supposed to be submitted (late september), X264 was the only publicly available codec that was capable of encoding in High Profile. Unless you count the experimental Ateme beta that's floating around. IMHO X264 wins the quality comparison by default becuase of this. Why bother testing in high profile?


However, you do have a point about 2-pass. Some codecs might perform better than others in 2-pass and would be unfairly represented.

[IgorC]

Main Concept also is HP H.264. Ateme provided it's HP version for this test.
I wonder where is QT7 H.264? It's quite good and popular.

[Revgen]

Main Concept also is HP H.264. Ateme provided it's HP version for this test.
I wonder where is QT7 H.264? It's quite good and popular.

Manufacturers have to submit their codecs to be benchmarked. They obviously didn't.

In light of what you just mentioned, I'd now like to see a comparison between the HP profiles of X264 and Ateme.

[IgorC]

Try search on forum . You will find comparing test between Ateme H.264 , x264.
Ateme H.264 was better. I also have tried Main Concept HP H.264 and x264 HP. They were on pair and maybe it seemed to me that MC was slightly better.So it's diffcult to say what be better x264,Ateme or MC.

[bond]

only because you support a few high profile features doesnt mean automatically that you are better than every main profile codec...

[Kostarum Rex Persia]

I think that DmitriyV2 have to compare H.264 codecs in much fair way.He didn't answer us why his team don't test High Profile.

I am afraid that final result of the testing wouldn't be fair enough.

[Revgen]

Try search on forum . You will find comparing test between Ateme H.264 , x264.
Ateme H.264 was better. I also have tried Main Concept HP H.264 and x264 HP. They were on pair and maybe it seemed to me that MC was slightly better.So it's diffcult to say what be better x264,Ateme or MC.

The comparisons between x264 and ateme that I've been able to find were done with older versions (about .26x to .27x) instead of the .293 version that Dmitry is using.

They are interesting, but I'd like to see some results that are more recent. I'd do it myself if I knew how to ahold of the Ateme beta. It seems that there is some registration process that I have to do, but unfortunately I don't know what it is.

[Revgen]

only because you support a few high profile features doesnt mean automatically that you are better than every main profile codec...

I assume your replying to me. :)

I was under the assumption that X264 was the only HP codec that Dmitry had available for him to test. Doing tests in HP wouldn't have been worthwile if there was no other HP codec to compare too.

I'm now aware that the Ateme codec that they are using supports HP as well.

I wasn't trying to imply that in any way that High Profile codecs are inherently better than their Main Profile counterparts.

[DmitriyV2]

are you only testing these small clips?
do i understand it right that you are mainly testing 1pass?
you still have the limitation to main profile?
if this is true, sorry, imho your test setup totally punishes good codecs
We add 1000 frames clip. We test also High Profile, if it's implemented. Also we prepare special comparison of 1- and 2-pass modes (on one chart to compare there results for different codecs). Please note: not all codecs support 2-pass mode, so base comparison mode is 1-pass.

[DmitriyV2]

Ateme provided it's HP version for this test..
Sure.

I wonder where is QT7 H.264? It's quite good and popular.
We try to contact with developers with several ways without success, unfortunately.

We ask for developers support due to 2 reasons:
1) (main) We prefer to use parameters received from developers (such situation is more correct).
2) Now all tested codecs was command-line and this fact was seriously decrease technical complexity of mass testing.

If anybody will help us to find a contact with Apple developers - this will be very good (we already plan next 3-rd comparison).

[DmitriyV2]

Manufacturers have to submit their codecs to be benchmarked. They obviously didn't.
Sure.
In light of what you just mentioned, I'd now like to see a comparison between the HP profiles of X264 and Ateme.
Yes, we will compare HP versions.

[DmitriyV2]

Great, I am very interested in the developing of the new video codec era...
Thank you! :)
I think that DmitriyV2 have to compare H.264 codecs in much fair way.He didn't answer us why his team don't test High Profile.
Do not worry, we will test it. :) Only few codecs support HP, so main comparison will be for Main profile, but we will also compare HP with MP.

[LordRPI]

If anybody will help us to find a contact with Apple developers - this will be very good (we already plan next 3-rd comparison).

A good first step on contacting the Apple developers would be to join the mailing list and post a message regarding this comparison. You can join the QuickTime-API mailing list here: http://lists.apple.com/mailman/listinfo/quicktime-api

[DmitriyV2]

A good first step on contacting the Apple developers would be to join the mailing list and post a message regarding this comparison. You can join the QuickTime-API mailing list here: http://lists.apple.com/mailman/listinfo/quicktime-api
Looks like really good starting poing for contact. Thanks! Hope next comparison will be with Apple H.264.