Invitation to 5th H.264/AVC codecs comparison

[DmitriyV2]

Dear all,

Sorry for delay, but we move forward with 5th H.264 codecs comparison.

Please forward this information to video codecs developers!

Moscow State University Graphics & Multimedia Laboratory starts next
5-th H.264 codecs comparison. There is some information about it
below. Let us know if any questions.

Main issues:

* COMPARISON WILL BE SOON! (Sorry for delay).

* We are planning to include new codecs that did not participate in
previous comparison by chosing presets for them ourself.
We are hope to include more codecs into this comparison.

===========================================================================
CALL FOR MPEG4-AVC/H.264 CODECS
Fifth H.264 video codec comparison
For practical researchers and developers in the field of high-end video compression
===========================================================================

Scope of Test
-------------

* Encoding time, speed/quality analysis
* Objective quality measurements (PSNR, SSIM, Average Advantage, etc.)
* Analysis of averaged objective results
* Leaders in different areas
* Special analysis of codec parts

Important Dates
---------------

February, 20 - Deadline for preliminary receipt of a H.264 codecs
February, 27 - Deadline for receipt of a H.264 codec with required presets
March, 16 - Deadline for settling technical problems with codec’s functioning
April, 7 - Draft version of report that will be sent to all participants
April, 14 - Deadline for reception of comments to the draft
April, 28 - Comparison report release


Enhancements in comparison to Previous H.264/AVC Comparison
-----------------------------------------------------------

* We are planning to include new codecs that did not participate in
previous comparison by chosing presets for them ourself. For that
task we will use option analysis.

Anyway we will be glad to have a direct contact with codec
developers. The main benefit of direct participation for developers
is receiving Pro version of comparison free of charge.

* Codecs options analysis (see example at Options Analysis of MPEG-4
AVC/H.264 Codec x264)
http://compression.ru/video/codec_co...sis_08_en.html

* New type of special analysis for codecs
* Using natural sequences' special modification
* Using synthetic sequences
* Separate analysis of codecs main subsystems

* New sequences


Developer Deliverables
----------------------

The following deliverables should be provided by each developer:
* Codec files (CLI executable file is preferable)
* Short description of codec parameters
* Codec's presets with mentioning what H.264/AVC profiles are used

The full text of Call for Codecs is available at
http://compression.ru/video/codec_co...codecs_09.html

Variants of Participation
-------------------------

There are two variants for companies to participate in our comparison:

1. Participation for free. All results of your codec will be published,
except special cases of measurements problems due to codec
instability.

2. Private participation. A special report will be prepared only for your
company. This report contains:
* Your codec results and all material from the free version
* Special additional analysis of your codec

If you are interested in the private participation, please contact us
for details.

Useful Links
------------

* Fourth Annual MSU MPEG-4 AVC/H.264 Video Codec Comparison
http://compression.ru/video/codec_co...4_2007_en.html

* Options Analysis of MPEG-4 AVC/H.264 Codec x264
http://compression.ru/video/codec_co...sis_08_en.html

* Subjective Comparison of Modern Video Codecs
http://compression.ru/video/codec_co...arison_en.html

Sincerely yours,
Dr. Vatolin

[Dark Shikari]

Let's just say that we have evil plans for our entry in this comparison

Also, your mail server is down and I can't email any of you (message returned with "message too large" for even small messages).

[Cyber-Mav]

Originally Posted by Dark Shikari

Let's just say that we have evil plans for our entry in this comparison

hmmm, threaded lookahead ??

[Chengbin]

Originally Posted by Dark Shikari

Let's just say that we have evil plans for our entry in this comparison

A secret weapon under development that is almost done? Nice!

[IgorC]

I wonder if there is any H.264 encoder comparable to last revision of x264 at high quality settings.
And don't see the reason why x264 is compared on opsnr and ssim when its psy uses more advanced internal metrics.

[Raptus]

Originally Posted by IgorC

And don't see the reason why x264 is compared on opsnr and ssim when its psy uses more advanced internal metrics.

True, but what do you suggest? Doing a decent subjective test for video that shall produce statistically relevant results is quite difficult. A university should have the means to do it, though...
ITU-R has recommendations for that in the BT Series http://www.itu.int/rec/R-REC-BT/e

[Dark Shikari]

One of the problems I'm finding with this comparison so far is that the "fast" preset is more like a "slow" preset and the "slow" preset is more like an "insanely slow" preset. If I can use subme9/trellis2 on the "fast" preset, something is wrong.

[G_M_C]

Originally Posted by Dark Shikari

Let's just say that we have evil plans for our entry in this comparison

Also, your mail server is down and I can't email any of you (message returned with "message too large" for even small messages).

Since the "competition" should be well underway by now ....

... how did the EVIL PLAN work out Dark Shikari ?

And will the code of that evil plan be usable by us ?
/me is curious as to what the evil plan entailed

[Dark Shikari]

Originally Posted by G_M_C

Since the "competition" should be well underway by now ....

... how did the EVIL PLAN work out Dark Shikari ?

And will the code of that evil plan be usable by us ?
/me is curious as to what the evil plan entailed

The code of the evil plan was (mostly) already committed

[G_M_C]

Originally Posted by Dark Shikari

The code of the evil plan was (mostly) already committed

Ahh, i see; Quite a patch, that was. I'm still enjoying those improvements

Originally Posted by Sharktooth

i hope MSU labs changed the test methods since, despite what Sagittaire says, metrics do not represent quality at all.

True. But i wonder how you'd objectively measure differences in encoders without using "hard numbers". Or did you have other ways of measuring in mind ?

[Audionut]

I'd like to do a subjective comparison. But I don't think I'd get enough participation!!

I could spam it at a few forums I guess to increase participation.

[Sharktooth]

i hope MSU labs changed the test methods since, despite what Sagittaire says, metrics do not represent quality at all.

[Sharktooth]

blind tests.

[Sagekilla]

G_M_C: How do you tell if one video looks better than another, if you had nothing but those two videos on hand? You'd watch them, and say which one looks better to you. Same idea applies here, only with more restrictions to prevent skewing from having extra knowledge (which video was encoded with what settings, etc).

[G_M_C]

Originally Posted by Sagekilla

G_M_C: How do you tell if one video looks better than another, if you had nothing but those two videos on hand? You'd watch them, and say which one looks better to you. Same idea applies here, only with more restrictions to prevent skewing from having extra knowledge (which video was encoded with what settings, etc).

A "double blind" (*)test/screening you mean ? But you still need a fair number of people to make the test statistically sigificant.

(*)[funny]However the "Blind" part of it doesnt seem appropriate [/funny]

[Sagekilla]

No, doesn't necessarily have to be double blind. Yes, it is helpful to have the extra degree but you can get away with the testers knowing which video is which.

[Sharktooth]

blind test is the way to go for testing quality, since as i've said, metrics are juts... metrics. that means they measure differencies between samples weigthing them in some way. but the fact is, a better metric doesnt represent better visual quality.
lets make an example:
sample A: high quality picture
sample B: mid quality picture
sample C: low quality picture

all pictures represent the same image... just with different quality.
picture B is our source for comparation.

results for metrics>
picture A: low metric.
picture B: highest metric.
picture C: low metric.
conclusion: picture B has the highest metric. picture A even if it is the highest quality picture of the pack, is rated "low" coz it differs from picture B that is our source for comparation and has a low metric...

results for human eye>
picture A: highest quality
picture B: mid quality
picture C: worst quality
conclusion: if you look at the pictures with your eyes you will have no doubt that A is the best one... while metrics are telling you something else...

so, all in all, this is the demonstration that metrics do not represent quality and you must never trust them if you compare different encoders.
that's also the reason why elecard has higher psnr than x264 but x264 produces a much higher visual quality (and some "smart" people produce docs with graphs but without any kind of visual comparison...).

[Sagittaire]

Blind test don't work for video simply it's really hard to evaluate overall quality for long sequence (simple example is VBR vs CBR for the same codec). In practice blind test for video are generally less accurate than metric. It's like that. Speak about that with developper ... they don't trust generaly blind test for video. I have never see even here on doom9 really reliable blind test ...

[Sagittaire]

that's also the reason why elecard has higher psnr than x264 but x264 produces a much higher visual quality (and some "smart" people produce docs with graphs but without any kind of visual comparison...).

It's simply false ... for all affirmation ... lol

1) x264 without psy (AQ and SSD) produce better metric than Mainconcet SDK without psy (AQ and FGO) and with relative large margin. x264 is in fact the best in area for metric test ... ;-)
x264 use simply psy tools by default and not Mainconcept.

2) Some people even here on doom9 forum find that Mainconcept SDK produce better visual result than x264. The principal particulary for HVS is it's ... subjective ... and by definition you can't contradict that. In fact IMO x264 and Mainconcept SDK produce in most case comparable visual result and IMO you can notice real difference only for really particular sequences at really low quality encoding.

3) graph and metric are usefull for particular test like speed test simply because you must have really reliable quality reference. Make speed test with subjective comparison and without metric test is simply impossible. No way. It's like that. No possible discution here. Final point.

[Sharktooth]

Originally Posted by Sagittaire

Blind test don't work for video simply it's really hard to evaluate overall quality for long sequence (simple example is VBR vs CBR for the same codec). In practice blind test for video are generally less accurate than metric. It's like that. Speak about that with developper ... they don't trust generaly blind test for video. I have never see even here on doom9 really reliable blind test ...

in practice blind tests made lame the best mp3 encoder out there.
that is the proof blind tests work for encoders development.

Originally Posted by Sagittaire

It's simply false ... for all affirmation ... lol

1) x264 without psy (AQ and SSD) produce better metric than Mainconcet SDK without psy (AQ and FGO) and with relative large margin. x264 is in fact the best in area for metric test ... ;-)
x264 use simply psy tools by default and not Mainconcept.

2) Some people even here on doom9 forum find that Mainconcept SDK produce better visual result than x264. The principal particulary for HVS is it's ... subjective ... and by definition you can't contradict that. In fact IMO x264 and Mainconcept SDK produce in most case comparable visual result and IMO you can notice real difference only for really particular sequences at really low quality encoding.

3) graph and metric are usefull for particular test like speed test simply because you must have really reliable quality reference. Make speed test with subjective comparison and without metric test is simply impossible. No way. It's like that. No possible discution here. Final point.

1) there are no doubts x264 - if we talk about quality - is better than mainconcept h.264 encoder

2) psy opts are hardly subjective. if you have a source with grain, you expect the encoder to keep the grain... otherwise you filter it out before encoding. another point is artifacts. you dont want them if they're not on the source

3) i disagree since, as i said, metrics do not represent quality in any way.