Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 18th February 2009, 00:24   #1  |  Link
DmitriyV2
MSU G&M Lab
 
DmitriyV2's Avatar
 
Join Date: Oct 2004
Location: MSU G&M Lab
Posts: 318
Invitation to 5th H.264/AVC codecs comparison

Dear all,

Sorry for delay, but we move forward with 5th H.264 codecs comparison.

Please forward this information to video codecs developers!

Moscow State University Graphics & Multimedia Laboratory starts next
5-th H.264 codecs comparison. There is some information about it
below. Let us know if any questions.

Main issues:

* COMPARISON WILL BE SOON! (Sorry for delay).

* We are planning to include new codecs that did not participate in
previous comparison by chosing presets for them ourself.
We are hope to include more codecs into this comparison.

===========================================================================
CALL FOR MPEG4-AVC/H.264 CODECS
Fifth H.264 video codec comparison
For practical researchers and developers in the field of high-end video compression

===========================================================================

Scope of Test
-------------


* Encoding time, speed/quality analysis
* Objective quality measurements (PSNR, SSIM, Average Advantage, etc.)
* Analysis of averaged objective results
* Leaders in different areas
* Special analysis of codec parts

Important Dates
---------------

February, 20 - Deadline for preliminary receipt of a H.264 codecs
February, 27 - Deadline for receipt of a H.264 codec with required presets
March, 16 - Deadline for settling technical problems with codecís functioning
April, 7 - Draft version of report that will be sent to all participants
April, 14 - Deadline for reception of comments to the draft
April, 28 - Comparison report release


Enhancements in comparison to Previous H.264/AVC Comparison
-----------------------------------------------------------

* We are planning to include new codecs that did not participate in
previous comparison by chosing presets for them ourself. For that
task we will use option analysis.

Anyway we will be glad to have a direct contact with codec
developers. The main benefit of direct participation for developers
is receiving Pro version of comparison free of charge.

* Codecs options analysis (see example at Options Analysis of MPEG-4
AVC/H.264 Codec x264)
http://compression.ru/video/codec_co...sis_08_en.html

* New type of special analysis for codecs
* Using natural sequences' special modification
* Using synthetic sequences
* Separate analysis of codecs main subsystems

* New sequences


Developer Deliverables
----------------------

The following deliverables should be provided by each developer:
* Codec files (CLI executable file is preferable)
* Short description of codec parameters
* Codec's presets with mentioning what H.264/AVC profiles are used

The full text of Call for Codecs is available at
http://compression.ru/video/codec_co...codecs_09.html

Variants of Participation
-------------------------

There are two variants for companies to participate in our comparison:

1. Participation for free. All results of your codec will be published,
except special cases of measurements problems due to codec
instability.

2. Private participation. A special report will be prepared only for your
company. This report contains:
* Your codec results and all material from the free version
* Special additional analysis of your codec

If you are interested in the private participation, please contact us
for details.

Useful Links
------------


* Fourth Annual MSU MPEG-4 AVC/H.264 Video Codec Comparison
http://compression.ru/video/codec_co...4_2007_en.html

* Options Analysis of MPEG-4 AVC/H.264 Codec x264
http://compression.ru/video/codec_co...sis_08_en.html

* Subjective Comparison of Modern Video Codecs
http://compression.ru/video/codec_co...arison_en.html

Sincerely yours,
Dr. Vatolin
__________________
With regards
Dmitriy Vatolin
www.compression.ru/video/ (Last updates: MSU Video Quality Metric 2.5 - faster&more useful)

Last edited by DmitriyV2; 18th February 2009 at 00:27.
DmitriyV2 is offline   Reply With Quote
Old 18th February 2009, 01:47   #2  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,690
Let's just say that we have evil plans for our entry in this comparison

Also, your mail server is down and I can't email any of you (message returned with "message too large" for even small messages).

Last edited by Dark Shikari; 18th February 2009 at 01:51.
Dark Shikari is offline   Reply With Quote
Old 19th February 2009, 01:56   #3  |  Link
Cyber-Mav
Registered User
 
Join Date: Dec 2005
Posts: 244
Quote:
Originally Posted by Dark Shikari View Post
Let's just say that we have evil plans for our entry in this comparison
hmmm, threaded lookahead ??
Cyber-Mav is offline   Reply With Quote
Old 19th February 2009, 02:49   #4  |  Link
Chengbin
Registered User
 
Join Date: Oct 2007
Posts: 1,059
Quote:
Originally Posted by Dark Shikari View Post
Let's just say that we have evil plans for our entry in this comparison
A secret weapon under development that is almost done? Nice!
Chengbin is offline   Reply With Quote
Old 19th February 2009, 06:38   #5  |  Link
IgorC
Registered User
 
Join Date: Apr 2004
Posts: 1,246
I wonder if there is any H.264 encoder comparable to last revision of x264 at high quality settings.
And don't see the reason why x264 is compared on opsnr and ssim when its psy uses more advanced internal metrics.
IgorC is offline   Reply With Quote
Old 19th February 2009, 11:44   #6  |  Link
Raptus
heretic nuB
 
Join Date: Sep 2003
Posts: 71
Quote:
Originally Posted by IgorC View Post
And don't see the reason why x264 is compared on opsnr and ssim when its psy uses more advanced internal metrics.
True, but what do you suggest? Doing a decent subjective test for video that shall produce statistically relevant results is quite difficult. A university should have the means to do it, though...
ITU-R has recommendations for that in the BT Series http://www.itu.int/rec/R-REC-BT/e

Last edited by Raptus; 19th February 2009 at 11:50.
Raptus is offline   Reply With Quote
Old 19th February 2009, 12:07   #7  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,690
One of the problems I'm finding with this comparison so far is that the "fast" preset is more like a "slow" preset and the "slow" preset is more like an "insanely slow" preset. If I can use subme9/trellis2 on the "fast" preset, something is wrong.
Dark Shikari is offline   Reply With Quote
Old 19th February 2009, 13:55   #8  |  Link
Audionut
Registered User
 
Join Date: Nov 2003
Posts: 1,254
I'd like to do a subjective comparison. But I don't think I'd get enough participation!!

I could spam it at a few forums I guess to increase participation.
__________________
http://www.7-zip.org/
Audionut is offline   Reply With Quote
Old 19th February 2009, 13:57   #9  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,415
Quote:
Originally Posted by Dark Shikari View Post
One of the problems I'm finding with this comparison so far is that the "fast" preset is more like a "slow" preset and the "slow" preset is more like an "insanely slow" preset. If I can use subme9/trellis2 on the "fast" preset, something is wrong.
"slow" and "fast" are sujective if you don't have quality reference. For example ATI encoder is certainely really fast but produce insanely bad quality.


Quote:
I wonder if there is any H.264 encoder comparable to last revision of x264 at high quality settings.
And don't see the reason why x264 is compared on opsnr and ssim when its psy uses more advanced internal metrics.
It's easy to desactivate psy option like AQ and SSD ...


Quote:
Originally Posted by Audionut View Post
I'd like to do a subjective comparison. But I don't think I'd get enough participation!!

I could spam it at a few forums I guess to increase participation.
Expenrience from dev show that subjective comparison produce not really good result.
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9

Last edited by Sagittaire; 19th February 2009 at 14:00.
Sagittaire is offline   Reply With Quote
Old 19th February 2009, 14:06   #10  |  Link
Audionut
Registered User
 
Join Date: Nov 2003
Posts: 1,254
Quote:
Originally Posted by Sagittaire View Post
Expenrience from dev show that subjective comparison produce not really good result.
How can it produce not a good result?


Quote:
Originally Posted by Dark Shikari on Mailing list View Post
Since x264's recent psy optimizations have proven quite definitively that one can achieve greatly improved visual quality while decreasing both PSNR and SSIM, will their be a subjective comparison as well as an objective one? In addition, what about the fact that encoders have different settings optimized for PSNR, SSIM, and visual quality,
respectively?
__________________
http://www.7-zip.org/
Audionut is offline   Reply With Quote
Old 19th February 2009, 15:06   #11  |  Link
Chengbin
Registered User
 
Join Date: Oct 2007
Posts: 1,059
Quote:
Originally Posted by Dark Shikari View Post
One of the problems I'm finding with this comparison so far is that the "fast" preset is more like a "slow" preset and the "slow" preset is more like an "insanely slow" preset. If I can use subme9/trellis2 on the "fast" preset, something is wrong.
LOL! I think x264 is fast enough on that machine to use any insanely slow settings and still qualify for the fps requirement. This comparison will be VERY interesting.

I was reading the 2006 video codec comparison, and x264, at r544, fared up very well. This comparison could be very funny, as x264 will completely own the other codecs.

It is too bad that they don't have an i7 rig to test these codecs. x264 will have even more of an advantage.
Chengbin is offline   Reply With Quote
Old 19th February 2009, 15:47   #12  |  Link
Raptus
heretic nuB
 
Join Date: Sep 2003
Posts: 71
Quote:
Originally Posted by Audionut View Post
How can it produce not a good result?
He is referring to the quality/accuracy of the conclusion of those tests, not how well x264 might do in it.

I was just reading through the old test papers and found quite a few interesting bits. For instance, i'd very much like to get a current version of this paper. Doesn't cover all settings and isn't based on subjective metric but gives you a good overview of the speed/quality impact of different settings.
Raptus is offline   Reply With Quote
Old 19th February 2009, 17:40   #13  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,415
This test is really impressive ...

Here my little (and old) comparison with similar protocole ...



Anyway for this old test aku say that i8x8,i4x4,p8x8 is the best for quality/speed
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9
Sagittaire is offline   Reply With Quote
Old 19th February 2009, 18:15   #14  |  Link
IgorC
Registered User
 
Join Date: Apr 2004
Posts: 1,246
No ssim no psnr are required. They are obsolete.
I thought it was already time to think different.
IgorC is offline   Reply With Quote
Old 19th February 2009, 20:06   #15  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,415
Quote:
Originally Posted by IgorC View Post
No ssim no psnr are required. They are obsolete.
I thought it was already time to think different.
Unfortunaly there are not other way for make objective quality test or speed test.
Moreover SSIM and PSNR work very well if you work with high delta.
Moreover psy optimisation don't always work very well for all source or ... for all eyes.
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9

Last edited by Sagittaire; 19th February 2009 at 20:09.
Sagittaire is offline   Reply With Quote
Old 30th March 2009, 12:50   #16  |  Link
G_M_C
Registered User
 
Join Date: Feb 2006
Posts: 1,076
Quote:
Originally Posted by Dark Shikari View Post
Let's just say that we have evil plans for our entry in this comparison

Also, your mail server is down and I can't email any of you (message returned with "message too large" for even small messages).
Since the "competition" should be well underway by now ....

... how did the EVIL PLAN work out Dark Shikari ?

And will the code of that evil plan be usable by us ?
/me is curious as to what the evil plan entailed
G_M_C is offline   Reply With Quote
Old 30th March 2009, 13:05   #17  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,690
Quote:
Originally Posted by G_M_C View Post
Since the "competition" should be well underway by now ....

... how did the EVIL PLAN work out Dark Shikari ?

And will the code of that evil plan be usable by us ?
/me is curious as to what the evil plan entailed
The code of the evil plan was (mostly) already committed
Dark Shikari is offline   Reply With Quote
Old 30th March 2009, 14:32   #18  |  Link
Sharktooth
Mr. Sandman
 
Sharktooth's Avatar
 
Join Date: Sep 2003
Location: Haddonfield, IL
Posts: 11,768
i hope MSU labs changed the test methods since, despite what Sagittaire says, metrics do not represent quality at all.
Sharktooth is offline   Reply With Quote
Old 30th March 2009, 14:59   #19  |  Link
G_M_C
Registered User
 
Join Date: Feb 2006
Posts: 1,076
Quote:
Originally Posted by Dark Shikari View Post
The code of the evil plan was (mostly) already committed
Ahh, i see; Quite a patch, that was. I'm still enjoying those improvements

Quote:
Originally Posted by Sharktooth View Post
i hope MSU labs changed the test methods since, despite what Sagittaire says, metrics do not represent quality at all.
True. But i wonder how you'd objectively measure differences in encoders without using "hard numbers". Or did you have other ways of measuring in mind ?
G_M_C is offline   Reply With Quote
Old 30th March 2009, 15:04   #20  |  Link
Sharktooth
Mr. Sandman
 
Sharktooth's Avatar
 
Join Date: Sep 2003
Location: Haddonfield, IL
Posts: 11,768
blind tests.
Sharktooth is offline   Reply With Quote
Reply

Tags
codecs, comparison, h.264/avc

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 02:32.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2017, vBulletin Solutions Inc.