Why dont we have a test for video codecs at different bitrates ?

[St Devious]

Like the guys at Hydrogenaudio forums have tests for audio at different bitrates , is it possible to conduct similar tests here for different video codecs and different video content at different bitrates .

[Dark Shikari]

Its been done...

Basically the unsurprising conclusion is twofold.

a) In terms of raw quality, H.264 > VC-1 > MPEG-4 ASP (Xvid/DivX) > MPEG-2
b) The higher the bitrate, the lower the difference between the codecs.

For example, if you encode a 720p film at 500kbps, the H.264 might look bearable if its denoised enough, but VC-1 will probably look worse at 1 megabit than H.264 at 500kbps.

On the other hand, at 5 megabit or 10 megabit for the same film, you might not even be able to tell the difference between the two codecs because the advantage of H.264 is very much dulled at such a bitrate.

If you add other codecs to the mix:

H.264 > VP7 > Snow > VC-1 > MPEG-4 ASP (Xvid/DivX) > MPEG-2 (there are plenty more that I don't feel like sorting ATM)

[shon3i]

IMHO MPEG4-ASP always done better job than VC-1, maybe i doing something wrong but always when i tryed VC-1 a im very disapointed with quality of encode

[Soulhunter]

IMHO its like...

http://img179.imageshack.us/img179/2319/4776546mb7.png


Bye

[Sharktooth]

not at all... at high bitrate they all tend to converge except h.264 than can go a step further and be lossless.

[St Devious]

so as the H.264 and VC-1 encoders r getting updated all the time (Correct me if i am wrong ) , wouldn't it be nice to hold a test with common video samples of every kind to everyone and encode it with the competing codecs at different bitrates .

Since many of the developers r present at this forum , everyone looks to this forum to provide the guidance concerning the latest developments and better codecs . So i think we should hold a test and provide some hard evidence which is easier to comprehend and understand like screenshot comparisons , video samples comparisons unlike the PSNR Graphs .

[Dark Shikari]

so as the H.264 and VC-1 encoders r getting updated all the time (Correct me if i am wrong ) , wouldn't it be nice to hold a test with common video samples of every kind to everyone and encode it with the competing codecs at different bitrates .

Since many of the developers r present at this forum , everyone looks to this forum to provide the guidance concerning the latest developments and better codecs . So i think we should hold a test and provide some hard evidence which is easier to comprehend and understand like screenshot comparisons , video samples comparisons unlike the PSNR Graphs .
Its been done... MSU runs yearly tests, for example, and Doom9 has has a number of past tests also.

Quality honestly hasn't changed that much in the past year; most of the features that really boosted the power of x264, for example (RDO, etc) were implemented a while back.

[Soulhunter]

not at all... at high bitrate they all tend to converge except h.264 than can go a step further and be lossless.1st: I wrote "imo" and personal opinions cant be right or wrong... 2nd: The pic only represented my impression of lossy compression ranging from low bitrates [the usual DVD movie -> 1CD situation] up to high bitrates [ie. DVD movie -> DVD5] but not the "beyond usable for DVD backups" scope [ie. 10000kbps for 720x576 or lossless compression]. And at the mentioned high bitrates, XviD is imo able to outperform x264 n VC1, at least thats my impression... While XviD looks nearly 100% transparent to me, VC1 and x264 still have some visible differences compared to the source... Not always, but often! :]


Bye

[zambelli]

The reason why there aren't new tests done every week is easy: lack of free time. Testing and comparing codecs is very time-consuming. You have to pick good and representative (and hopefully royalty free) content, preferably in several common formats (CIF, SD, HD 720p, HD 1080i)... Then you gotta contact the codec developers to make sure you've got the latest versions of the codecs and the best settings (because if the settings come from the codec teams they can't later complain that the wrong settings were used :) )... Then you have to encode all the content... And then you have to set up A/B comparisons, run PSNR/SSIM scripts, etc.

When you add it all together, that's probably good 1-2 weeks of full-time work right there. Now imagine if somebody who's already got a full-time job tried to do that in their free time. :)

[blizard]

The reason why there aren't new tests done every week is easy: lack of free time. Testing and comparing codecs is very time-consuming. You have to pick good and representative (and hopefully royalty free) content, preferably in several common formats (CIF, SD, HD 720p, HD 1080i)... Then you gotta contact the codec developers to make sure you've got the latest versions of the codecs and the best settings (because if the settings come from the codec teams they can't later complain that the wrong settings were used :) )... Then you have to encode all the content... And then you have to set up A/B comparisons, run PSNR/SSIM scripts, etc.

When you add it all together, that's probably good 1-2 weeks of full-time work right there. Now imagine if somebody who's already got a full-time job tried to do that in their free time. :)

Add to this all the flame war about who use the best setting/codec/source and all those question from first time user of anything called a codec on this site. All this would flood doom9 With lazy encoders/transcoder. People that never read a forum rule before posting on this site or try to gather their own knowledge by searching forum before posting.
After all how useful would those test be when encoding depends on how good the source are, what kind of movie/video that are to be encoded etc?