Codec Shootout 2007?

[shevegen]

I disagree that decoding speed should not be a judging criteria.
For me, it is very interesting to read about it - and also quite important.

[Dark Shikari]

Decoding speed is very important; CoreAVC, for example, shows to be roughly twice as fast as libavcodec in my personal tests even on a single thread, and twice as fast again if using two cores.

This means that while FFDshow chokes on the 1920x1200 video a friend of mine made, CoreAVC handles it quite easily with tons of CPU to spare.

And that's just within a single encoding/decoding standard.

[check]

decoding speed should not be tested because:
o it's impossible to objectively compare decoders. Take h264. The two most common decoders are libavcodec and coreavc. Coreavc is faster, yet it produces horrible errors on some streams. To me, this means ffdshow is obviously superior, but my computer can play 20mbit 1080p with ffdshow. decoder comparisons are too subjective.
o This is an encoder test. Testing decoders should be a separate test if it is at all done. Otherwise there are two variables being tested, which is just a bad idea.

In short, the CC07 is about testing encoder prowess, not overall ability of formats from creation to playback.

[Manao]

decoder comparisons are too subjectiveNo. Once the criteria for the comparisons are clearly defined, decoders - and especially AVC/H264 ones - can be tested completely objectively. CoreAVC is faster than FFShow. CoreAVC decodes PAff while FFDShow doesn't. FFDShow is more standard compliant(a.k.a bugfree) than CoreAVC on progressive/MBAff streams. You see, it's all objective.

Edit : I forgot : FFDShow / lavc is free, CoreAVC isn't :p

[check]

OK, you're right, but I miswrote what I said! :p. What I meant is that while you can objectively compare decoders on individual points, there's really no way to compare them overall.
In your above example, the score is 2-2. Which point is worth more? Since I don't care about decoding speed, does that mean lavc trumps coreavc?
In other words, I think it's certainly possible to compare decoders, but there is no way to rank them in a manner that is important.

[Inventive Software]

Also bear in mind, Doom9's sort-of moved to test viable backup solutions (even if they contain several 3rd party kits); audio encoder / decoder, video encoder / decoder, and container parser. At least that's how I understood his last conclusions. ;)

[twolfe18]

i agree with check in the sense that decoders should not be tested with the encoder. it is better to just test that separately, otherwise things get too complicated.

as for the encoder test, i think that we should be testing pretty much all HD stuff. the format used to be compressing DVD to CD, but now i think we should do HD-DVD/Bluray to DVD. things to consider would be encoding speed, decoding speed (yes, i know that kinda contradicts what i said, but i think that there has to be some comparison between each codec as opposed to between each decoder for each codec), and of course video quality. i personally would weight these factors as 65% picture quality, 25% encoding speed, 10% decoding speed.

[CruNcher]

Imho Decoding isn't that important anymore it was some years ago yes but now more and more chips that are capable of decoding AVC are being manufactured and available in big masses and also GFX manufactures adopted to it very fast in the last year (and have solutions now available PV2/UVD).
It's wide market support is much much bigger then that for ASP ever was (it's Hollywood accepted :P) (also the reason DivX is going to die if they don't adapt fast it would be the best time i think to release "Q" now :P ) anyway i just like Decoder tests to see how good the coders behind the decoder are and what to think about the R&D capabilities of a company and ofcourse also older Hardware that you still wan't to use and very very important because of interoperability problems between Open Source and Commercial Applications. But i don't see why Doom9 should do these Decoder tests also, he's allready under time presure with the Encoding stuff.

[leiming2006]

I wonder how long it will be before I can use a beta version of xvidavc.
It seems quite wonderful but I am not able to build one from the source code.

[CruNcher]

I wonder how long it will be before I can use a beta version of xvidavc.
It seems quite wonderful but I am not able to build one from the source code.

Source where who what ???

[Shinigami-Sama]

OK, you're right, but I miswrote what I said! :p. What I meant is that while you can objectively compare decoders on individual points, there's really no way to compare them overall.
In your above example, the score is 2-2. Which point is worth more? Since I don't care about decoding speed, does that mean lavc trumps coreavc?
In other words, I think it's certainly possible to compare decoders, but there is no way to rank them in a manner that is important.

if one decoder isn't up to standards it shouldn't even be considered imo, they should all decode to the exact same picture, speed should be the only difference.
that drops core right away - plus I'm not sure if doom9 could even get a copy of current release of core

[check]

if one decoder isn't up to standards it shouldn't even be considered imo, they should all decode to the exact same picture, speed should be the only difference.
that drops core right away - plus I'm not sure if doom9 could even get a copy of current release of core
Interestingly enough, coreavc1.3 is fully compliant, but there are decoding errors occasionally seen with it. This is actually due to x264 being a little laxer with some requirements than coreavc is willing to be (mv range =<512 afaik).

[Shinigami-Sama]

Interestingly enough, coreavc1.3 is fully compliant, but there are decoding errors occasionally seen with it. This is actually due to x264 being a little laxer with some requirements than coreavc is willing to be (mv range =<512 afaik).

last I seen 1.3 wasn't available yet, but if its back in standards that a good thing

[zambelli]

i agree with check in the sense that decoders should not be tested with the encoder. it is better to just test that separately, otherwise things get too complicated.
I think it all depends on the context of the shootout. There's nothing wrong with only testing encoders if you name the shootout "a video encoder shootout". But in the past Doom9 has called it a "codec shootout" but only focused on encoding, which in my opinion is not true to the name. A codec involves both encoding and decoding, so if something is going to be named "codec shootout", it would only make sense that both encoders and decoders were tested.

That said, I think it's perfectly fine if an encoder and decoder tests are independent of each other. In other words, if the DivX decoder does poorly, that shouldn't affect the score of the DivX encoder, even if they are meant to be used together. Encoders and decoders of the same brand/author used to be tied in together far more tightly in the past than they are today when most codecs are just implementations of broader standards and encoder/decoder decoupling is possible.

as for the encoder test, i think that we should be testing pretty much all HD stuff. the format used to be compressing DVD to CD, but now i think we should do HD-DVD/Bluray to DVD. things to consider would be encoding speed, decoding speed (yes, i know that kinda contradicts what i said, but i think that there has to be some comparison between each codec as opposed to between each decoder for each codec), and of course video quality. i personally would weight these factors as 65% picture quality, 25% encoding speed, 10% decoding speed.
Obviously the scope of a shootout is entirely dependent on the resources (time, machines, people, $$$) available to the author, but in an ideal world I think a modern codec shootout should focus on at least these scenarios:

SD encoding for web streaming (this is a huge and often overlooked market, allowing places like YouTube to get away with terrible quality)
SD encoding for offline playback (this could include the standard DVD backup scenario)
HD encoding for VOD (focus on low-bitrates, could include the HDTV backup scenario)
HD encoding for HD-DVD and BluRay (focus on achieving transparency to the master)

[skal]

SD encoding for web streaming (this is a huge and often overlooked market, allowing places like YouTube to get away with terrible quality)


I'd say YouTube has other constraints than just codec quality to consider. Like, sticking to a given FlashPlayer version and the FLV format that comes with.

Oh, but YouTube is *also* in h264 over iPhone/AppleTV now.

Skal

[zambelli]

I'd say YouTube has other constraints than just codec quality to consider. Like, sticking to a given FlashPlayer version and the FLV format that comes with.
But they entirely control the decoding pipeline, and it's very easy to upgrade the Flash ActiveX control to one that supports their latest codec. If Adobe decided to switch from On2 codecs to something else tomorrow, I don't think it'd be too difficult for them to force a client upgrade. Anyway... My point was essentially that low-bitrate SD and sub-SD video is far from dead.
Oh, but YouTube is *also* in h264 over iPhone/AppleTV now.

I think this is one of those scenarios where VC-1 and Windows Media really have an edge over Flash and H.264. Same WMV files could be used both for streaming (soon to be enhanced even further with Silverlight) and mobile device playback (an experience that could even be extended to iPods if Apple chose to add a SMPTE VC-1 decoder). MBR (multi-bitrate) streaming is another option. Generally Windows Media scales very well in these type of scenarios. Anyway... That's a little off topic but I thought I'd point it out.

[skal]

But they entirely control the decoding pipeline, and it's very easy to upgrade the Flash ActiveX control to one that supports their latest codec. If Adobe decided to switch from On2 codecs to something else tomorrow, I don't think it'd be too difficult for them to force a client upgrade.
That's oh-so Microsoft'ish. Where is it written that people like being forced to upgrade, or being service-packed every morning? I'm thankful to YouTube for still being able to watch their site with Flash7 under Linux.

I think this is one of those scenarios where VC-1 and Windows Media really have an edge over Flash and H.264. Same WMV files could be used both for streaming (soon to be enhanced even further with Silverlight) and mobile device playback (an experience that could even be extended to iPods if Apple chose to add a SMPTE VC-1 decoder). MBR (multi-bitrate) streaming is another option. Generally Windows Media scales very well in these type of scenarios. Anyway... That's a little off topic but I thought I'd point it out.
Is the ad-break over?

[Inventive Software]

There's nothing much wrong with plugging your product as an alternative. We've seen more blatant examples. ;)

as for the encoder test, i think that we should be testing pretty much all HD stuff. the format used to be compressing DVD to CD, but now i think we should do HD-DVD/Bluray to DVD. things to consider would be encoding speed, decoding speed (yes, i know that kinda contradicts what i said, but i think that there has to be some comparison between each codec as opposed to between each decoder for each codec), and of course video quality. i personally would weight these factors as 65% picture quality, 25% encoding speed, 10% decoding speed.

Donate lots of money to me for an HD-DVD or BluRay drive, and I'll happily do HD-DVD / BluRay to CD in my codec test. As I'm limited on income however, it'll be free lossless HD footage (Elephants Dream being one example) unless that changes. ;)

DVD to CD is still very relevant in the new days of MPEG-4 standalones. There are more on the market than ever offering MPEG-4 ASP compatibility and caveats in equal measure, so that's why the test still stands the test of time and relevancy IMHO.

[Die Kuh macht Muh]

I plan to do one in the summer (June / July / August time). Stay tuned. ;)
So... how's the codec test coming along? I am anxiously waiting!

[Inventive Software]

That was in May, before I lost the power supply for my laptop which does the encoding. Stay tuned still. Apologies for the long wait. I'm also on holiday BTW. ;)

[baer999]

What about a new codec comparison? Will there be any or won't?

[Shinigami-Sama]

I forgot about this actually...

any chance of maybe splitting it up between a group of people and doing it that way to save the weeks of encoding time?

I'd donate some cycles

[finas]

Doom9,

Can you advance a date for a codec shootout made by you! Can't even remember the last time you did one ;)

[dimzon]

Doom9,

Can you advance a date for a codec shootout made by you! Can't even remember the last time you did one ;)
+1
:)

[jurid]

Doom9,

any news on the codec shootout?

[Shinigami-Sama]

Doom9,

any news on the codec shootout?

hes at an electronics show right now, so we're not ganna hear from him for at least another week unless he can find a hotspot and the time to use it

[Ranguvar]

I forgot about this actually...

any chance of maybe splitting it up between a group of people and doing it that way to save the weeks of encoding time?

I'd donate some cycles
Yeah, good idea.

My new quad core is at your service, Doom9 :D

[jurid]

Yeah, good idea.

My new quad core is at your service, Doom9 :D

Doom9,

are you back? any news on the shootout? my dual-processor Precision 490 feels lonely.