Doom9's 2005 codec comparison

[Doom9]

You're always going to lose detail when compressing a source. For a really good comparison, I would be able to use uncompressed sources, but that content is the holy grail and Hollywood will rather burn itself to the ground than ever spread such content.

[*.mp4 guy]

The source was encoded with a mpeg2 codec. That bias the comparison towards ASP because it looks the same way mpeg2 does ( since they share the same DCT ), while AVC definitely looks different. However, from a real source, the difference in quality might ( and I say might because I have actually seen the proof yet, lacking of unencoded material ) be quite bigger than Doom9's test showed.
I would agree with you except I have re-encoded quicktime hd h.264 tralers and they give more or less the same results as dvds do.

[Doom9]

except I have re-encoded quicktime hd h.264 tralers and they give more or less the same results as dvds do.But do we know in which format Apple gets those trailers?

[CruNcher]

You're always going to lose detail when compressing a source. For a really good comparison, I would be able to use uncompressed sources, but that content is the holy grail and Hollywood will rather burn itself to the ground than ever spread such content.


Did you ever tried to get in touch with a Studio maybe they give you a license for a trailer for scientific purposes, who knows. I know i most probably dream but hey trying costs nothing :P I wouldn't ask Universal, Warner or Sony tough :P maybe something more with a family attitude Lionsgate maybe ? hey im sure Boll KG is not so hard and would agree giving you some Trailer from Alone in the Dark, Bloodrayne, or maybe the new Dungeon Siege in Source Form, would be good Advertising also for them, complete Source of a Film im in doubt but who knows :)

[zambelli]

Doom9:

I'll take the opportunity to chime in with some feedback on my particular experience with this year's shootout (VC-1 submission) and some suggestions for next year's shootout.

First of all, kudos to you for organizing the codec shootout and spending 2 weeks of your winter holidays encoding movies and having "fun" with pre-release quality software. :) The hard work is much appreciated by everyone, I'm sure. This codec shootout is your project, so it's only fair that you get to determine your own metodology. Personally I don't look at the shootouts as absolute proof of one codec's superiority over another, but I appreciate the independent nature of the comparison and I think it gives a good general idea of what to expect from each codec in terms of quality/performance.

I'd like to second some of DigitAl56K's concerns, particularly the one about the downside of organizing the codec shootout around Christmas holidays. During the second half of December the Microsoft corporate campus in Redmond is half-empty, as I'm sure is the case with any other corporate environment in America and Europe. Most people are on vacation, many aren't even checking their email. If some devs are working around Christmas time, there's a strong chance they're already busy working on some high-priority project and can't afford to spend time fixing other new bugs. 48 hours might seem like a lot of time to fix 1 bug, but when your devs are working on high-profile projects such as Vista and HD-DVD - it's next to impossible to pull them away from whatever they're doing and make them investigate and fix a bug for a comparatively low-priority codec shootout. While I understand that Christmas holidays might be the most convenient time for you to do the codec shootout, I have to say it's the most inconvenient time for development teams to assist you. Any other time of the year would probably work better. Perhaps we could reach some sort of compromise?

Before the codec shootout even started, I voiced my concern over the lack of HD content in this year's comparison. I understand that it's mostly a practical problem of aquiring high quality HD content. While Hollywood studios might not be willing to share their uncompressed HD sources with you, keep in mind that most studios and post-production facilities probably have royalty-free HD content they could share out. It might not be as cool as Matrix: Revolutions, but it'd do the job. Microsoft has HD recording equipment and I'm sure DivX, Ahead, Mainconcept and others do too, so unless you're concerned about a "content bias" there's always the option of contacting the software makers themselves early enough and asking them for HD content. In fact, if you get multiple sources, you could even put together a feature-length collage. With HDTV already widely available in the U.S. and HD-DVD and Blu-Ray on the horizon, I'm counting on next year's comparison featuring some nice 1080p content. :)

Finally, I'd like to revisit the topic of post-processing in the codec comparison. Although the FAQ explains that DECoding is an integral part of the coDEC solution, I think the focus of this shootout has historically been on the encoding part, and now more than ever. As an example, encoding performance is reviewed, while decoding performance isn't even mentioned. Furthermore, these days not all encoders have their own matching decoders but instead many depend on generic decoders such as ffdshow. That means that any difference between, for example, Ateme and x264 will not be the result of the decoding process but of the encoding process. The future might also bring us competing implementations of VC-1 encoding, but not necessarily of VC-1 decoding. While that actually slightly evens out the playing field, I think it only goes to prove that the decoder is just an unnecessary abstraction in the equation. You mention in many of your screenshots how the encoded image lacks fine detail compared to the source. In my experience, the lack of detail is more often a consequence of the decoder's filtering, deblocking and deringing than an indicator of encoder quality. I always find it surprising how much detail is actually retained in encoded clips, yet the detail completely disappeares after post-processing is applied. Something to consider.

[CruNcher]

In my experience, the lack of detail is more often a consequence of the decoder's filtering, deblocking and deringing than an indicator of encoder quality. I always find it surprising how much detail is actually retained in encoded clips, yet the detail completely disappeares after post-processing is applied. Something to consider.

Hmm zambelli everyone here knows that but did you ever deactivated the inloop deblocker in a decoder when the encoder used it @ low bitrate, not a very good idea (unless you wan't a blockfest of mini blocks) and it definately wouldn't preserve details as those are allready gone in th encoding process, for the ASP codecs that where tested and VC-1 what you say would apply but not for the AVC codecs in the test, only very good Film Grain modeling (else the scene would life) and inloop deblocking off or a higher bitrate could preserve the blurry look . Inloop Deblocking (pre- processing) and post-processing are 2 completly different things.

[Doom9]

One important fact of the comparison background seems to have gone missing here: it's a codec comparison in a typical DVD backup scenario. It's not a synthetic comparison, but a real world one. Hence containers, hence audio, hence full movies. There's a huge difference between that scenario and encoding HD trailers from apple or using MPEG test sequences. It's also a logicistal matter. Main round encoding lasted 48 hours.. in that time, I managed to encode Matrix 3 10 times, despite VC-1 being what I consider extremely slow. So take that extremely slow and multiply it by a factor of 9 for a 1080p encoding... that's 4.5 days spent encoding a single movie. So going for one HD movie instead of an SD movie, you can extend encoding from two days to about a month.. during that time, the more you use your PC for anything, the slower it gets, there's the logistical issue of moving my PC from where I sleep and the suicide risk in case I have to restart a job.
And in that month, there can be significant changes in a codec, so it could very well happen that by the time you're done encoding, the most recent codec build makes your results obsolete before you've even done the review.

48 hours might seem like a lot of time to fix 1 bugI consider it quite the opposite, it's a rescue op for catastrophic issues like missing the bitrate ;) I expect to receive software that has been tested and shouldn't fall on my head when I use it. I consider the 2 business day rule just enough to make some last minute adjustments in case you made a wrongful submission, nothing more. I never have and never will expect that issues like the ones that came up in the finals could actually be addressed in that short amount of time.
But I'd like to point out that I got two bugfixed encoders from the XviD team, a fixed decoder from DXN on very short notice, as well as a new encoder build and settings from Elecard after telling them they were qualified for the main round (you saw the time schedule between qualification and main round, there wasn't a lot of time).

As far as decoders go, with most codec makers moving to AVC, the whole postprocessing issue becomes rather moot, and in my own experience, deactivating deblocking for an ASP codec does as much to help in certain scenes as it does hurt in others. And doesn't VC-1 have an inloop filter as well?

As an example, encoding performance is reviewed, while decoding performance isn't even mentioned.I know you want to get at "VC-1 takes less computing power to decode than AVC" with this one, but my box has no problem with 1080p AVC content, and it's you guys from Microsoft that will do away with this problem within a years time by forcing almost everybody to get an up-to-date GFX card for Vista, which in terms will offer built-in hardware acceleration for video decoding. And as far as hardware goes, the point is moot anyway because we have three mandatory codecs for both HD DVD and Blu-ray. And in Europe, we'll actually see quite a but of DVB-S2 broadcasts using AVC so set-top boxes will be able to handle that as well.

[zambelli]

One important fact of the comparison background seems to have gone missing here: it's a codec comparison in a typical DVD backup scenario. It's not a synthetic comparison, but a real world one. Hence containers, hence audio, hence full movies.
Yes, but the same prinicipals are applied to transcoding TV content, too. That's just as real world of a scenario and it follows the same rules, yet often includes mixed content AND HDTV. :)

I consider the 2 business day rule just enough to make some last minute adjustments in case you made a wrongful submission, nothing more. I never have and never will expect that issues like the ones that came up in the finals could actually be addressed in that short amount of time.
Right, and I wasn't complaining really about the 2 business days rule but was just pointing out how fixing anything in 2 days is difficult when most of your staff is on vacation.

I know you want to get at "VC-1 takes less computing power to decode than AVC" with this one...
You'd think so, but that's not where I was going. :) I pointed out the lack of decoding performance numbers as an indicator that the comparison is encoder-centric, not as a suggestion that you should start testing decoding performance.

[seewen]

Thanks a lot for this codec comparison! I'm waiting for it every year.

But this time I must say that I don't agree with the results... But I understand it's a matter of personal taste:

Ateme (according to the posted samples) and especially x264 (according to samples, and personal experience), constantly smooth the pictures. All the grain is virtually gone. (On movies.. not cartoons/Animes).
And I must say that personally, I much prefer XviD results, which doesn't smooth too much the picture, and which keep more grain and details.

But I understand completely that some(most?) prefer the smoothed/cleaned result.


Thanks a lot for this comparison.

[CruNcher]

seewen alot of people don't like this look (includes me) and if you like Grain Preserved try EDP in my signature (Better Profile) :)

[skal]

Hi,
sorry to jump in late, but: i'm having an uneasy feeling when i see 100+ people company representatives complaining about the lack of manpower...
Just my morning mood, please ignore :)

[zambelli]

Hi,
sorry to jump in late, but: i'm having an uneasy feeling when i see 100+ people company representatives complaining about the lack of manpower...
Just my morning mood, please ignore :)
Good call. Next year - no Christmas holidays for anyone.
Must be my late night mood, please ignore. :sly:

[Doom9]

I much prefer XviD results, which doesn't smooth too much the picture, and which keep more grain and details.Actually, XviD gets rid of the grain, too.. I might not have made that clear enough. And the in-loop filter is configurable in AVC.. the default in most codecs is quite strong, you might like the results better if you turn it down a notch or two (as was done in most contestants).

[CruNcher]

For HD stuff @ High Bitrates you can even turn inloop completly off it will still look visualy perfect and you save decoding complexity :)

[DigitAl56K]

Hi,
sorry to jump in late, but: i'm having an uneasy feeling when i see 100+ people company representatives complaining about the lack of manpower...

While zambelli does make a good point (;)) remember also that in a 100+ people company there are likely only a handful whose level of technical expertise and specialist field is appropriate for this kind of work.

[bobololo]

My turn to throw a few words into this thread :)

First I wouldn't be honest if I don't admit some disappointments regarding the fact we couldn't keep our throne this year ;). Anyway whatever the causes that make us fall on that night sequence, was it a bug or not, it is incontestable that the margins have become extremely tight and for that impressive improvement, this year's winner desserves its place. On our side, we know now that keeping the highest rank definitively requires us to push forward continuously the progress and to maintain unsubtle advance over the competitors. And even if applying such policy to the many fields we're addressing is becoming a serious puzzle, be sure we're committed to do our best and we'll sort it out !

That said, the previous posts stir up some thoughts I found useful to express. I hope those comments won't be interpreted like complaints about the comparison but instead will bring some idea to make it better.

The topic about the settings provided for the comparison had been discussed in several posts and I think it spots out an interesting trend in modern codecs. They are becoming more and more complex, they offer many options, quality / speed compromises, and compatibility settings covering at best the users' multiple needs. We're far from the time when encoders have a single bitrate parameter ! Considering this, how can we define the "best" settings for a subjective evaluation ? As the results illustrated it, different strategies were adopted. Most of them emphasize the best efficiency-performance trade-off but having their respective woking zone quite scattered, I don't have the feeling that the codecs were competiting exactly in the same event. It's like you have to decide which of a sprinter or a 400m runner is the best athlete ;) . One approach that could help to reduce this effect would consist to specify for instance a speed target on a specific hardware and require the encoders to give their best quality according this constraint. Then it's up to the codec to stay above that minimal limit and if it goes much faster, it's its loose not exploiting all the resource available to it. That would additionnaly ensure not to spend to much time with the encoding stage !

Another point about the settings. Some of them depends much on personal tastes. At this point it's extremely hard for the developers to decide for the evaluator. Especially it could also depends on the viewing environment (type of screen, how the place is lighted, etc.). Wouldn't be possible to adjust those settings on 1 or 2 short clips accordingly to the tester's taste ?

Lastly and probably the most debatable point : wouldn't be that codec comparison even more revelant if it were led by a few people instead of a single person ? It has nothing to do with Doom9's judgement, but drawing conclusions and deciding which is the best (whatever the field) based on a single person's subjective perception is quite unusual, isn't it ? I'm pretty sure, there are over there plenty of board members who would be happy to take part to that project. That could reduce Doom9 load and bring more solid results. It's always easier to argue against a single man opinion than a result consolidated by 10 or more persons. Hydrogen Audio public listening tests are a good example even it's not the same kind of comparison. What do you think ?

disclaimer: just to make sure, that were my personal opinions and they don't involve ateme's official position.

[Doom9]

One approach that could help to reduce this effect would consist to specify for instance a speed target on a specific hardware and require the encoders to give their best quality according this constraint.But how is this feasible? AFAIK your codec is the only one that can target a specific encoding speed.. others simply cannot. The people working on the codecs without such functionality would effectively be forced to buy the exact system I use for encoding, then having to find the settings that get as close as possible to that speed goal. I don't see a lot of contestants putting up with that. And depending on where you set this threshold.. I have a very powerful machine.. when I set the treshold to real-time encoding, what about people who have 5+ year old machines (I agree, they should upgrade ;)).. they might chose different settings because they don't want to wait for days.
And there's another point: what do you do with encoders that max out? I'm sure you'd actually have to slow down your ASP encoder to encode at only real-time speed on my machine.. and that would probably apply to a few more codecs.. so in the end we have those that can go for max quality, while others have to use a severely restricted featureset to even get to that speed.

But in a way, I'm doing half that by requiring a minimum encoding speed, and I'm seriously considering a steep hike in the minimum required speed for the next round (I'm dreaming of an AMD dual core system with both cores running at 3 GHz). Yet one of the main differences of codecs besides quality is speed.. if a codec can only use one core or if it uses both cores make a huge amount of difference and would put an additional burden on those codecs that have no SMP optimization (as there's no way I'm ever going to give up on using SMP optimization).

I don't have the feeling that the codecs were competiting exactly in the same event.Well.. the source (distance, track properties) and time measuring equipment (eyes) were the same.. You should rather compare it to F1 racing though.. you can have different tires, different car chassis, use different materials, motors, etc.. all that has an effect and in the end the only thing that counts is that you cross the finish line in first place. The same applies for other mechanized sports as well, so they have a much higher resemblance than a track race.

At this point it's extremely hard for the developers to decide for the evaluator. That's part of the fun, isn't it? And it's part of me covering my own ass.. I don't need to be attacked for using certain settings after the comparison, there's enough discontent as it is and I've seen how things went in past years when I picked the settings instead.

Wouldn't be possible to adjust those settings on 1 or 2 short clips accordingly to the tester's taste ?Don't you already have such a benchmarks by looking at results from previous comparisons?

wouldn't be that codec comparison even more revelant if it were led by a few people instead of a single person ?That is very much debateable instead. Personally I obviously only trust myself to make that decision.. for me a comparison where 10 other people have a different opinion, I couldn't possibly write that up anymore. In the end, you could only show how people rated things, but it's no longer possible to make subjective comments at all as you'd end up with a bunch of contradicting statements.

That could reduce Doom9 load and bring more solid results. I don't see the load angle at all.. in fact I see it the reverse way. In addition to encoding, you have to distribute review clips, so it's cutting, uploading, traffic, bandwidth and possible legal issues to boot. Then you have to ensure the same review environment which means additional work for me explaining and for you fixing problems (remember the 2004 comparison and taking screenshots? And remember the WMV9 results that got messed up somehow? Now imagine those issues multiplied by the number of reviewers... player, filters installed, GFX cards, drivers, it all can have an influence).

Sure, I cannot cover a broad spectrum of opinions with just one reviewer, but what is all the info on page 1 for if not for letting people reproduce the results? They can get their hands on 3 out of the 4 main round contestants, and a bunch of other codecs that delivered good results but didn't quite make it (VP7, XviD AVC (that one soon)).

[Manao]

The people working on the codecs without such functionality would effectively be forced to buy the exact system I use for encoding, then having to find the settings that get as close as possible to that speed goal.Well, for open source codecs, finding a willing tester with the proper hardware isn't that hard. And in any case, interpolating the expected results is possible ( as soon as the computer is roughly the same ).

Note also that if you specify an encoding speed, we would be on par with other codecs, because we would definitely not use our 'target speed' encoding quality, since this one gives a constant speed ( not an overall speed, meaning the hard parts of the movie, which are incidentaly the slowest one too, would be encoded with the lowest quality settings )And there's another point: what do you do with encoders that max out?Fair point. Yet, maxing out an AVC codec is quite hard ( lets forget that ATI exists at the moment ;) ), and since you already made two categories ( ASP, AVC ), you could set two target speeds.the only thing that counts is that you cross the finish line in first placeIn the case of the codecs, the track isn't a one dimension line, it's a two dimensionnal plane ( quality, speed ). You still have to say where to draw the line :)I don't need to be attacked for using certain settings after the comparisonFair point.

The only solution I can think of - developpers giving a set of settings with some of them being intervals ( [-4..0] for deblocking, for example ) would quite complicate and slow down your job.I don't see the load angle at all.. in fact I see it the reverse waySo do I :)

[Doom9]

Well, for open source codecs, finding a willing tester with the proper hardware isn't that hard.It's still an n-dimensional problem.. and let's just say my experience with volunteers hasn't exactly been great (lavc...)

and since you already made two categories ( ASP, AVC ), you could set two target speeds.Those are likely to go away in the future and to be replaced with generic categories where codecs are placed into at random.. this is to ensure that in the end, I get the 4 (4 is a good number for the final actually.. and considering that I'm considering a full length HDTV full res movie for the future... a small number is so much more important) codecs that I deem to be best, not some that made it just because there was no competition in their category.

And naturally an ASP codec going 40 fps wouldn't thrill me when I have AVC encoders doing more than that at good quality (leaving out ATI yet again.. it's encoding speed rivals that of XviD).

I actually would like to increase the minimal encoding speed to real-time.. the reason I have only slowly raised the bar is it makes entry very hard.

And by the way, had the SPR results been on par, then the ateme encoder would've been the winner.. but who is to say that if you slow the ateme decoder by 10 fps, that the change is noticeable? It actually could be an interesting experiment ;) Isn't it usually so that activating the extreme options slow down an encoder incredibly while only bringing very little to the table?

[Manao]

but who is to say that if you slow the ateme decoder by 10 fps, that the change is noticeable? It actually could be an interesting experimentI'd say that x264 would still have a slight edge, because it looks slightly more sharp ( on matrix 3, the only movie I thoroughly tested ), and slightly less stable too ( but that's less annoying to me, on the overall ). To see the difference, you'd need to pause the movie and that's definitely not a proper to compare visually the codec. So I'd say in the end they are on the same level.

Anyway, you can make the following test : encode the movie with 5% less bitrate, and try to see the difference. I don't think I can see it. It amounts - roughly - to the same ( actually, 5% reduction file is bigger than the quality lost by 10 more fps - at these speeds ).

[Caroliano]

@Archimedes: (http://forum.doom9.org/showthread.php?p=758348#post758348) Another example of how metrics are not so good. For example: In "The Confrontation" scene of Steamboy the quality diference between Xvid and x264 is very big IMHO, not only 0,049 SSIM. Ringing, blocking and black over brown problems with Xvid, and none of them with x264.

You don't comented about the ringing issue on the scene "in the train". Xvid has much less ringing than Divx in this scene.

A coment: In the last round, you refered many times to Xvid performace in Divx comentaries, even though it was before Xvid in your list of screenshots. The rule is normaly to refer to an already showed thing, not an future thing, to explain a point. This was a bit inconvenient, not much of course.

Thanks for the comparison.

[virus]

Those are likely to go away in the future and to be replaced with generic categories where codecs are placed into at random.. this is to ensure that in the end, I get the 4 (4 is a good number for the final actually.. and considering that I'm considering a full length HDTV full res movie for the future... a small number is so much more important) codecs that I deem to be best, not some that made it just because there was no competition in their category.
Sounds like you're going to organize comparisons like a Fifa World Cup draw :D

(actually, that looks like a good idea, when you want to quickly select competitors among a large group, avoiding too much "matches" - that's why such kind of things exist in sport competitions. All you need to do is choose a wise way to create the groups - putting x264 and ateme together in the qualifications would probably deprive the final round of one of the best 2-3 codecs... but if I understand your words correctly we're already singing the same tune here ;))


...and now, since I'm here, I'll have to post my comment on the comparison itself :)

What I want to say is: thanks to Loren, Alex W., Christian, Alex I., Radek, Tuukka, Mathieu, Måns and all the other people who have contributed great patches to x264 during 2005. The progress this codec has made during this year is incredible, no matter how you look at it. Hats off.

[bobololo]

But how is this feasible? AFAIK your codec is the only one that can target a specific encoding speed.. others simply cannot. The people working on the codecs without such functionality would effectively be forced to buy the exact system I use for encoding, then having to find the settings that get as close as possible to that speed goal. I don't see a lot of contestants putting up with that. And depending on where you set this threshold.. I have a very powerful machine.. when I set the treshold to real-time encoding, what about people who have 5+ year old machines (I agree, they should upgrade ;)).. they might chose different settings because they don't want to wait for days.

I don't think it would be so hard to achieve especially if some tolerance is allowed. I guess the best think would consist to ask to contestants if they prefer to provide their codec setting having in mind a indicative speed target or if they prefer operating freely (and blindly regarding how other codecs would be configured). Personaly I would go for the first choice. And if you consider your hardware is running too fast, just increase the required performance ;).

And there's another point: what do you do with encoders that max out? I'm sure you'd actually have to slow down your ASP encoder to encode at only real-time speed on my machine.. and that would probably apply to a few more codecs.. so in the end we have those that can go for max quality, while others have to use a severely restricted featureset to even get to that speed.

As long as the main criterion of the comparison is the quality, I don't see why maxed-out codec would suffer from this constraint ? They simply have to give the best quality they can ignoring that speed limit since they are far above. And beside that doesn't prevent you (as you actually do) to have a sidenote telling which codec offers the best quality/speed compromise.

Well.. the source (distance, track properties) and time measuring equipment (eyes) were the same.. You should rather compare it to F1 racing though.. you can have different tires, different car chassis, use different materials, motors, etc.. all that has an effect and in the end the only thing that counts is that you cross the finish line in first place. The same applies for other mechanized sports as well, so they have a much higher resemblance than a track race.

I would agree if your comparison used objective metrics to decide the winner like with MSU or Sagittaire's comparison. In a F1 race, the winner is declared by a very actual and objective fact which is totally different from your subjective evaluation. I would rather assimilate that activity to a ice-skating or gymnastics event where a jury has to rate the contestants by giving marks.

That's part of the fun, isn't it? And it's part of me covering my own ass.. I don't need to be attacked for using certain settings after the comparison, there's enough discontent as it is and I've seen how things went in past years when I picked the settings instead.

Well, that's not so fun (to me at least :)). And you don't need to choose all the settings by yourself. The devs could provide you the basic configuration and a few customisable parameters you can adjust to your taste. Is that so incongruous ?

That is very much debateable instead. Personally I obviously only trust myself to make that decision.. for me a comparison where 10 other people have a different opinion, I couldn't possibly write that up anymore. In the end, you could only show how people rated things, but it's no longer possible to make subjective comments at all as you'd end up with a bunch of contradicting statements.

What prevent you to present different writes up giving the different points of view ? Wouldn't that approach more democratic and even more interesting for the reader to see the different opinions ? And to be honest, if the jury is made of skilled and open-minded (ie not a zealot) persons like we can find many here, I'm pretty sure that it won't have big divergences in the opinions. If you refer to many situations (whatever the domain) where a subjective judgement has to be made, the jury is usually composed of several members and it's rather rare to have a single ruler acting like a "god" that makes the decision alone. Actually I don't see any example of this case, do you have ?

I don't see the load angle at all.. in fact I see it the reverse way. In addition to encoding, you have to distribute review clips, so it's cutting, uploading, traffic, bandwidth and possible legal issues to boot. Then you have to ensure the same review environment which means additional work for me explaining and for you fixing problems (remember the 2004 comparison and taking screenshots? And remember the WMV9 results that got messed up somehow? Now imagine those issues multiplied by the number of reviewers... player, filters installed, GFX cards, drivers, it all can have an influence).

Of course you're right, scaling up the evaluation process involves logistical issues and no doubt it's far to be easy to setup such an environment. However, if you allow me to be a little bit teasing, I would say those arguments are loosers' ones ;). Dropping ideas by invoking technical difficulties without trying to find some solutions to work them around is a behaviour that could in some way be similar to bitchy people complaining about dct issues making their champion performing poorly ;) I'm kidding obviously ! Well actually my only point is that if some ideas or concepts can significantly move things forward, it may be worth to make the efforts to think about and to try to find some means so they could be carried out. I guess that's how things progress and how unbelievable things came to life. I'm totally off-topic but just consider the toughness of sending something to the space. If the person who had that idea at first and concentrated himself upon the issues he would be faced to, I guess he would give up at once to our great loss. Don't you agree ? :)

[Doom9]

I would rather assimilate that activity to a ice-skating or gymnastics event where a jury has to rate the contestants by giving marks.Actually it's a little bit of both as quality is subjective and speed is objective.

Is that so incongruous ?If you want to trade places with me if you get flanked from the left, right and back because of your settings.. I really don't need that anymore.

Actually I don't see any example of this case, do you have ?Do you have any examples of a subjective video quality comparison where that was done? I've seen very little such comparisons and generally they were done by a single person, and by looking at screenshots, not the video. My comparison can only be an indicator, it's still up to the individual user to make the final choice.. no matter how many people you put in a jury, in the end each and everbody not on the jury can still judge differently and if they differ, it's just a further proof that quality is in the eye of the beholder.

However, if you allow me to be a little bit teasing, I would say those arguments are loosers' onesNo, I consider myself a realist. We had to do this funny test at work once where they try to see how well you interact together and what kind of person you are... I was quite shocked to be told I had conservative tendencies as I consider myself very progressive and forward looking, but I've gotten to terms with the fact that while I really like new things, I also have a tendency to try and find the flaws right away to crush a new idea. I know these difficulties can be overcome in this particular case, but at a price. The price is significant additional effort and headaches. For instance you could encode the resulting video losslessly using a common codec, so there's only one playback chain, making it easier to replicate, but on the downside, that increases the size of the files you have to move around. At the end, do not forget that it's my free time we're talking about.. I'm the only one who gets to decide how it's being spent. Codec comparisons are enough of a headache as it is and I am very reluctant to make changes that increase the amount of head pain. And it's not like I'm just discarding without thinking through the process.. finding the problems requires that ;)

[kwtc]

Well, after all that reading, I guess if I had to choose an H.264 encoder for home use, I would not hesitate long and I would pick... x264 since the Ateme codec is not available to home users !

[KillNoise]

No doubt: Encoder parameter settings should allways be provided by developper: Common users desire default working parameters to get good results quickly, rather than experimenting with many passes of try and error. At most, one might be willing to select from few parameter sets depending on certain types of footage, e.g. movie, interlaced TV, cartoon, b/w and perhaps whether affected by strong grain or noise.
Generally i see responsibility of developers to reveal their experience by recommending useful parameters and giving explicit advice for adjustment - not leaving you alone with a bunch of mysterious controls ! Codec performance will be rated by results achieved in practice, rather than potential excellence after expert tweaking parameters individually for each source.


May i suggest small modification of the competion categories for your future comparisons ?
Would be good to not only compared contestants in seperate "weight class" categories, but also have them compete under different conditions based on relevant application "Use Cases".
What are these ? In my view, mainly:

Category 1: Encoding for playback on today's common stand-alone players
==> Standard MPEG-4 ASP, commonly restricted to not using less supported features like GMC or Qpel
==> Visual quality competition for common target size = 2 CDs for one movie
(1400 MB / 90 min --> overall bitrate ca. 2000 kbps for video stream excl. Audio)

Category 2: Quality/space enhanced encoding for playback on PC
==> competition based on smaller target size = single CD for one movie:
(700 MB / 90 min --> overall bitrate ca. 900 kbps for video stream excl. Audio)

I think this competition-split would make results much more practically usefull, while no longer having to perform frame quality comparisons across categories (in fact, these become two competions independent of each others !)


Next, what will be the worth of a competitions winner, if it is not available to the public ?
For example, you have allways been testing Atemes "Edge of Technology" releases, while the licensed version actually available to users through Nero Digital still remained a much older one.
==> would be usefull (challenging public release scedule !) to seperate competition score categories into:

a) Encoders already available within some handy environment (e.g. VirtualDub)

b) Open class "Edge of Technolgy" previews of codecs not yet available for public


Further i see two other relevant application categories, which you might consider for one of your future comparisons:

Category 3: Encoding for very low bitrate streaming applications
==> Visual quality competition based on common low constant bitrate setting
(e.g. 256 kbps CBR at common reduced resolution like 320x240)
This category might give a chance for codecs specialized for this area (e.g. On2 VP7 or Snow wavelet codec)

Category 4: VCR-like realtime encoding
(i know many users doing that for simple delayed watching or even for archive, not willing to setup delayed encoding after intermediate 'raw'-storage)
==> Constantly fast encoding, suited for "on the fly" live TV recording (native interlaced field encoding or perhaps field-deinterlaced to 25 fps for PAL)
==> Visual quality competition based on suitable common average bitrate setting
= some given single-pass target size (e.g. 2000 kbps ABR)

Ok: Speed criteria for realtime encoding qualification are a bit difficult to define - must be something like: "Live encoding without frame-drops on common Mid-Class Multimedia-PC reference hardware" (let's say typically those with SSE-capable single-core CPU working at 2...3 GHz). For simplicity it should be sufficient if you define some minimum reference frame rate to be met on your own PC (or your previous modell if you still have it running).
The effort will be worth a very interesting competition, giving MPEG-4 ASP another chance and having DivX, XviD and Nero/Ateme race in a somewhat different discipline !

[siddharthagandhi]

I say NeroDigital AVC H.264 Standard Profile is the best codec out there. Out of everything.