Doom9's 2005 codec comparison

[IgorC]

Before to create this item I've read a FAQ http://www.doom9.org/codecs-203-6.htm

However I didn't find answer to few questions.

1. The purpose of this test is find best videocodec but with trade on quality/speed. It's good idea. All videos were encoded on dualcore PC. What percentage today of people who have a dualcore? Even in 2006 many people (or even most) will still have onecore cpu. By fastness VP7 is on par with H.264 encoders on onecore PCs.

2. VP7 wasn't tested in 2d round because it was one of the worst codec in H.264 group. Why VP7 was put into H.264 family codec? It will be more fair to compare 3 groups - H.264, MPEG-4 ASP and alternative. As it was mentioned VP7 was best alternative codec.
What is the reason to limite test for H.264 and MPEG-4 groups? ;)
Alternative codecs are also interesting for the markets.
Look at vp6 in flash player, winamp, skype, mobile devices.

3. All setings for each are provided by devs of each codec. But was it fair to compare Ateme with 2 ref. while x264 had 4 ref.? Developers simply don't know what to expect as final output and what settings will be best for speed/quality trade. It's impossible to predict what inloop filter (-6;+6) will be better for video.

4. Before 2005 comparison test picture less or more represented a quality of video. But in this test screenshots don't make any cense. As it was mentioned Xvid AVC was very blocky , but looking at screenshots Xvid AVC was sharpest.

5. Only 4 codecs in second round?

I'm sorry. I found this comparison useless for me. IMHO even Sagittarie's metric test is more real (viable).

[dimzon]

:goodpost: :rolleyes:

[Doom9]

What percentage today of people who have a dualcore? Oh my, I expected criticism for a lot but having a dual core? You are just jealous man. If the speed table shows anything it's that having a dual core chip pays off huge in video encoding. I didn't spend more money on a CPU than most people do on a whole PC just because I can.. I spent that money for video encoding.
By fastness VP7 is on par with H.264 encoders on onecore PCs. That is not true. Look at the numbers here: http://forum.doom9.org/showthread.php?t=94226&highlight=amd
The speed gain for x264 is 58.22%. So, if we presume the same gain for Matrix3, translating the encoding speed from the comparison into single core speed would mean 24.26 fps. And, you cannot compare that with VP7's 18.79 fps, because VP7, as well as any other codec using VDub, will take advantage of the dual core CPU as well. In the first pass, CPU usage averaged about 80%, during the second pass 60%. So, making a very unscientific quick calculation, let's say 70% average, and one core only being 50%, the relative speed gain for VP7 would be 29%, and single core encoding speed would be 14.56 fps, or a whopping 40% encoding speed difference when only using a single core. The difference would be even greater for the ateme encoder, roughly similar for the Elecard encoder, and XviD AVC encodes at 21.19 fps on a single core.

Why VP7 was put into H.264 family codec?You may have read the FAQ but apparently not the entire comparison.. the group distribution change was explained in the main round article.

Developers simply don't know what to expect as final output and what settings will be best for speed/quality trade. They do not? I know for a fact that several codec makers actually make test encodings.. they knew the sources in advance. Some that do test encodings include ateme, akupenguin (x264), isibaar (xvid). And as the FAQ says, if you have questions about the settings, ask the developers.

But in this test screenshots don't make any cense.I never claimed it does.. once again you must not've read the article because you seem to be completely oblivious to the single most important fact about screenshots and video quality: Then my usual disclaimer: Screenshots are not necessarily representative of the video quality. It might be that a screenshot of codec A just happens to fall onto an I-frame which is the least compressed (and thus looks pretty good), whereas quality quickly degrades after the frame shown in the screenshot. And a codec B might have a heavily compressed B-frame at the same position, so comparing the two would be inherently unfair. Bottom line: Take screenshots with a grain of salt, make sure you read all my comments and perform your own tests when in doubt. Every human being perceives visual quality differently and you might come to completely different conclusions watching the same clips as I did. Also, this time I made a point of looking at the source throughout the comparison so I'll have some comments on quality compared to the source as well.
Even today, I have not compared screenshots.. I don't know how good or bad they look.. the review is based on perceived image quality.. if the shots don't match that I'm sorry but I cannot write a review based on screenshots ever, no subjective quality test must ever be based on looking at individual frames - they never have and never will be truly represenative of what you perceive when you watch a video.

[Sagittaire]

Yes it's true but doom9 use complete movie for this test ... and make complete test with 3 complete movie is considerable job. Doom9 is simply human I think ... ;-)

Personnaly I use only 3 trailers (more or less 3000 frame) and it's more 10 days simply for encoding part (3 resolutions x 3 trailers x 3 bitrate x 6 codecs)

[CruNcher]

Yes Igorc but theirfore Ateme used the Extra Quality Mode where x264 used subme 6 and i would say best mode is compareable to subme 6 so Ateme had an advantage their overall it seemed to not cover up the 2 more reference frames that x264 used tough.
You could say in this comparesion not the better codec won but the better settings did.

[Doom9]

5. Only 4 codecs in second round? Actually, this allowed me to perform the most thorough review ever on movies number two and three. I have spent more time reviewing these 4 codecs on my two movies than I spent in previous years reviewing all codecs on a particular source. The more codecs you have to look at, the harder it gets to perform a thorough review.. one factor is time, another one is still recalling what you saw in the same clip 8 codecs ago.. the time you spend actually doesn't increase linearly, it increases exponentially due to this little factlet. Even the groups of five (plus source each time) in the main round were rather hard to handle.. the lower the number the better for the review.

[Sagittaire]

In fact in doom9 test there are something which is completely unusable : the speed test.

I think that speed test must be do with:
(1) speed for exactly the same quality
or
(2) quality for exactly the same speed

For test (1) you must use metric. For test (2) you can use eyes but make encoding with exactly the same fps is difficult in practice.

Personnaly I will certainely use fps = Fct (OPSNR) and fps = Fct (SSIM) for 3 differents quality bitrate level.

[superdump]

2. VP7 wasn't tested in 2d round because it was one of the worst codec in H.264 group. Why VP7 was put into H.264 family codec? It will be more fair to compare 3 groups - H.264, MPEG-4 ASP and alternative. As it was mentioned VP7 was best alternative codec.
What is the reason to limite test for H.264 and MPEG-4 groups? ;)
Alternative codecs are also interesting for the markets.
Look at vp6 in flash player, winamp, skype, mobile devices.
The regrouping was explained. If VP7 couldn't compete with the other codecs in the new group of h.264 and non-MPEG-4, then why should it matter that it wasn't tested any further? As you said, the purpose of this codec comparison is to discover which codec(s) are good. VP7 wasn't up to the same standards as the other codecs in the areas tested so it didn't go through. I don't see the problem.

3. All setings for each are provided by devs of each codec. But was it fair to compare Ateme with 2 ref. while x264 had 4 ref.? Developers simply don't know what to expect as final output and what settings will be best for speed/quality trade. It's impossible to predict what inloop filter (-6;+6) will be better for video.
That's simply not true. The developers are well aware of the capabilities of the codecs and did much testing to come to the decisions about which sets of options to use. The developers are aware of which sources Doom9 uses in his comparisons and I believe were made aware of which new sources he intended to use. As such they could provide a good option set for each source as necessary. With regard to the deblocking filter level, the familiarity with the codecs that their respective developers have allows them to choose a good level.

4. Before 2005 comparison test picture less or more represented a quality of video. But in this test screenshots don't make any cense. As it was mentioned Xvid AVC was very blocky , but looking at screenshots Xvid AVC was sharpest.
I disagree. Sometimes the h.264 codecs were sharper, sometimes they were much smoother. They were usually smoother when the ASP codecs were very blocky which suggests those scenes are very difficult to encode. Regardless, it is clearly stated that the test is subjective and you should reach your own conclusions which may well differ to those of Doom9. That is the reason he displays the images and gives comments as well regarding his overall impression of a scene.

5. Only 4 codecs in second round?
Why include codecs that are not up to the competition if you're looking purely for the 'best'?

I'm sorry. I found this comparison useless for me. IMHO even Sagittarie's metric test is more real (viable).
As the FAQ states:
Q7: Your comparison sucks. Why don't you make a real comparison? I'll tell you what to do..

Please tell the guy who is holding you at gunpoint and forces you to read my article to let out his anger somewhere else. If there's nobody behind you, then ask yourself: who forced you to come here?

[Sagittaire]

The developers are well aware of the capabilities of the codecs and did much testing to come to the decisions about which sets of options to use.

I have a doubt when I see the setting used for libavcodec ... :)

[Doom9]

In fact in doom9 test there are something which is completely unusable : the speed test.How about speed at settings that represent a good compromise between quality and speed? In fact looking at some of the settings I'd even say those are pretty much the settings anybody should be using for day to day encoding. VHQ4, ESA me, 15 references, all those things just don't make a lot of sense.
I think that speed test must be do with:
(1) speed for exactly the same qualityWhich is just plain impossible? As you well know, there are many means of measuring quality, metric or not, and they all come to different conclusions given the same source. I could go down that line now and come up with some rather unfavorable conclusions about metric comparisons, but I consider it a matter of tact to not rip appart other people's work in this area.. in fact I try not to comment at all so to not try to influence people.

[superdump]

I have a doubt when I see the setting used for libavcodec ... :)
I wouldn't poke that fire. Though it would appear a couple of suboptimal choices were made there.

[Manao]

Actually, the only settings really hard to decide is deblocking : we don't have Doom9's eyes, so we don't know which settings will please him most, and between -4 and 0, it's really only a matter of taste. All the rest ( number of refs, quality settings... ) were made in order to give what the developpers think is the best speed / quality tradeoff the codec can achieve. With Ateme's codec, that tradeoff is best / 2 ref, with x264 it's subme 6 / 4 ref.

If we ( ateme ) had thought that 4 ref would have brought a better speed / quality tradeoff, we would have used them.

Coming back to what could / might be improved in the comparison : nowadays, codecs allow to reach a wide range of speed/quality tradeoff. We know you emphasize quality over speed, yet, even knowing that, it's quite hard to find a suitable tradeoff, i.e. the one that will please you most. Obviously, this year, speed was only taken into account for qualifications. Would have we known that, we ( ateme & x264 ) would have been able to slow down the codec a lot, increasing by quite a bit the quality.

This also would avoid ending up with ateme being 25% faster than x264, while that speed gain not being taken into account into the final rating of the codec. Would the speed have been the same, a tie might have happen.

[Sagittaire]

Speculations:

-> perhabs that with the same speed Ateme is better than x264
-> perhabs that with same quality x264 is faster than xvid


but it is not a criticism for doom9. Make a perfect test is impossible ... :)

[Doom9]

Obviously, this year, speed was only taken into account for qualifications. Not true, qualification and main round. And if the settings for SPR and Steamboy would've significantly changed from the Matrix3 settings with respect to speed, I tend to think I'd have picked up on that. Speed has never been measured for all three sources and I don't really have plans to change that. But, thanks to the advent of dual core chips and the fact that a codec never nearly maxes out both cores, I can use my PC and still get a rather reliable measurement of speed for the other two sources.

Would the speed have been the same, a tie might have happen.On the other hand you can now claim the title of fastest high quality AVC encoder, and the night scene results that made the difference might be due to a bug in psy3. I doubt I'd hear any complaints if the ateme encoder would've won.. so I'd be extremely careful in this area. It's quite obvious that anybody but the winner is happy at the end of the day, but the right thing to do is try harder next time.

I can spell out the speed table for you: ATI has the fastest AVC encoder, at abysmal quality. Ateme has the fastest AVC encoder and delivering high quality output at it. x264 is a good bit slower but still offers good speed and high quality output. In the end it's still up to you to decide which one you'll use.. I can live just fine with 40 fps but 50 fps makes me even happier in the light of the quality I have seen (night scene in SPR set aside).

[Revgen]

First of all, both Ateme and X264 were pretty much neck and neck except for one scene in SPR. I'm sure the Ateme devs will probably solve it since X264 didn't have the problem.

AVC is a codec that is still evolving and both X264 and Ateme are going to show some flaws.


Now for my opinion on dual-core.

I've always thought of Doom9 as being a cutting edge resource when it comes to video and multimedia. Dual-Core is the cutting edge right now when it comes to video encoding. If you're a video encoding enthusiast, a dual-core CPU is a must-have from now to within the next 6 months. Dual-Core CPU's are becoming more affordable every month they are on the shelves. And the Price/Performance ratio when it comes to video encoding is well worth it.


If this was a codec review intended for mainstream users ala PC World or PC Magazine then I could see your point when it comes to Dual-Core users. This is an enthusiast article and therefore uses equipment that enthusiasts have or are seriously considering to aquire.

[Manao]

Not true, qualification and main roundI counted the main round as qualification ( none of the codecs were present in the actual qualification round ).And if the settings for SPR and Steamboy would've significantly changed from the Matrix3 settings with respect to speed, I tend to think I'd have picked up on thatI didn't say the contrary. I've no problem with how you measured the speed, it wasn't the point of my comment. I rather wanted to emphasize that nowhere in the final round is a mention of encoding speed.
I doubt I'd hear any complaints if the ateme encoder would've wonOk, firstly it's up to you to be objective - you're the tester. We - the developpers - can only be subjective, however hard we try not to. Secondly, I think I raise a legitimate issue : when both speed and quality differ, only setting a tradeoff will allow to set a winner.

[Doom9]

I rather wanted to emphasize that nowhere in the final round is a mention of encoding speed.Am I the only one thinking that people would read the main round as well, seeing as it's the article that contains the most codecs and without reading it nobody really gets how a codec made it into the finals?
Or am I just thinking people will want to read what took me hours to write?
when both speed and quality differ, only setting a tradeoff will allow to set a winner.It's always been that, and I recall praising the ateme encoder for quality and speed in the conclusion of the main round.

[Doobie]

1. The purpose of this test is find best videocodec but with trade on quality/speed. It's good idea. All videos were encoded on dualcore PC. What percentage today of people who have a dualcore? [

I still have a 32-bit processor. But, dual-core is the future. And, we can hardly fault the man for not testing every kind of processor (and a thousand other factors that could affect relative speeds). "I know, A is faster than B on an Intel, but what about on AMD?" The codec comparison does give us at least a rough idea of what to expect on any machine.

I'm sorry. I found this comparison useless for me. IMHO even Sagittarie's metric test is more real (viable).

That's a bit harsh.

[lexor]

Ok, firstly it's up to you to be objective - you're the tester. We - the developpers - can only be subjective, however hard we try not to. Secondly, I think I raise a legitimate issue : when both speed and quality differ, only setting a tradeoff will allow to set a winner.
that point is not valid, yes maybe if you slow down ateme it will get higher quality, but you didn't, nor do I think you speed it up to make it qualify for doom9 comparison, you made it the way it is for whatever reasons you have not because doom told you he wants/likes faster codec (though I'm sure he as all of us likes faster encoders)

point being you made it run at that speed, bacause you (presumably) did testing and decided that's the best speed/quality trade off you can make, you don't get to complain (unless to ateme's testing team/management). in addition if you were to give that encoder to the public, that's the speed/quality we'd get, so doom's comparison is fully valid.

having said all that, I soooooo wish x264 would be faster. the speed on 1080i -> 720p @ 5000kbps was horrid (under 5fps) granted resize takes part of blame. or ateme available to public and even faster.

[foxyshadis]

Only thing I wish is that the comments on each codec were available in a list, so that I don't have to click on every one each scene to get to them. ^^; But it's your comparison.

Oh, and some of the cornier taglines were real groaners. :p

I'm really surprised that grain introduction isn't a part of the AVC codec post-processing (lavc or ateme), and even more, a part of the AVC standard itself. Add a rough calculation of the type and amount of grain to each frame header. Obviously HDX has made great strides in hiding smearing behind grain, even if the encoder is immature, it'd be really cool to see how AVC could be improved by it. (I can never get ffdshow's grain filter to give me pleasing results.)

[lexor]

I'm really surprised that grain introduction isn't a part of the AVC codec post-processing (lavc or ateme), and even more, a part of the AVC standard itself. Add a rough calculation of the type and amount of grain to each frame header. Obviously HDX has made great strides in hiding smearing behind grain, even if the encoder is immature, it'd be really cool to see how AVC could be improved by it. (I can never get ffdshow's grain filter to give me pleasing results.)
for the love of might, film grain is an artifact, its the friggin graining of the friggin film! why in blazes would anyone want to put noise/garbage in their movie? with all the filter and whatnot we have in avisynth to get rid of just such garbage.

[foxyshadis]

Because it helps to hide oversmooth encoded/filtered areas, and it gives a grittier, somewhat richer look to live footage. That's the reason it's added to films. (And just because I like it, in moderation. ^.~) But grain/sharpness/smoothness is an extremely subjective subject that comes up now and then in the avisynth forum, I won't get into it.

[zambelli]

I'm really surprised that grain introduction isn't a part of the AVC codec post-processing (lavc or ateme), and even more, a part of the AVC standard itself. Add a rough calculation of the type and amount of grain to each frame header. Obviously HDX has made great strides in hiding smearing behind grain, even if the encoder is immature, it'd be really cool to see how AVC could be improved by it. (I can never get ffdshow's grain filter to give me pleasing results.)
IMHO... Removing grain at encode time and then adding it in decoding is a nice trick, but I wouldn't give it any more credit than just that - a "trick". You're taking information that's an integral part of the analog source, destroying it, interpolating information that it might've been hiding, then showering the image with random noise in an attempt to restore the removed information. While I'm not questioning the method's effect on perceived video quality, objectively it's very dodgy and will probably not help any codec in a direct frame-by-frame comparison such as Doom9's.

[charleski]

Because it helps to hide oversmooth encoded/filtered areas, and it gives a grittier, somewhat richer look to live footage. That's the reason it's added to films. (And just because I like it, in moderation. ^.~)
As someone coming from an analog photography background, where grain was a necessary physical evil that had to be fought for decades, this affectation makes me mad :). Grain is noise, that's all there is to it. Grain was often accepted because the processes that produced it also produced an increase in critical contrast levels that would allow a sharper image, or one that could be exposed acceptably under lower light levels (two sides of the same coin). While there are still situations that will produce visible grain using modern film stock, those situations are rare, and most of the 'grain' you see nowadays is nothing more than posturing.

[zambelli]

Question for Doom9 (out of curiosity, not criticism :) ):

I notice that most of the "Steamboy" screenshots look like they were taken from dark, low motion scenes. Am I mistaken? Why no screenshots from high motion scenes such as the "wheel chase" (roughly 30k - 34k frame range in NTSC version)? I remember some of those scenes where he's spinning within the wheel really giving the codec a run for its money.

[Doom9]

actually, I had a long look at the chase scene, and it's not difficult enough.. I want to see a bunch of artifacts. But the first review scene includes a nice explosion, the last scene a whole battle.. in the end there's enough action.

I'm looking forward to the day all movies are shot completely digital and I hope that this will mark the end of all grain. Whenever I'm in the movies, I wish I could get rid of the grain... when I look up to the sky there is no grain.. when I look at a human face, there's no grain, when I look at the walls in my room there's no grain, when I'm being chased by 20 cops in NFSMW, there's no grain (but a very beautiful high res image).. there's no grain in real life so there shouldn't be in movies either.

Grain removal and re-adding is an optional part of the AVC HP specs.. but it's probably not that easy. I'd much rather have grain-free movies..

[lexor]

yeah, I noticed that too zambelli, my monitor is callibrated for colour correction for my photography editing work, so it's not brightness or contrast on my end, the source itself is dark. some of them I thought there was nothing but black with like a few areas (<10% I'd say)worth of light objects. Why would you compare something that no one would see? I mean unless there are colour encoding problems who's going to notice blocks/atrifacts of black on black?

[*.mp4 guy]

The problem is most likely that the pictures are on a white background, try veiwing them fullscreen in a mediaplayer and they should look more normal. All the same since most people would never bother to do that perhaps brighter screenshots would be a good idea.

[zambelli]

actually, I had a long look at the chase scene, and it's not difficult enough.. I want to see a bunch of artifacts. But the first review scene includes a nice explosion, the last scene a whole battle.. in the end there's enough action.
I won't argue with that since it's a pretty subjective thing. I'm sure we all have our favorite frames that we keep coming back to when we do codec comparisons. I liked the "wheel chase" scene a lot because it was bright and contained some unusual motion (such as 360º frame rotation).

Whenever I'm in the movies, I wish I could get rid of the grain... when I look up to the sky there is no grain.. when I look at a human face, there's no grain, when I look at the walls in my room there's no grain, when I'm being chased by 20 cops in NFSMW, there's no grain (but a very beautiful high res image).. there's no grain in real life so there shouldn't be in movies either.
That's all true, but also note that your vision can handle more than 24 fps, captures more than 100 degrees of view (plus peripheral vision), always processes in color, doesn't compress data and always gives you the same color calibration. :) However, this is not how we necessarily expect film or video to look. Purists might argue that films should look exactly like real life because they're reflection of real life, but I think most filmmakers would argue that the very imperfect nature of analog film technology is "perfect" for filmmaking because it produces a more dreamlike vision of life and allows the viewer to draw the line between film and real life. It's more of an artistic choice these days than a technical one, I think.

[foxyshadis]

I'd be happy with no grain if the detail was kept as crisp and lifelike as in real life, or even immensely high-bitrate mpeg2. HD resolutions and bitrates will certainly help. But for now it is a "neat trick" to increase percieved quality in smoothed out scenes, much like putting a sharpen filter in the decoding chain (although limitedsharpen is a bit more than a trick).

Thanks for clarifying about the specs, doom9.

[Mug Funky]

for the love of might, film grain is an artifact, its the friggin graining of the friggin film! why in blazes would anyone want to put noise/garbage in their movie? with all the filter and whatnot we have in avisynth to get rid of just such garbage.

it's a legitimate part of the signal!

there's bands that artificially add vinyl surface noise to their recordings, and it's generally agreed upon that it does in fact add something to the music.

it is the same with film - the choice of film stock and processing is often an artistic decision that should not be messed with. look at SPR as a classic example... you think given the budget they had they could afford to shoot 70mm, complete with insane amounts of lighting and a really slow ISO rating, so everything they shot looked absolutely smooth... so you must conclude that the grain/handheld style/fast shutter/high-speed film was intentional. i suspect a lot of grain was in fact added in post. then there's movies like "traffic", where the choice of stock reflected not only the location (mexico was lo-fi and pushed, america was very smooth and pulled) but the changing moods in the film. it's not our place to change those decisions. that's the reason grain addition was included in h.264 - you can get compressibility and grain as well.

true enough, a lot of sources can do without grain (anime, not counting some of the cyberpunk stuff), and it's usually unintentional (especially with older films/anime/whatnot), but it is an incomplete encoder that can't handle a little noise.

now if the bandwidth is extremely limited, then noise reduction is desirable. or if you're doing restoration work and the director/project manager specifically wants the noise removed, then obviously it should be done. but i think it's dangerous to develop codecs using nothing but smooth, clean sources, because when noise comes along, it's going to bite in the proverbials.

[edit]

@ doom9:

human eyesight has quite a bit of grain... but there's a good postprocessor working in the chain as well. true, nowhere near as much as a 35mm print, but i wouldn't let that spoil the movie experience (there'll be plenty of stuff in the average cinema to do that anyway - i've seen people answer their mobile and actually start chatting).

[CruNcher]

We all know what happens if a encoder can not handle Filmgrain the scene beginns to life hehe. :P I see it as artistic thing the same as zambelli and Mug Funky certain directors wan't this unnatural look because it shouldn't be a documentary it's a Movie even Lucas added Noise to certain Scenes in all Episode 1 2 3 it's also often used to cover CGI Scenes and if a Encoder can not preserve that noise it's at risk to be taken seriously by Hollywood same as happened in July when H.264 was at risk being chosen as a official Blu-Ray standard, because of that matter High Profile came along and everyone was happy, or maybe not. ;)

[DigitAl56K]

As the product lead for the DivX codec, I'd like to throw my $.02 into the discussion. In fact, I'd like to do so in part by sharing my experience of the codec comparison with an underlying goal that is not to dispute the results, but to inspire further thought around the way they are conducted, the comparisons drawn, their presentation, and aspects of responsibility that are involved.

Firstly, let me begin by thanking our host, Doom9, for the opportunity to participate in this years comparison. Those who followed the event from the beginning will be aware that DivX pre-qualified for round two this year based on past performance. Doom9 was also quick to respond to our e-mails, accepted a small correction to our settings slightly after the deadline, and allowed us to rebuild our decoder after finding a bug in the post-processing. I'm sure the same courteous attitude was shown to each of the participating developers. However, preparing for a codec comparison is a daunting task, no matter how gracious the host!

Like most developers, DivX received notification of the comparison two weeks prior to its beginning. Now, two weeks sounds like a lot of time to prepare for a codec comparison, but in reality things are not quite that easy. The infamous Gej, who normally helps out with such things, was flying abroad and so the task fell upon myself and just half of the codec team (the other half being entrenched in core technology research). At the time we were working hard to release DivX 6.1, develop components for our software partners, and to provide support other DivX projects such as the Radium Player, Converter 6.1 preview, and Dr DivX 2 OSS. Many people were also out on vacation for the holiday season and so in the end we had only a few days to devote to this years comparison, but such is the way of things. The real problems came in knowing what to shoot for in terms of quality and performance.

Similar to several of the other codecs, DivX 6.1 can be configured to encode at literally hundreds of frames per second in the "Fastest" and "High Performance" modes, sacrificing quality for performance, or at tens of frames per second with a variable emphasis on quality (we actually think that contrary to the note in the comparison DivX 6.1 High Performance mode outperforms Nero ASP in terms of quality and performance, but I digress... ;)). Although Doom9 had indicated that he would probably bias quality over performance, we were at a complete loss with regards to where we should aim for, given that the only strict rule we had to work with was >=10fps. On an AMD X2 4600+ we would have struggled to do less than 10fps, so this really offered very little guidance. One of the outcomes of developing our encoder settings so late in the game was that we were able to see the results of the round one qualifications, and so we could approximate the kind of performance/quality trade-off as applied to the XVID codec, which is a good reference for the ASP codec class. Here we judged that this must be where Doom9 personally considers the balance of encoding rate versus visual quality to be optimal and so we derived settings for the DivX 6.1 encoder that offered similar results in both respects - it was the only real indication we had to go by. We also considered possible AVC codec configurations - surely these would not use near-optimal settings and risk a severe performance drop with respect to the ASP codecs?

In round two, providing comparable settings to XVID paid off. DivX came top two of all the ASP codecs, qualifying for the final round where we had also provided settings in the same performance/quality bracket, expecting similar metrics to weigh in on the final judgements. Unfortunately, it seems that performance was not really an issue at all here, and as Manao already commented there was no mention of performance in the final round results - they tended to focus solely on quality. DivX placed 4/4 in the final round results - surpassing the quality of the XVID codec on 1/3 occasions. We had expected either a 1/3 or 2/3 outcome due to the fact we had deliberately positioned ourselves closely with XVID. Given that every developer had the challenge of picking a fairly arbitrary performance trade-off, if the encoding settings for any of the encoders had varied slightly the results could have looked very different, not only for DivX but also for Ateme. Perhaps they would have looked exactly the same. We don't know. The problem is that it's not really possible or valid to compare codecs when each encodes at a different rate - and at any fixed encoding rate the results may tend to bias one group of codecs over another (e.g. ASP will likely outperform AVC if the encoding rate is fixed relatively high). A true analysis requires analyzing results for hundreds of different tests using different sources and configurations. We have built systems that run so many metric tests each day on our own codec alone it would take hours to perform a comprehensive review of all the results on a daily basis.

A misconception also arises when you believe that in the two week preparation period we had time to analyze the competition in depth. If you have ever even attempted your own codec comparison you will know that to be thorough you could spend an entire two week period examining a single codec encoding with various combinations of content types, resolutions, bitrates, settings, pass modes, performance constraints etc. and still have absolutely no idea where the competing developer might have chosen their performance/quality balance, given they had as much information as you did, which is not very much ;) To add to the long list of reasons why performance in the final round is such an important factor, we had launched DivX 6.1 the week prior to the comparison with full HT, SMP, and dual core processor support. Because more time is spent in the encoder in the higher quality modes each time the quality setting is increased on a dual core machine, like Doom9's X2, you get less performance penalty per db PSNR gained (or visual quality in general, if you don't like statistical metrics) than you would on a machine with a single logical processor. We had chosen a combination of Balanced/Better quality modes for multipass and had foregone using Extreme quality mode, Insane quality mode, and H.263 Optimized quantization - each of these increasingly benefiting from multiprocessing (plug: see pretty charts on divxlabs.com for details).

I'd like to say that I'm also surprised HD content did not play a role in this years testing - typically I've found that performance is a critical factor for both HD encoding and playback. I also find that the effect on quality metric is very different for ASP vs AVC with HD sources - ASP tends to perform extremely comparably but with much better performance.

So what's the underlying message to this story? First, it's not possible to take a set of codecs, suggest guidelines, test only 3 pieces of content and create a good comparison. None of the developers can hope to do much better than taking an educated guess not only at what settings to provide based on the guidelines, but what the competing developers might do, what the metrics will be for the results, and how they will be messaged - which brings me neatly to my second point: The way results are presented is critical.

Today, a long time member of the DivX forums posted this in a message: "Doom9 says DivX sucks, compared to other solutions". You may laugh but this came from someone who is always polite, intelligent, and thoughtful in his posts (naming no names of course, leadman584 ;)). To read the conclusions page of the final round one might think DivX came last in the codec comparison, yet it had already been pre-qualified for round one, placed top two for ASP in round two, and outperformed XVID in 1/3 tests according to Doom9. I understand that Doom9.org as a community tends not to cater to those who do not care to spend the time reading the materials provided for them thoroughly, but there is a certain journalistic responsibility to be assumed when such a significant member of the digital video field presents what has turned into a celebrated annual event among enthusiasts. After reading the comparison I realized that I myself, as someone who regularly analyzes various codecs, had paid little attention to the codec configurations used in the testing - and even if I had I am not necessarily even familiar with all of the settings each codec actually makes available as to be able to judge it's potential with respect to the actual results illustrated. It would also have been unlikely that I would make a mental correlation between encoding rate and the quality of the screenshots presented for each codec even had the timing been given. Because the comparison was split into three parts, although an entertaining presentation, if I put myself in the shoes of the typical reader I would not understand that DivX did not come last, it came second top in the ASP category.

Aside from all of this, there are fundamental problems with codec comparisons with respect to use cases. In this comparison, like the other codecs, DivX was configured in an entirely unconstrained manner - we had to use settings that we could absolutely not recommend to the general consumer in order to ensure the right performance/quality balance with the competition. Keeping the marketing to a minimum, everyone here should be familiar with the fact that we've spent several years now developing support for DivX in consumer electronics devices such as PMPs, DVD players, and so forth. To achieve this, constraints are normally placed upon the encoder to ensure the bitstreams generated do not exceed the capabilities of various device categories, often restricting features such as QPel or MPEG quantization, as well as placing far more stringent demands upon the rate control algorithm - an area that caused many other codecs in the comparison to be disqualified in earlier rounds. To this end DivX is optimized for it's intended primary use case - maximum quality at flexible performance points while maintaining compatibility with what we term the "DivX ecosystem". Although we've worked on QPel and MPEG quantization in the 6.1 release, expecting comparisons such as these to take place, this is not where our attention is centered through much of our development cycle. This means that placed in an environment where the fundamental use cases of the codec are disregarded there will likely be unfavorably biased results. Consider: Is a videoconferencing codec poor quality because it doesn't compress movies very well? Is MPEG-2 video inefficient because you only test it at low bitrates? Such comparisons also fail to reflect upon formats as a whole: I personally think that DivX 6.1 codec gives excellent quality video for DivX Certified devices, it's important to me that you can use it in the new DivX Media Format with all it's cool features, and I find many of the yet unreleased internal projects we are developing very impressive. Of course I am biased :) Perhaps though, when we compare elements of a format, such as a video codec, they need to be put into context. Would it not be more meaningful to say "XVID and DivX both rank very highly if you're looking for a codec that works in most DVD players today" than to say "XVID and DivX came third and fourth"?

I'll end with with these thoughts: Please don't try to deconstruct this post piece by piece. Here's a reminder: The intention is to inspire further thought and discussion around codec comparisons in general. Granted, I am not the most eloquent of writers, and it might be easy to mistake some of the points raised here for nitpicking and criticism - that is not the intention. I think Doom9 deserves a lot of respect for the dedication he shows the digital video community and the effort he placed in this comparison and those gone by in previous years. I discuss this from my perspective at DivX because this is the only perspective that I have on the subject.

Some things to think about:

How many tests must you run with different codecs before you have something resembling representative results? How many sources must be used, how many permutations of settings, how many permutations of bitrates, how many permutations of resolution, how many permutations of hardware before you can really say X is better than Y?
How do you test each codec with respect to it's intended use case, whether it be videoconferencing, movies, anime, general purpose, low resolution, HD, etc. - and then compare that to other codecs with different primary use cases? How do you put a codec in context of the format it belongs to as a whole and explain the benefits of that format with respect to the benefits of formats to which the other codecs belong?
How do you ensure developers know what mark to target with their recommended settings? How do you know the resulting grouping is close across different developer submissions? How do you trade off performance and quality - especially when performance changes across codecs and across hardware?
In your conclusions, can you somehow resolve all of the above questions?
How can you prevent the general public, and even the average enthusiast, from simply reading your personal conclusion, looking at a set of screenshots, and walking away with no impression other than the one immediately presented to them? Is it possible?
Fundamental question: Are public codec comparisons really a good idea? My own opinion is that they are not. I believe it takes a specialist to determine appropriate test criteria for a single given use case, analyze codecs within that very strict set of criteria, to interpret the results correctly, to apply that interpretation to a given use case, and to avoid applying it generally.


Have fun :)

[Revgen]

I believe that Divx 6.1 is a great codec when it comes to the mainstream users. It's also great for compatibility with set-top devices.

If a friend asked me for advice on what codec to use I'd probably recommend DIVX because it would be easier for them to use. For myself, I'm always going to use X264, since the quality is great and the performance is good on my AMD X2 4600+.

However, I can understand your concern as far as public codec comparison's go. Someone may read Doom9's comparison and automatically conclude that "Divx Sucks" even if they haven't fully read the review and understand what Doom9 focuses on.

But, as I've mentioned in my previous post, Doom9 is an enthusiast community. Codec comparison's are done with a video enthusiast in mind. Not mainstream users who may be interested in compatibility, ease of use, and other similar features.

It's not Doom9's fault if people don't understand the context of how these reviews are done.

[DigitAl56K]

It's not Doom9's fault if people don't understand the context of how these reviews are done.

Very true, but this does not mean you can't be conscious of the way most people will read (skim) your publications and counteract accordingly. The problem is not with Doom9, but unless you address the problem a lot of people will misinterpret what you're saying. If most people are going to take away incorrect conclusions are you doing a good or bad thing by publishing it? You are only misinforming the majority of the people, albeit not maliciously. Is that the intention of such a comparison? Surely the intent must be the opposite, or else why not let the majority of the people do their own comparison? :)

[*.mp4 guy]

All that stuff in post #33. (http://forum.doom9.org/showthread.php?p=758300#post758300)
I think the point of the speed/quality trade off was to get the codec developers to give settings that would hit the "sweet spot" where quality is almost the best it can be, but speed doesn't suffer to much. If speed wasn't a factor X264 would have been using esa search and Divx would have been using Insane mode and Ateme would have been using -qual full. If that happened the encoding could have taken several weeks, which obviously would be ridiculouse. So then where do you draw the line? You obviously don't want to aim for the all out best quality or best speed, because the results would be ridiculouse. so the only real option then is to let those most qualified people (presumably the developers) decide where the speed quality trade off should be set.

Public comparisons are inevitable, and with the rabid marketing departments most (all) companies have any company sponsored comparisons would be a joke. So then you have the media, tv isn't interested, neither is radio, so that leaves the magazines, but magazines arent always run by the most knowledable or unbiased people. So really if you want a codec somparison you either take the wattered down possibly inacurate version you may be able to find in a magazine, or you go looking for something else. Doom9's comparisons are the best something else out there.

[CruNcher]

DigitAl56K from what i read i could think it would have been better DivX Inc put DivX Q in the race instead of DivX 6.1 ;) and about the so called "DivX ecosystem" i'am a happy owner of a Medion 7457 DivX Player it uses the MediaTek MT1389 Chip it is DivX Certified and Technicaly should be able to play DMF with the recently released new MediaTek firmware, but the problem is i don't know what the Reseller in this case Medion has for a contract with the Player Developer in this case MediaTek and most likely abondoned the support for the Player long time ago now im the dumb idiot customer who bought a throw away Player. I would have no problems investing some money for the new Firmware because the DMF support MediaTek also has to pay to you but this is no option that is given to the customer. Do i really have to buy a complete new Player with a updated Firmware now ? that you call "ecosystem"
i call this throw away system, world polution in the name of DivX Inc :P

[DigitAl56K]

CruNcher: It will still play your DivX files, just without some of the fancy new DivX media features (and some of the new features even work on most of the old players). But this is off-topic to an extent ;) The codec is compatible across the ecosystem, DivX media is backwards compatible such that the main title will play in older devices. The point I was making is that DivX codec is only one facet of DivX. Similarly, there are other codecs with a larger format surrounding them that should also be considered.

P.S. Not many formats add a ton of features and still work in older hardware :)

Alrighty - bed time for me. Hopefully there will be some interesting discussion here in the morning rather than random flaming ;)

[CruNcher]

Yeah but i would like to pay for the Fancy stuff but not the entire Price of a new Player jesus :P i know you as a Marketing Boy wont give me an answer about this as i myself know how this taiwan stuff works 6 months later you can forget any support but i call this a catastrophical system and DivX Inc works inside this System. I gonna wait and also will tell this to everybody i know that they should avoid Mpeg-4 ASP throw away Players and better wait for a well defined Industry supported solution like HD-DVD or Blu-Ray.
This is no Flame war just hard facts.

[Archimedes]

Fritz Framalyzer's results of the codec shoot-out.

The results are based on the ultra high bandwith images. All png images where first converted to the bmp format and then analyzed with Fritz Framalyzer. The values (Q in Y) are based on the well known SSIM metric. Higher values means better quality.

May be a test for the SSIM metric. ;-)

(Edit.)

[Sagittaire]

I'd like to say that I'm also surprised HD content did not play a role in this years testing - typically I've found that performance is a critical factor for both HD encoding and playback. I also find that the effect on quality metric is very different for ASP vs AVC with HD sources - ASP tends to perform extremely comparably but with much better performance.

You are sure for that : personnaly in my test AVC is better than ASP with 2dB on OPSNR test.

Ateme Main Profil (Nero)
http://multimediacom.free.fr/Download/NDAVCMP_IceAge_720p_1250.mp4

Ateme High Profil
http://multimediacom.free.fr/Download/NDAVCHP_IceAge_720p_1250.mp4

x264 High Profil
http://multimediacom.free.fr/Download/x264MP_IceAge_720p_1250.mp4

x264 Main Profil
http://multimediacom.free.fr/Download/x264MP_IceAge_720p_1250.mp4

XviD MPEG4 ASP
http://multimediacom.free.fr/Download/XviD_IceAge_720p_1250.avi


AVC outperform ASP visually too and by far ...

[Doom9]

The problem is most likely that the pictures are on a white backgroundThat's just it... I noted that too when I wrote the article.. the Steamboy shots are not impressive at all. But watching the video on 23 inches, things change a lot. When I first watched Steamboy I made a mental note to include the chase scene as a review scene. However, when I went back and looked at the encoded video, I suddenly realized that this scene isn't that interesting from a "finding imperfections" point of view

[Sagittaire]

Anyway Steamboy is a very good choice for anime test : image is perfect and beautifull for this movie.

[Doom9]

Now, two weeks sounds like a lot of time to prepare for a codec comparison, but in reality things are not quite that easy. I fully appreciate that.. it is not much different with the time allotted to fix things. It also isn't much different on my end.. I have at the maximum two weeks, and over Christmas nobody is around so unless I spot problems early on, I'm screwed as well.
"XVID and DivX both rank very highly if you're looking for a codec that works in most DVD players today" Is untrue. I make it a habit of browsing to the largest CE store in a 50 km radius (it's right next to the highway and on my way to work when I take the car) to get an overview of what products are being sold... a few days before Christmas, from the 20 or so DVD players they had available, I spotted 3 that handle DivX.. the rest were plain old DVD players. I know given your background that's one issue that's important to you, but, Nero has the exact same background, and back in the day it took a lot of convincing for being able to take the codec out of its environment. A year from now you can argue that AVC and VC-1 equals hardware support because of the advent of HD DVD and Blu-ray players.. quickly turning the plate around.

And as far as the context goes, I already told Al, I don't like to repeat myself. If somebody reads the comparison and thinks, why are there only 4 codecs, then he stupidly skipped page one where it's made very clear that there are more codecs, and didn't bother to read the FAQ (where it is also explained that there are different rules this time round). The new three round system, for the first time ever, allowed the inclusion of a whopping 16 (even 17 but I never put the ATI shots in since I only get the encoder after the main round had started) codecs, almost twice as much as in every previous comparison, the main round was the largest ever, and the review was the most detailed ever.. last year I spent about 3 hours on SPR over 9 codecs, this year it was 3 hours on 4 codecs.

Perhaps we've become a society where reading no longer matters, but I refuse to give in to the TV mentality and have everything pre-chewed for you. When I read my favorite PC mag, they also quickly refer previous articles and don't go into a half page summary of the previous article.. either you bother to read up, or you're just being left in the cold. This forum is based on the same principle.. we even have a rule to enforce it. DivX is the second best ASP codec in this comparison, the main round conclusion says so, but it's also number 4 out of 4 of the best four codecs.. both statements are true, and they need to be seen in context to one another. With HD DVD players we may even get into a situation where hardware playback equals AVC, not ASP.

And as I said when I answered the first questions from your side, it's about PC playback.. hardware playback is nothing more than a sidenote, and you could've imposed other requirements for the specs that wouldn't put all those limitations upon ASP hardware playback. In a way I'm looking forward to the HD formats because they are unlikely to restrict AVC the way the DivX certification restricts ASP on hardware.

[lexor]

I also don't like argument about 2weeks notice being short, you aren't supposed to fix things at this point, you supposed to give input on settings. If doom9 found something wrong that means it's also wrong for all your customers, and either you didn't follow through bug reports or your QA dept. missed it. What you mean to tell me if doom didn't point out the flaws they wouldn't have been fixed? IMHO he shouldn't have allowed for fixes to be sent in at all, since that's not what customers are getting from any of the participants, so the review is flatterring to all devs, the customers don't get quite the quality and support doom9 enjoyed (or not).

[vtarpara]

This was a really detailed comparison,(props!), but after reading the posts it has confused me regarding why AVC was developed in the first place. It is my understanding that Mpeg-4 AVC would replace Mpeg-2 in HD-DVD and Blueray systems allowing bitrates of HD videos to be the same as their DVD mpeg-2 counterparts. What this comparison seems to indicate is that ASP for the moment is prefered not only for equivalent quality reasons but more so because of speed. So then why is ASP not pushed more by the studios? Shouldn't AVC quality be equal to the original Mpeg-2 (the screenshots clearly show otherwise)? One thing I hate with mpeg-4 codecs is that alot of face and texture detail is lost when compared to the mpeg-2, from the hype, I thought AVC was supposed to take care of that compared to ASP. Another issue is about the scalabililty of ASP vs. AVC. With Divx touting itself as an HD AVC alternative quality wise, does HD content size differ significantly if optimal bitrates for both ASP and AVC are used? Also how do the various codecs handle low bitrate content? With PMPs(and hopefully the 6G iPod) able to play most container formats, are there significant differences between AVC and ASP that would result in a dramatic filesize reduction between the two while maintaining equivalent quality?

If I were to answer my own questions and doubts, I would say the following:

1. Both ASP and AVC are capable of mpeg-2 level detail and quality if encoded from the primary digital source i.e. HDCAM, DVCPRO HD, Cineon.
2. To acheive mpeg-2 level quality, a higher bitrate would be required i.e no 700MB CD size restriction for the video.
3. ASP would require slightly (possibly negligible) increase in bitrate relative to the capacity availible on DVDs, BD-roms, and HD DVDroms to match the quality of AVC
4. Facial and texture detail is lost in these comparisons because you are recompressing a highly compressed video stream incurring digital generation loss (degree of loss depends on encoder).
5. With regard to the next-gen homevideo footage (progressive/non-interlaced), ASP will be likely be prefered from a home-user perspective due to its speed.
6. When dealing with high-capacity formats such as blu-ray and HD-DVD, the ASP vs. AVC debate isn't relavent.

The six points above are simply assumtions that I want clarified because of my confusion. I maybe missing the point completely regarding the intended use of the codecs. Doom9 or others, can you give me any insight? Thanks in advance!

[buzzqw]

1. Both ASP and AVC are capable of mpeg-2 level detail and quality if encoded from the primary digital source i.e. HDCAM, DVCPRO HD, Cineon.

Yes

2. To acheive mpeg-2 level quality, a higher bitrate would be required i.e no 700MB CD size restriction for the video.

False. To achive "mpeg-2 level quality" a lower bitrate is needed, and even lower for avc. Somewhere is written that over 2000kbs for avc is a overkill, so much lower of typical 8000 for mpeg2

3. ASP would require slightly (possibly negligible) increase in bitrate relative to the capacity availible on DVDs, BD-roms, and HD DVDroms to match the quality of AVC

Yes. Not so slightly

4. Facial and texture detail is lost in these comparisons because you are recompressing a highly compressed video stream incurring digital generation loss (degree of loss depends on encoder).

Yes (and depends on bitrate also)


5. With regard to the next-gen homevideo footage (progressive/non-interlaced), ASP will be likely be prefered from a home-user perspective due to its speed.

True/False. The pc performance are always increasing. Is possible that today is preferred asp but in future avc. Pn a typical future pc, will run at same fps as asp.
(like old divx3 to xvid)

6. When dealing with high-capacity formats such as blu-ray and HD-DVD, the ASP vs. AVC debate isn't relavent.

Yes. The capacity of this formats is enough ... BUT don't think always at dvd resolution try the HD resolution (till 4k*2k)... at this width/height even a hdvd could be filled up

All is imho !

BHH

[vtarpara]

I should've clarified. When I said "higher bitrate" i mean't it should use a higher bitrate than that used in the codec comparison methodology. The comparison chose the highest bitrate that would fit in a 700MB CD. When dealing using codecs for archiving on a DVD, a 700MB limit wouldn't apply. I didn't mean to imply that mpeg-4 bitrates would be equal to those used for standard-definition mpeg-2. Obviously you wouldn't want to use over 2mbps for AVC or ASP when dealing with standard def content, HD is another matter, that would probably be between 2-10mbps depending on resolution and framerate of the source. Thanks for the clarification!

On a separate note, will the next-gen players support ASP? I read in these forums that AVC support doesnt necessarily imply ASP files will be playable even though ASP is easier to decode.

[Manao]

So then why is ASP not pushed more by the studiosBecause ASP came too early, and because the gap between ASP and mpeg2 isn't that wide.What this comparison seems to indicate is that ASP for the moment is prefered not only for equivalent quality reasons but more so because of speedThe source was encoded with a mpeg2 codec. That bias the comparison towards ASP because it looks the same way mpeg2 does ( since they share the same DCT ), while AVC definitely looks different. However, from a real source, the difference in quality might ( and I say might because I have actually seen the proof yet, lacking of unencoded material ) be quite bigger than Doom9's test showed.One thing I hate with mpeg-4 codecs is that alot of face and texture detail is lost when compared to the mpeg-2, from the hype, I thought AVC was supposed to take care of that compared to ASPIt was encoded at very low bitrates. Much lower than anything that'll be used for HD DVD.Another issue is about the scalabililty of ASP vs. AVC. With Divx touting itself as an HD AVC alternative quality wise, does HD content size differ significantly if optimal bitrates for both ASP and AVC are usedLet's throw wild figures without anything to back them up ( from lack of good HD sources ) : i'd say 15 to 25% filesize reduction with AVC, in comparison to ASP.Also how do the various codecs handle low bitrate content?There AVC is quite better than ASP, it's perhaps the only ground on which everybody agree.With regard to the next-gen homevideo footage (progressive/non-interlaced), ASP will be likely be prefered from a home-user perspective due to its speed.Speed. Indeed, at the present time, speed is an issue. Yet, x264 isn't even half a slow as XviD, so the gap, encoding wise, is rather small. Decoding wise, the hardware should be able to decode AVC 1080i at high bitrate, so that won't be an issue ( it is now, but in a matter of month, it won't be anymore ).1. Both ASP and AVC are capable of mpeg-2 level detailAll codecs can achieve that level of details, provided they are given enough bitrate. In fact, AVC can even provide much more details ( lossless mode, and lossy modes that can go quite high in PSNR - and in bitrate too... ). However, AVC can provide that level of details at lower bitrates than ASP and mpeg2.

[DigitAl56K]

Perhaps we've become a society where reading no longer matters, but I refuse to give in to the TV mentality and have everything pre-chewed for you. When I read my favorite PC mag, they also quickly refer previous articles and don't go into a half page summary of the previous article.. either you bother to read up, or you're just being left in the cold. This forum is based on the same principle.. we even have a rule to enforce it.

Yes, again I don't think you are at fault - the question is how far do you go towards helping those that are. Obviously we have different viewpoints on the matter and that's OK :)

lexor: We had only just released DivX 6.1 at the time of the comparison. In almost every new release of such significance problems crop up - QA, no matter how extensive, can only catch so many problems. If you look around the DivX Encoding forum you'll actually find that with the help of the forum members we've tracked down a couple of issues that we found very difficult to replicate here due to hardware despite the fact we were given step-by-step instructions on how to do it. We're working to fix all of the issues that have been reported since the release in an upcomming post on divxlabs.com . Having said all of this, the reason that I illustrated this point in my post, even with the understanding that we were likely to get questioned on it, was to highlight that although I do not necessarily think codec comparisons like these are necessarily a good idea, I do very much appreciate the time Doom9 invested in working with us and his positive attitude despite one or two little problems.

Once again, the point of the post was to discuss codec comparisons in general, it seems that we meander off-topic a little ;)

[Doom9]

You're always going to lose detail when compressing a source. For a really good comparison, I would be able to use uncompressed sources, but that content is the holy grail and Hollywood will rather burn itself to the ground than ever spread such content.

[*.mp4 guy]

The source was encoded with a mpeg2 codec. That bias the comparison towards ASP because it looks the same way mpeg2 does ( since they share the same DCT ), while AVC definitely looks different. However, from a real source, the difference in quality might ( and I say might because I have actually seen the proof yet, lacking of unencoded material ) be quite bigger than Doom9's test showed.
I would agree with you except I have re-encoded quicktime hd h.264 tralers and they give more or less the same results as dvds do.

[Doom9]

except I have re-encoded quicktime hd h.264 tralers and they give more or less the same results as dvds do.But do we know in which format Apple gets those trailers?

[CruNcher]

You're always going to lose detail when compressing a source. For a really good comparison, I would be able to use uncompressed sources, but that content is the holy grail and Hollywood will rather burn itself to the ground than ever spread such content.


Did you ever tried to get in touch with a Studio maybe they give you a license for a trailer for scientific purposes, who knows. I know i most probably dream but hey trying costs nothing :P I wouldn't ask Universal, Warner or Sony tough :P maybe something more with a family attitude Lionsgate maybe ? hey im sure Boll KG is not so hard and would agree giving you some Trailer from Alone in the Dark, Bloodrayne, or maybe the new Dungeon Siege in Source Form, would be good Advertising also for them, complete Source of a Film im in doubt but who knows :)

[zambelli]

Doom9:

I'll take the opportunity to chime in with some feedback on my particular experience with this year's shootout (VC-1 submission) and some suggestions for next year's shootout.

First of all, kudos to you for organizing the codec shootout and spending 2 weeks of your winter holidays encoding movies and having "fun" with pre-release quality software. :) The hard work is much appreciated by everyone, I'm sure. This codec shootout is your project, so it's only fair that you get to determine your own metodology. Personally I don't look at the shootouts as absolute proof of one codec's superiority over another, but I appreciate the independent nature of the comparison and I think it gives a good general idea of what to expect from each codec in terms of quality/performance.

I'd like to second some of DigitAl56K's concerns, particularly the one about the downside of organizing the codec shootout around Christmas holidays. During the second half of December the Microsoft corporate campus in Redmond is half-empty, as I'm sure is the case with any other corporate environment in America and Europe. Most people are on vacation, many aren't even checking their email. If some devs are working around Christmas time, there's a strong chance they're already busy working on some high-priority project and can't afford to spend time fixing other new bugs. 48 hours might seem like a lot of time to fix 1 bug, but when your devs are working on high-profile projects such as Vista and HD-DVD - it's next to impossible to pull them away from whatever they're doing and make them investigate and fix a bug for a comparatively low-priority codec shootout. While I understand that Christmas holidays might be the most convenient time for you to do the codec shootout, I have to say it's the most inconvenient time for development teams to assist you. Any other time of the year would probably work better. Perhaps we could reach some sort of compromise?

Before the codec shootout even started, I voiced my concern over the lack of HD content in this year's comparison. I understand that it's mostly a practical problem of aquiring high quality HD content. While Hollywood studios might not be willing to share their uncompressed HD sources with you, keep in mind that most studios and post-production facilities probably have royalty-free HD content they could share out. It might not be as cool as Matrix: Revolutions, but it'd do the job. Microsoft has HD recording equipment and I'm sure DivX, Ahead, Mainconcept and others do too, so unless you're concerned about a "content bias" there's always the option of contacting the software makers themselves early enough and asking them for HD content. In fact, if you get multiple sources, you could even put together a feature-length collage. With HDTV already widely available in the U.S. and HD-DVD and Blu-Ray on the horizon, I'm counting on next year's comparison featuring some nice 1080p content. :)

Finally, I'd like to revisit the topic of post-processing in the codec comparison. Although the FAQ explains that DECoding is an integral part of the coDEC solution, I think the focus of this shootout has historically been on the encoding part, and now more than ever. As an example, encoding performance is reviewed, while decoding performance isn't even mentioned. Furthermore, these days not all encoders have their own matching decoders but instead many depend on generic decoders such as ffdshow. That means that any difference between, for example, Ateme and x264 will not be the result of the decoding process but of the encoding process. The future might also bring us competing implementations of VC-1 encoding, but not necessarily of VC-1 decoding. While that actually slightly evens out the playing field, I think it only goes to prove that the decoder is just an unnecessary abstraction in the equation. You mention in many of your screenshots how the encoded image lacks fine detail compared to the source. In my experience, the lack of detail is more often a consequence of the decoder's filtering, deblocking and deringing than an indicator of encoder quality. I always find it surprising how much detail is actually retained in encoded clips, yet the detail completely disappeares after post-processing is applied. Something to consider.

[CruNcher]

In my experience, the lack of detail is more often a consequence of the decoder's filtering, deblocking and deringing than an indicator of encoder quality. I always find it surprising how much detail is actually retained in encoded clips, yet the detail completely disappeares after post-processing is applied. Something to consider.

Hmm zambelli everyone here knows that but did you ever deactivated the inloop deblocker in a decoder when the encoder used it @ low bitrate, not a very good idea (unless you wan't a blockfest of mini blocks) and it definately wouldn't preserve details as those are allready gone in th encoding process, for the ASP codecs that where tested and VC-1 what you say would apply but not for the AVC codecs in the test, only very good Film Grain modeling (else the scene would life) and inloop deblocking off or a higher bitrate could preserve the blurry look . Inloop Deblocking (pre- processing) and post-processing are 2 completly different things.

[Doom9]

One important fact of the comparison background seems to have gone missing here: it's a codec comparison in a typical DVD backup scenario. It's not a synthetic comparison, but a real world one. Hence containers, hence audio, hence full movies. There's a huge difference between that scenario and encoding HD trailers from apple or using MPEG test sequences. It's also a logicistal matter. Main round encoding lasted 48 hours.. in that time, I managed to encode Matrix 3 10 times, despite VC-1 being what I consider extremely slow. So take that extremely slow and multiply it by a factor of 9 for a 1080p encoding... that's 4.5 days spent encoding a single movie. So going for one HD movie instead of an SD movie, you can extend encoding from two days to about a month.. during that time, the more you use your PC for anything, the slower it gets, there's the logistical issue of moving my PC from where I sleep and the suicide risk in case I have to restart a job.
And in that month, there can be significant changes in a codec, so it could very well happen that by the time you're done encoding, the most recent codec build makes your results obsolete before you've even done the review.

48 hours might seem like a lot of time to fix 1 bugI consider it quite the opposite, it's a rescue op for catastrophic issues like missing the bitrate ;) I expect to receive software that has been tested and shouldn't fall on my head when I use it. I consider the 2 business day rule just enough to make some last minute adjustments in case you made a wrongful submission, nothing more. I never have and never will expect that issues like the ones that came up in the finals could actually be addressed in that short amount of time.
But I'd like to point out that I got two bugfixed encoders from the XviD team, a fixed decoder from DXN on very short notice, as well as a new encoder build and settings from Elecard after telling them they were qualified for the main round (you saw the time schedule between qualification and main round, there wasn't a lot of time).

As far as decoders go, with most codec makers moving to AVC, the whole postprocessing issue becomes rather moot, and in my own experience, deactivating deblocking for an ASP codec does as much to help in certain scenes as it does hurt in others. And doesn't VC-1 have an inloop filter as well?

As an example, encoding performance is reviewed, while decoding performance isn't even mentioned.I know you want to get at "VC-1 takes less computing power to decode than AVC" with this one, but my box has no problem with 1080p AVC content, and it's you guys from Microsoft that will do away with this problem within a years time by forcing almost everybody to get an up-to-date GFX card for Vista, which in terms will offer built-in hardware acceleration for video decoding. And as far as hardware goes, the point is moot anyway because we have three mandatory codecs for both HD DVD and Blu-ray. And in Europe, we'll actually see quite a but of DVB-S2 broadcasts using AVC so set-top boxes will be able to handle that as well.

[zambelli]

One important fact of the comparison background seems to have gone missing here: it's a codec comparison in a typical DVD backup scenario. It's not a synthetic comparison, but a real world one. Hence containers, hence audio, hence full movies.
Yes, but the same prinicipals are applied to transcoding TV content, too. That's just as real world of a scenario and it follows the same rules, yet often includes mixed content AND HDTV. :)

I consider the 2 business day rule just enough to make some last minute adjustments in case you made a wrongful submission, nothing more. I never have and never will expect that issues like the ones that came up in the finals could actually be addressed in that short amount of time.
Right, and I wasn't complaining really about the 2 business days rule but was just pointing out how fixing anything in 2 days is difficult when most of your staff is on vacation.

I know you want to get at "VC-1 takes less computing power to decode than AVC" with this one...
You'd think so, but that's not where I was going. :) I pointed out the lack of decoding performance numbers as an indicator that the comparison is encoder-centric, not as a suggestion that you should start testing decoding performance.

[seewen]

Thanks a lot for this codec comparison! I'm waiting for it every year.

But this time I must say that I don't agree with the results... But I understand it's a matter of personal taste:

Ateme (according to the posted samples) and especially x264 (according to samples, and personal experience), constantly smooth the pictures. All the grain is virtually gone. (On movies.. not cartoons/Animes).
And I must say that personally, I much prefer XviD results, which doesn't smooth too much the picture, and which keep more grain and details.

But I understand completely that some(most?) prefer the smoothed/cleaned result.


Thanks a lot for this comparison.

[CruNcher]

seewen alot of people don't like this look (includes me) and if you like Grain Preserved try EDP in my signature (Better Profile) :)

[skal]

Hi,
sorry to jump in late, but: i'm having an uneasy feeling when i see 100+ people company representatives complaining about the lack of manpower...
Just my morning mood, please ignore :)

[zambelli]

Hi,
sorry to jump in late, but: i'm having an uneasy feeling when i see 100+ people company representatives complaining about the lack of manpower...
Just my morning mood, please ignore :)
Good call. Next year - no Christmas holidays for anyone.
Must be my late night mood, please ignore. :sly:

[Doom9]

I much prefer XviD results, which doesn't smooth too much the picture, and which keep more grain and details.Actually, XviD gets rid of the grain, too.. I might not have made that clear enough. And the in-loop filter is configurable in AVC.. the default in most codecs is quite strong, you might like the results better if you turn it down a notch or two (as was done in most contestants).

[CruNcher]

For HD stuff @ High Bitrates you can even turn inloop completly off it will still look visualy perfect and you save decoding complexity :)

[DigitAl56K]

Hi,
sorry to jump in late, but: i'm having an uneasy feeling when i see 100+ people company representatives complaining about the lack of manpower...

While zambelli does make a good point (;)) remember also that in a 100+ people company there are likely only a handful whose level of technical expertise and specialist field is appropriate for this kind of work.

[bobololo]

My turn to throw a few words into this thread :)

First I wouldn't be honest if I don't admit some disappointments regarding the fact we couldn't keep our throne this year ;). Anyway whatever the causes that make us fall on that night sequence, was it a bug or not, it is incontestable that the margins have become extremely tight and for that impressive improvement, this year's winner desserves its place. On our side, we know now that keeping the highest rank definitively requires us to push forward continuously the progress and to maintain unsubtle advance over the competitors. And even if applying such policy to the many fields we're addressing is becoming a serious puzzle, be sure we're committed to do our best and we'll sort it out !

That said, the previous posts stir up some thoughts I found useful to express. I hope those comments won't be interpreted like complaints about the comparison but instead will bring some idea to make it better.

The topic about the settings provided for the comparison had been discussed in several posts and I think it spots out an interesting trend in modern codecs. They are becoming more and more complex, they offer many options, quality / speed compromises, and compatibility settings covering at best the users' multiple needs. We're far from the time when encoders have a single bitrate parameter ! Considering this, how can we define the "best" settings for a subjective evaluation ? As the results illustrated it, different strategies were adopted. Most of them emphasize the best efficiency-performance trade-off but having their respective woking zone quite scattered, I don't have the feeling that the codecs were competiting exactly in the same event. It's like you have to decide which of a sprinter or a 400m runner is the best athlete ;) . One approach that could help to reduce this effect would consist to specify for instance a speed target on a specific hardware and require the encoders to give their best quality according this constraint. Then it's up to the codec to stay above that minimal limit and if it goes much faster, it's its loose not exploiting all the resource available to it. That would additionnaly ensure not to spend to much time with the encoding stage !

Another point about the settings. Some of them depends much on personal tastes. At this point it's extremely hard for the developers to decide for the evaluator. Especially it could also depends on the viewing environment (type of screen, how the place is lighted, etc.). Wouldn't be possible to adjust those settings on 1 or 2 short clips accordingly to the tester's taste ?

Lastly and probably the most debatable point : wouldn't be that codec comparison even more revelant if it were led by a few people instead of a single person ? It has nothing to do with Doom9's judgement, but drawing conclusions and deciding which is the best (whatever the field) based on a single person's subjective perception is quite unusual, isn't it ? I'm pretty sure, there are over there plenty of board members who would be happy to take part to that project. That could reduce Doom9 load and bring more solid results. It's always easier to argue against a single man opinion than a result consolidated by 10 or more persons. Hydrogen Audio public listening tests are a good example even it's not the same kind of comparison. What do you think ?

disclaimer: just to make sure, that were my personal opinions and they don't involve ateme's official position.

[Doom9]

One approach that could help to reduce this effect would consist to specify for instance a speed target on a specific hardware and require the encoders to give their best quality according this constraint.But how is this feasible? AFAIK your codec is the only one that can target a specific encoding speed.. others simply cannot. The people working on the codecs without such functionality would effectively be forced to buy the exact system I use for encoding, then having to find the settings that get as close as possible to that speed goal. I don't see a lot of contestants putting up with that. And depending on where you set this threshold.. I have a very powerful machine.. when I set the treshold to real-time encoding, what about people who have 5+ year old machines (I agree, they should upgrade ;)).. they might chose different settings because they don't want to wait for days.
And there's another point: what do you do with encoders that max out? I'm sure you'd actually have to slow down your ASP encoder to encode at only real-time speed on my machine.. and that would probably apply to a few more codecs.. so in the end we have those that can go for max quality, while others have to use a severely restricted featureset to even get to that speed.

But in a way, I'm doing half that by requiring a minimum encoding speed, and I'm seriously considering a steep hike in the minimum required speed for the next round (I'm dreaming of an AMD dual core system with both cores running at 3 GHz). Yet one of the main differences of codecs besides quality is speed.. if a codec can only use one core or if it uses both cores make a huge amount of difference and would put an additional burden on those codecs that have no SMP optimization (as there's no way I'm ever going to give up on using SMP optimization).

I don't have the feeling that the codecs were competiting exactly in the same event.Well.. the source (distance, track properties) and time measuring equipment (eyes) were the same.. You should rather compare it to F1 racing though.. you can have different tires, different car chassis, use different materials, motors, etc.. all that has an effect and in the end the only thing that counts is that you cross the finish line in first place. The same applies for other mechanized sports as well, so they have a much higher resemblance than a track race.

At this point it's extremely hard for the developers to decide for the evaluator. That's part of the fun, isn't it? And it's part of me covering my own ass.. I don't need to be attacked for using certain settings after the comparison, there's enough discontent as it is and I've seen how things went in past years when I picked the settings instead.

Wouldn't be possible to adjust those settings on 1 or 2 short clips accordingly to the tester's taste ?Don't you already have such a benchmarks by looking at results from previous comparisons?

wouldn't be that codec comparison even more revelant if it were led by a few people instead of a single person ?That is very much debateable instead. Personally I obviously only trust myself to make that decision.. for me a comparison where 10 other people have a different opinion, I couldn't possibly write that up anymore. In the end, you could only show how people rated things, but it's no longer possible to make subjective comments at all as you'd end up with a bunch of contradicting statements.

That could reduce Doom9 load and bring more solid results. I don't see the load angle at all.. in fact I see it the reverse way. In addition to encoding, you have to distribute review clips, so it's cutting, uploading, traffic, bandwidth and possible legal issues to boot. Then you have to ensure the same review environment which means additional work for me explaining and for you fixing problems (remember the 2004 comparison and taking screenshots? And remember the WMV9 results that got messed up somehow? Now imagine those issues multiplied by the number of reviewers... player, filters installed, GFX cards, drivers, it all can have an influence).

Sure, I cannot cover a broad spectrum of opinions with just one reviewer, but what is all the info on page 1 for if not for letting people reproduce the results? They can get their hands on 3 out of the 4 main round contestants, and a bunch of other codecs that delivered good results but didn't quite make it (VP7, XviD AVC (that one soon)).

[Manao]

The people working on the codecs without such functionality would effectively be forced to buy the exact system I use for encoding, then having to find the settings that get as close as possible to that speed goal.Well, for open source codecs, finding a willing tester with the proper hardware isn't that hard. And in any case, interpolating the expected results is possible ( as soon as the computer is roughly the same ).

Note also that if you specify an encoding speed, we would be on par with other codecs, because we would definitely not use our 'target speed' encoding quality, since this one gives a constant speed ( not an overall speed, meaning the hard parts of the movie, which are incidentaly the slowest one too, would be encoded with the lowest quality settings )And there's another point: what do you do with encoders that max out?Fair point. Yet, maxing out an AVC codec is quite hard ( lets forget that ATI exists at the moment ;) ), and since you already made two categories ( ASP, AVC ), you could set two target speeds.the only thing that counts is that you cross the finish line in first placeIn the case of the codecs, the track isn't a one dimension line, it's a two dimensionnal plane ( quality, speed ). You still have to say where to draw the line :)I don't need to be attacked for using certain settings after the comparisonFair point.

The only solution I can think of - developpers giving a set of settings with some of them being intervals ( [-4..0] for deblocking, for example ) would quite complicate and slow down your job.I don't see the load angle at all.. in fact I see it the reverse waySo do I :)

[Doom9]

Well, for open source codecs, finding a willing tester with the proper hardware isn't that hard.It's still an n-dimensional problem.. and let's just say my experience with volunteers hasn't exactly been great (lavc...)

and since you already made two categories ( ASP, AVC ), you could set two target speeds.Those are likely to go away in the future and to be replaced with generic categories where codecs are placed into at random.. this is to ensure that in the end, I get the 4 (4 is a good number for the final actually.. and considering that I'm considering a full length HDTV full res movie for the future... a small number is so much more important) codecs that I deem to be best, not some that made it just because there was no competition in their category.

And naturally an ASP codec going 40 fps wouldn't thrill me when I have AVC encoders doing more than that at good quality (leaving out ATI yet again.. it's encoding speed rivals that of XviD).

I actually would like to increase the minimal encoding speed to real-time.. the reason I have only slowly raised the bar is it makes entry very hard.

And by the way, had the SPR results been on par, then the ateme encoder would've been the winner.. but who is to say that if you slow the ateme decoder by 10 fps, that the change is noticeable? It actually could be an interesting experiment ;) Isn't it usually so that activating the extreme options slow down an encoder incredibly while only bringing very little to the table?

[Manao]

but who is to say that if you slow the ateme decoder by 10 fps, that the change is noticeable? It actually could be an interesting experimentI'd say that x264 would still have a slight edge, because it looks slightly more sharp ( on matrix 3, the only movie I thoroughly tested ), and slightly less stable too ( but that's less annoying to me, on the overall ). To see the difference, you'd need to pause the movie and that's definitely not a proper to compare visually the codec. So I'd say in the end they are on the same level.

Anyway, you can make the following test : encode the movie with 5% less bitrate, and try to see the difference. I don't think I can see it. It amounts - roughly - to the same ( actually, 5% reduction file is bigger than the quality lost by 10 more fps - at these speeds ).

[Caroliano]

@Archimedes: (http://forum.doom9.org/showthread.php?p=758348#post758348) Another example of how metrics are not so good. For example: In "The Confrontation" scene of Steamboy the quality diference between Xvid and x264 is very big IMHO, not only 0,049 SSIM. Ringing, blocking and black over brown problems with Xvid, and none of them with x264.

You don't comented about the ringing issue on the scene "in the train". Xvid has much less ringing than Divx in this scene.

A coment: In the last round, you refered many times to Xvid performace in Divx comentaries, even though it was before Xvid in your list of screenshots. The rule is normaly to refer to an already showed thing, not an future thing, to explain a point. This was a bit inconvenient, not much of course.

Thanks for the comparison.

[virus]

Those are likely to go away in the future and to be replaced with generic categories where codecs are placed into at random.. this is to ensure that in the end, I get the 4 (4 is a good number for the final actually.. and considering that I'm considering a full length HDTV full res movie for the future... a small number is so much more important) codecs that I deem to be best, not some that made it just because there was no competition in their category.
Sounds like you're going to organize comparisons like a Fifa World Cup draw :D

(actually, that looks like a good idea, when you want to quickly select competitors among a large group, avoiding too much "matches" - that's why such kind of things exist in sport competitions. All you need to do is choose a wise way to create the groups - putting x264 and ateme together in the qualifications would probably deprive the final round of one of the best 2-3 codecs... but if I understand your words correctly we're already singing the same tune here ;))


...and now, since I'm here, I'll have to post my comment on the comparison itself :)

What I want to say is: thanks to Loren, Alex W., Christian, Alex I., Radek, Tuukka, Mathieu, Måns and all the other people who have contributed great patches to x264 during 2005. The progress this codec has made during this year is incredible, no matter how you look at it. Hats off.

[bobololo]

But how is this feasible? AFAIK your codec is the only one that can target a specific encoding speed.. others simply cannot. The people working on the codecs without such functionality would effectively be forced to buy the exact system I use for encoding, then having to find the settings that get as close as possible to that speed goal. I don't see a lot of contestants putting up with that. And depending on where you set this threshold.. I have a very powerful machine.. when I set the treshold to real-time encoding, what about people who have 5+ year old machines (I agree, they should upgrade ;)).. they might chose different settings because they don't want to wait for days.

I don't think it would be so hard to achieve especially if some tolerance is allowed. I guess the best think would consist to ask to contestants if they prefer to provide their codec setting having in mind a indicative speed target or if they prefer operating freely (and blindly regarding how other codecs would be configured). Personaly I would go for the first choice. And if you consider your hardware is running too fast, just increase the required performance ;).

And there's another point: what do you do with encoders that max out? I'm sure you'd actually have to slow down your ASP encoder to encode at only real-time speed on my machine.. and that would probably apply to a few more codecs.. so in the end we have those that can go for max quality, while others have to use a severely restricted featureset to even get to that speed.

As long as the main criterion of the comparison is the quality, I don't see why maxed-out codec would suffer from this constraint ? They simply have to give the best quality they can ignoring that speed limit since they are far above. And beside that doesn't prevent you (as you actually do) to have a sidenote telling which codec offers the best quality/speed compromise.

Well.. the source (distance, track properties) and time measuring equipment (eyes) were the same.. You should rather compare it to F1 racing though.. you can have different tires, different car chassis, use different materials, motors, etc.. all that has an effect and in the end the only thing that counts is that you cross the finish line in first place. The same applies for other mechanized sports as well, so they have a much higher resemblance than a track race.

I would agree if your comparison used objective metrics to decide the winner like with MSU or Sagittaire's comparison. In a F1 race, the winner is declared by a very actual and objective fact which is totally different from your subjective evaluation. I would rather assimilate that activity to a ice-skating or gymnastics event where a jury has to rate the contestants by giving marks.

That's part of the fun, isn't it? And it's part of me covering my own ass.. I don't need to be attacked for using certain settings after the comparison, there's enough discontent as it is and I've seen how things went in past years when I picked the settings instead.

Well, that's not so fun (to me at least :)). And you don't need to choose all the settings by yourself. The devs could provide you the basic configuration and a few customisable parameters you can adjust to your taste. Is that so incongruous ?

That is very much debateable instead. Personally I obviously only trust myself to make that decision.. for me a comparison where 10 other people have a different opinion, I couldn't possibly write that up anymore. In the end, you could only show how people rated things, but it's no longer possible to make subjective comments at all as you'd end up with a bunch of contradicting statements.

What prevent you to present different writes up giving the different points of view ? Wouldn't that approach more democratic and even more interesting for the reader to see the different opinions ? And to be honest, if the jury is made of skilled and open-minded (ie not a zealot) persons like we can find many here, I'm pretty sure that it won't have big divergences in the opinions. If you refer to many situations (whatever the domain) where a subjective judgement has to be made, the jury is usually composed of several members and it's rather rare to have a single ruler acting like a "god" that makes the decision alone. Actually I don't see any example of this case, do you have ?

I don't see the load angle at all.. in fact I see it the reverse way. In addition to encoding, you have to distribute review clips, so it's cutting, uploading, traffic, bandwidth and possible legal issues to boot. Then you have to ensure the same review environment which means additional work for me explaining and for you fixing problems (remember the 2004 comparison and taking screenshots? And remember the WMV9 results that got messed up somehow? Now imagine those issues multiplied by the number of reviewers... player, filters installed, GFX cards, drivers, it all can have an influence).

Of course you're right, scaling up the evaluation process involves logistical issues and no doubt it's far to be easy to setup such an environment. However, if you allow me to be a little bit teasing, I would say those arguments are loosers' ones ;). Dropping ideas by invoking technical difficulties without trying to find some solutions to work them around is a behaviour that could in some way be similar to bitchy people complaining about dct issues making their champion performing poorly ;) I'm kidding obviously ! Well actually my only point is that if some ideas or concepts can significantly move things forward, it may be worth to make the efforts to think about and to try to find some means so they could be carried out. I guess that's how things progress and how unbelievable things came to life. I'm totally off-topic but just consider the toughness of sending something to the space. If the person who had that idea at first and concentrated himself upon the issues he would be faced to, I guess he would give up at once to our great loss. Don't you agree ? :)

[Doom9]

I would rather assimilate that activity to a ice-skating or gymnastics event where a jury has to rate the contestants by giving marks.Actually it's a little bit of both as quality is subjective and speed is objective.

Is that so incongruous ?If you want to trade places with me if you get flanked from the left, right and back because of your settings.. I really don't need that anymore.

Actually I don't see any example of this case, do you have ?Do you have any examples of a subjective video quality comparison where that was done? I've seen very little such comparisons and generally they were done by a single person, and by looking at screenshots, not the video. My comparison can only be an indicator, it's still up to the individual user to make the final choice.. no matter how many people you put in a jury, in the end each and everbody not on the jury can still judge differently and if they differ, it's just a further proof that quality is in the eye of the beholder.

However, if you allow me to be a little bit teasing, I would say those arguments are loosers' onesNo, I consider myself a realist. We had to do this funny test at work once where they try to see how well you interact together and what kind of person you are... I was quite shocked to be told I had conservative tendencies as I consider myself very progressive and forward looking, but I've gotten to terms with the fact that while I really like new things, I also have a tendency to try and find the flaws right away to crush a new idea. I know these difficulties can be overcome in this particular case, but at a price. The price is significant additional effort and headaches. For instance you could encode the resulting video losslessly using a common codec, so there's only one playback chain, making it easier to replicate, but on the downside, that increases the size of the files you have to move around. At the end, do not forget that it's my free time we're talking about.. I'm the only one who gets to decide how it's being spent. Codec comparisons are enough of a headache as it is and I am very reluctant to make changes that increase the amount of head pain. And it's not like I'm just discarding without thinking through the process.. finding the problems requires that ;)

[kwtc]

Well, after all that reading, I guess if I had to choose an H.264 encoder for home use, I would not hesitate long and I would pick... x264 since the Ateme codec is not available to home users !

[KillNoise]

No doubt: Encoder parameter settings should allways be provided by developper: Common users desire default working parameters to get good results quickly, rather than experimenting with many passes of try and error. At most, one might be willing to select from few parameter sets depending on certain types of footage, e.g. movie, interlaced TV, cartoon, b/w and perhaps whether affected by strong grain or noise.
Generally i see responsibility of developers to reveal their experience by recommending useful parameters and giving explicit advice for adjustment - not leaving you alone with a bunch of mysterious controls ! Codec performance will be rated by results achieved in practice, rather than potential excellence after expert tweaking parameters individually for each source.


May i suggest small modification of the competion categories for your future comparisons ?
Would be good to not only compared contestants in seperate "weight class" categories, but also have them compete under different conditions based on relevant application "Use Cases".
What are these ? In my view, mainly:

Category 1: Encoding for playback on today's common stand-alone players
==> Standard MPEG-4 ASP, commonly restricted to not using less supported features like GMC or Qpel
==> Visual quality competition for common target size = 2 CDs for one movie
(1400 MB / 90 min --> overall bitrate ca. 2000 kbps for video stream excl. Audio)

Category 2: Quality/space enhanced encoding for playback on PC
==> competition based on smaller target size = single CD for one movie:
(700 MB / 90 min --> overall bitrate ca. 900 kbps for video stream excl. Audio)

I think this competition-split would make results much more practically usefull, while no longer having to perform frame quality comparisons across categories (in fact, these become two competions independent of each others !)


Next, what will be the worth of a competitions winner, if it is not available to the public ?
For example, you have allways been testing Atemes "Edge of Technology" releases, while the licensed version actually available to users through Nero Digital still remained a much older one.
==> would be usefull (challenging public release scedule !) to seperate competition score categories into:

a) Encoders already available within some handy environment (e.g. VirtualDub)

b) Open class "Edge of Technolgy" previews of codecs not yet available for public


Further i see two other relevant application categories, which you might consider for one of your future comparisons:

Category 3: Encoding for very low bitrate streaming applications
==> Visual quality competition based on common low constant bitrate setting
(e.g. 256 kbps CBR at common reduced resolution like 320x240)
This category might give a chance for codecs specialized for this area (e.g. On2 VP7 or Snow wavelet codec)

Category 4: VCR-like realtime encoding
(i know many users doing that for simple delayed watching or even for archive, not willing to setup delayed encoding after intermediate 'raw'-storage)
==> Constantly fast encoding, suited for "on the fly" live TV recording (native interlaced field encoding or perhaps field-deinterlaced to 25 fps for PAL)
==> Visual quality competition based on suitable common average bitrate setting
= some given single-pass target size (e.g. 2000 kbps ABR)

Ok: Speed criteria for realtime encoding qualification are a bit difficult to define - must be something like: "Live encoding without frame-drops on common Mid-Class Multimedia-PC reference hardware" (let's say typically those with SSE-capable single-core CPU working at 2...3 GHz). For simplicity it should be sufficient if you define some minimum reference frame rate to be met on your own PC (or your previous modell if you still have it running).
The effort will be worth a very interesting competition, giving MPEG-4 ASP another chance and having DivX, XviD and Nero/Ateme race in a somewhat different discipline !

[siddharthagandhi]

I say NeroDigital AVC H.264 Standard Profile is the best codec out there. Out of everything.