Latest codec shootout....

[SpaceV]

First of all, awesome work and great display of the different
images for each codec. By loading the new pic over the old one you could really spot the differences very good. Thanks Doom9.

But now a question to the codec dev teams. I know Xvid and ON2
are regulars on this board but the other probabaly as well.

I wonder if you all feel correctly represented with this test or
if you would say "we encoded the same movie/scenes on our side and the they looked much better".
I know ON2 ( VP6 ) claims to have the "best" codec
which was "proven" wrong in this test. I also would like to know
if the correct settings were used, I read that Doom9 gave you time to submit your settings.

[Doom9]

@SpaceV: I can forward you the PMs from On2Tech with the settings which you can compare to what is written in my article.. or do I have to extract the registry settings for you so that you believe me that I've used what I was told to? I don't have any interest in screwing somebody who takes the time to actually test and submit settings to me.. and they have no interesting in submitting bad settings.

Of course, visual quality is in the eye of the beholder but the output is definitely the same codec makers got if they actually tested the settings they submitted.

[SpaceV]

@Doom9

Thanks for the response, I didn't assume that you were trying to screw anyone over. I am sorry if it came across that way.

I was thinking that maybe someone did not submit their settings or that they were from a different team member and they weren't given too much thought and now after the results are in, they would they that if you changed this or tweaked that setting with our codec, you would have gotten better results.

I am just curious if the dev teams would say "yes you got the best out of our codec" and that as good as it can look. Also if they
agree with the scenes choosen etc.

I just wanted to start a little discussion about the shootout and get some ideas / feedback from the codec people.

[bond]

all codec developers had enough time to submit their settings, they also knew what movies will be used in the comparsion...

i trust in the codec developers that they are able and have an interest to submit the settings which would output the optimum/the best possible output with their codecs

no developer can come afterwards and claim that other settings would have been better, such a behaviour would put a bad light on these developers regarding their skills and reliability

[Bulletproof]

I think if the post processing was disabled that VP6 would've been more of a competitor to XviD, I always thought VP6 was pretty bad until I was able to see the codec perform with post processing off.

[On2Tech]

Originally posted by SpaceV
First of all, awesome work and great display of the different
images for each codec. By loading the new pic over the old one you could really spot the differences very good. Thanks Doom9.

But now a question to the codec dev teams. I know Xvid and ON2
are regulars on this board but the other probabaly as well.

I wonder if you all feel correctly represented with this test or
if you would say "we encoded the same movie/scenes on our side and the they looked much better".
I know ON2 ( VP6 ) claims to have the "best" codec
which was "proven" wrong in this test. I also would like to know
if the correct settings were used, I read that Doom9 gave you time to submit your settings.

I submitted recommended settings to Doom9 and they represent the best settings I knew at that time. If there are better settings or tweaks that could have been done I have yet to discover them. However, he did use a codec that was slightly worse than theone we released though I don't know how much better the new version would have faired.

That being said I think its fair to point out that Doom9's tests represent his opinion. His test was on 3 movie / movie clips each at only 1 datarate. He then selected a few frames from each which exhibit why he made his choice. It's hardly a definitive test that proves VP6 is worse than another codec (or vice versa).

In the future we hope to change his opinion.

[temporance]

Thanks doom9 for all your hard work and a very nice looking comparison.

My $0.02: For the MPEG-4 codecs, it would be worth using a standard decoder so that we can decide which is the best encoding solution. From my point of view one of xvids strengths is its postprocessing (or maybe DivX's weakness is in this area, thx virux), and this is a weakness of DivX and the other MPEG-4 solutions.

Also now codecs are getting a variety of quality / speed settings and presumably the codec devs provided doom9 with the slowest, highest quality settings. So it's unfair to compare encoding times.

[virux]

From my point of view xvid's strength is in it's postprocessing,
AFAIK xvid's post processing doesnt even work properly yet (dshow filters)

[bilu]

Originally posted by temporance
My $0.02: For the MPEG-4 codecs, it would be worth using a standard decoder so that we can decide which is the best encoding solution. From my point of view xvid's strength is in it's postprocessing, and this is a weakness of DivX and the other MPEG-4 solutions. That would be correct from a standalone MPEG-4 player point of view, not a codec suite point of view.Playback:

I've used the default playback filter for 3ivX, NeroDigital and VP6, using automatic postprocessing strength selection. Where applicable I disabled the film grain effect.

For SBC and DivX5 I used the DivX5 filter at strength 4. I have been using the DivX5 filter to play back DivX3 movies a lot in the past and whenever I thought that something was odd I switched back to the DivX3 filter only to find out that the problem was the source, not the filter. I have also done the same during the comparison but I'm confident to say that the results would not be any different had I used the DivX3 filter for SBC playback (except for being more of a hassle to compare and make screenshots).

To decode ffvfw I've used ffdshow build dated October 28th.

To play XviD clips I used the XviD DS filter with both deblocking and deringing enabled (and film effect disabled). As you see all codecs used post-processing when possible.

Bilu

[temporance]

Originally posted by bilu
That would be correct from a standalone MPEG-4 player point of view, not a codec suite point of view. As you see all codecs used post-processing when possible.Yes, but the sort of person that reads the codec comparison is the sort of person that really wants to get the best from his codecs. This sort of person doesn't mind hacking around with a couple of FourCC's.

Maybe using the DivX encoder and the xvid decoder DSF gives me much better performance than using the same codec for both functions?

[bilu]

Teams have developed both encoders and decoders, and codecs were benchmarked using the whole suite as criteria.

The developers that had their chance to do a better encoder also had their chance to do a better decoder.

Your point of view is valid when comparing codecs to be played in standalones, using one decoder only. But if you want to compare the work developed by each team, then Doom9's approach is the correct one.

Bilu

[temporance]

Originally posted by bilu
Your point of view is valid when comparing codecs to be played in standalones, using one decoder only. But if you want to compare the work developed by each team, then Doom9's approach is the correct one.I'm not criticising Doom9's approach. I'm just trying to draw some more conclusions from his work (and provoke further discussion - maybe I'm a troll!?!?).

Having spent some more time studying the results it seems to me postprocesing is a weakness of the commercial codecs (especially causing softness and artifacts in DivX5). In fact, all the problems with DivX5 in the Futurama screenshots seem to be postprocessing problems and not an encoder weakness. So MPEG-4 hats off to the guys responsible for xvid's postprocessing! Nice work :)

doom9 wrote in his conclusion:
And DivX5 had visible problems with animated content (see the 2nd shot of Futurama, I think this could actually be a bug in the codec).Yes, it is, see the top sticky thread in the DivX forum: http://forum.doom9.org/showthread.php?s=&threadid=66226

Anyway, Happy New Year to all!

[Doom9]

@On2Tech: Of course, I've mentioned that your codec is still in development so you're welcome to participate in the next codec comparison. Though there's one noteworthy tidbit: I've used the screenshots where possible to point out things I mentioned during the review, but my comments are solely based on reviewing the encoded clips.... screenshots have nothing to do with my comments. You'd get the exact same comments without screenshots as I don't even bother looking at screenshots when writing up the entire thing.

And of course, 3 sources cannot possibly represent the entire spectrum.. but no comparison will ever be able to do that. I even think that VP6 didn't do that bad... just look at the last comparison featuring VP4 for a bad result. A codec comparision is only meant to give you an idea and there's plenty of lines encouraging you to make your own tests (which is the only way you can make sure you use the codec that looks best in your eyes). You can rely entirely on my results... which imho is fine because I tried my best, but you risk that your eyes will not agree with mine.

There is no way to objectively compare codecs... quality metrics are pretty much useless (see my FAQ) so every test will be subjective.. that's just the nature of it.

[Arcon]

argh, i thought the comparison page was broken or not completely finished since i didnt see any screens. after seeing this thread i read the source of the page and saw what you did.

so if javascript is mandatory, please tell it somewhere on that page so that people like me who don't use java/js/flash by default can enable it. thanks :)

[Doom9]

About the DS filters: actually I had to reconfigure quite a number of filters to get every filter only to play the proper content type. But in the process of having the wrong filter decode the content I found that often not using the original filter lead to worse results so unless you hold me at gunpoint there's no way I'm gonna change my testing method. And I'm quite happy with the DivX filter when it comes to postprocessing.. imho it does a good job and I wouldn't have used this particular filters for year to play my DivX3 content if I didn't think it was good.

Yes, but the sort of person that reads the codec comparison is the sort of person that really wants to get the best from his codecs. This sort of person doesn't mind hacking around with a couple of FourCC'sActually, there's a lot of people coming to read these comparisons that usually don't visit my page, thus average users not real experts. I'd expect real experts to stop at the codec settings and do their own testing.

[lat3ralis]

Great work as usual doom9 :D

On behalf of the community (users and developers), I'd like to thank you for your efforts. It's good to see open source prevailing once again in the area of multimedia compression. Congratulations to the XviD developers for the success of your codec in the comparison.

I have one question for doom9 though ...

I read that you used a special vfw build of ND. Presumably you obtained this from the ND developers. I was just wondering if they mentioned anything about officially releasing a vfw version of ND in the future.

Regards,
lat3ralis

[Doom9]

I read that you used a special vfw build of ND. Presumably you obtained this from the ND developers. I was just wondering if they mentioned anything about officially releasing a vfw version of ND in the future.
That is somehow in the 6 pages ;) To the best of my knowledge Ahead has no intention of ever releasing ND outside of their tool (Recode, NeroVision Express), except for on-chip implementations for standalones. Of course, if enough people bug them about it they might change their minds, who knows.

[Sagittaire]

@ Doom9 or all

compressibility (/xvid quant2) for Matrix and Saving Private Ryan?

[Doom9]

let's see... don't know don't care? We all know all these numbers are not perfectly reliable.. whereas I have never had reason to distrust my eyes.

[Sagittaire]

Matrix with 581 Kbps and SPR with 1020 Kbps in 640*272 with neutral bicubic: compressibility (XviD quant2) higher than 50% I think ...

all the codecs give very good results with this compressibility ... and DivX3 also for example ...

Matrix Relaoded or Attacks of the clones on 1cdr would have been a much better test I think ...

For the next time perhaps ...

[Koepi]

Sagittaire:

you don't even read the notes of doom9 (here and in the comparison) - that's the conclusion i come to if I read your post.

You do your tests and come to your results, Doom9 does it "professional" and uses settings that most users are applying for their encodes and which are most common.

It's coming to my mind that Doom9 is right, there actually are very little people doing anime or ultra low bitrate stuff, but those are shouting out loudest for support.

Regards
Koepi

[bilu]

Originally posted by Koepi
It's coming to my mind that Doom9 is right, there actually are very little people doing anime or ultra low bitrate stuff, but those are shouting out loudest for support. Doom9 is right, and as he said before ***everyone is invited to do their own benchmarking***

Doom9's are representative ones, but of course there are niches who would like to see how codecs behave under extreme conditions. These niches should do their own benchmarks based on their notion of extreme.

I'm not into the low bitrate stuff, but I believe anime has some of the most challenging stuff to motion estimation :) Nevertheless, I'm not an anime fan (with few exceptions) but would like to see some benchmarking done by the hands of anime experts. Although it brings nothing useful for me. :)


Bilu

[temporance]

I have a question for doom9 (or anyone who's replicated his Matrix / SPR / Futurama encodes):

What is the bitrate produced by each codec over the section(s) of video that were compared? I would guess each is pretty similar, but it would be interesting to know if the distribution of bits throughout the movie was significantly different between codecs.

E.g. doom9 studied frames 140155 - 153948 of the Matrix; what is the bitrate of this section:
3ivX: ? kbit/s
DivX3: ? kbit/s
DivX5: ? kbit/s
ffvfw: ? kbit/s
NeroDigital: ? kbit/s
RV9: ? kbit/s
VP6: ? kbit/s
XviD: ? kbit/s

TIA (and apologies if I missed this information somewhere else) :)

[Doom9]

I used to cut out the testclips because it made making screenshots easier, but with MPC's go to functionality that is no longer necessary so I don't know the sizes of the respective clips. I remember that they used to be rather similar though.

[temporance]

Originally posted by Doom9
I remember that they used to be rather similar though. Well, my attempt to replicate some of your tests have just finished encoding. I did xvid and DivX versions of the Matrix.

The section used for comparison was encoded at
762 kbit/s by xvid 1.0 b3 and
726 kbit/s by DivX 5.1.1
So xvid allocated 5% more bits to the sequence under scrutiny. Enough to make a visible difference? Maybe not, unless the two codecs were very close already.

Interestingly DivX encoded the credits at 574 kbit/s whereas xvid (with Qpel and different RC) encoded them at 361 kbit/s!!