doom9's codec shootout

[Gaia]

Originally posted by KAMiKAZOW
Great work, Doom9. Thanks a lot.

http://audio.ciara.us/test/128extension/results.html shows that WMA9 is quite good (better than Vorbis according to this test).

WMA9 is very bad. WMA Pro is totally different codec.

Read this quote from that test page:

"It's worth mentioning that the WMA codec used in this test is WMA PRO introduced in Windows Media 9). It has better quality, but is not backwards compatible with WMA standard, that is the codec that is actually supported by portable players, DVD players, etc. It's expected that devices supporting WMA Pro start being manufactured soon."

[unmei]

simply great! thanks a LOT doom9 :D

Just laugh at me, but i spent more than 6 hours by now analysing every pic in the matrix test and making a rating to get some picture in how the other codecs differ from xvid. This because i recently thought about trying out ND and VP6 and also was not so sure i shouldn't give divx5 another chance..well my conclusions shall not be here ..do it on your own. The only remarks on this i give here: no need anymore, thanks doom9, you saved me hours and frustration. And RV9 surprised me with it's details in Neo's face close-up.

I know you don't see the whole truth by comparing single pictures, but it is something i always did when evaluating codecs - and a codec that cannot even survive this stage is for me not worth being evaluated any further (read as: i won't use it for actual encodes).

I did this extended examination for matrix even tho i only encode anime. I'm not complaining and think Futurama is a good test to include, since for me, the animes i encode are a mixture of 'sharp and simple' animation like Futurama and the fuzzyness/complexity of real life pictures. Thus the matrix is more important to me, but of course if xvid totally lost out on Futurama it would have made me worry :)

Btw the first scene of Futurama (the rays) is a great test for my type of anime encoding (i still remember how i terribly failed to get that encoded right from TV - like two days only to realise this scene is a killer when combined with TV noise).

[mf]

/me wonders if doom9 liked The Animatrix :D.

[Doom9]

if doom9 liked The Animatrix Yeah, but some episodes more than others. From a story standpoint the 2nd renaissance was best but I didn't particularly like the visual style. From a visual standpoint I liked the last flight of the osiris best (guess what.. it has nothing to do with Anime style ;).

All in all I didn't like the visuals much, I only watched those episodes for the story content.

[mf]

Originally posted by Doom9
Yeah, but some episodes more than others. From a story standpoint the 2nd renaissance was best but I didn't particularly like the visual style. From a visual standpoint I liked the last flight of the osiris best (guess what.. it has nothing to do with Anime style ;).
Drat :p.

[SeeMoreDigital]

Thanks Doom9, I like the test you did very much.

Very suprised at who came in second for the speed test - didn't see that one comming!

I know you don't encoding like to encode using anamorphic frame sizes but I do wish you had done some comparisons. And yep, DivX5.1.1 in slow and slowest mode is 'as slow as a hearse on its way to a funeral' personally I just stick with standard mode!

I'm still in France, so I can't get to check out the forum as much as I would like.

Cheers you guys and Happy New Year

[Doom9]

I know you don't encoding like to encode using anamorphic frame sizes but I do wish you had done some comparisons.http://www.doom9.org/codecs-203-6.htm, Q2. I'm dangerously close to use r16 and strike everyone who ever mentions something I put in the FAQ.

[bilu]

Originally posted by SeeMoreDigital
I know you don't encoding like to encode using anamorphic frame sizes but I do wish you had done some comparisons. Wouldn't be fair if codecs didn't have the same amount of pixels to process.

Bilu

[Soulhunter]

Soulhunter: There's a torrent link here (you'll need a bt-client):
Thank's !!! ;)

Bye

[sillKotscha]

Originally posted by Doom9
http://www.doom9.org/codecs-203-6.htm, Q2. I'm dangerously close to use r16 and strike everyone who ever mentions something I put in the FAQ.

:D

hope you had enuff coffee and/or beer during the shootout...

and folks remember one thing - he did it in the time of x-mas!! So congrats to you doom9 spending your time in front of a PC and not with beloved friends (despite the ones here :) )

/me wouldn't have done it and I believe no one else... so please, no complaints!!

cheers Sill

[Doom9]

actually I don't drink coffee and reviewing codecs with alcohol in my bloodstream would be unprofessional so I never drank a drop of alcohol when reviewing or writing the comparison.

[sillKotscha]

my statement was more or less rethoric :cool:

no coffee :confused: - and concerning "alcohol in my bloodstream would be unprofessional"... well this one is rethoric as well I guess ;)

[Elrik]

Curious... In "The Matrix" test, DivX5 keeps the color quite close to the original (especially in the close-up of Neo holding a cellphone), while XviD changes it a bit (darkens and/or saturates it somewhat). In "SPR", however, XviD is the one that keeps the color close to the original (especially in the first underwater scene), while DivX5 does not (sort of lightens it). What causes it? Is it how each codec encodes different sources/scenes or is it post-processing settings?

Could it be that in SPR underwater scene DivX5 lightens it somewhat because it smears it too much and, hence, destroys the darker color details, making the overall scene appear lighter? But then... it smears the details in Neo close-up too, yet it keeps the color nearly identical to the original...

[Wilbert]

A small remark. Please correct me if I'm wrong, which may very well be possible.

In the encodings, a very low bitrate is used for the Matrix (and almost twice as large for SPR ). The reason is that there is aimed for 1 CD encodings. That's a choice, and there is nothing wrong with this. The results don't say anything abouut 2 CD rips without pp or noisy sources. Of course, nobody claims that it would. (Personally I would love to see also a comparison for these two cases. In case somebody starts complaining: no, I don't have time for it.)

My point is that the post processing (and pre processing) is very important in these lowbitrate encodings. Because, one of the important things in this comparison, is the amount of detail versus blocks.

It seems to me that codecs with a very good deblocking/deringing code (like XviD) has a clear advantage above codecs with less good deblocking code. But, of course, codecs are more than good deblocking/deringing code.

Any thoughts on this?

[SeeMoreDigital]

Originally posted by Doom9
http://www.doom9.org/codecs-203-6.htm, Q2. I'm dangerously close to use r16 and strike everyone who ever mentions something I put in the FAQ. I don't think proposing to strike members because you either don't think they have read your Codec Shoot-Out FAQ (http://www.doom9.org/index.html?/codecs-203-6.htm) or you don't like (or agree) with what they have posted is very fair Doom9!

I understand your reasons for not generating anamorphic encodes. However, it's my opinion that doing such a test, may be useful and throw up completely different results!

"Push each codec to it's limit I say"!

It's also my opinion that some codecs/applications handle cropping and resizing better than others, which in turn affects the encoding speed, codec performance, encode image quality etc. Which is why I am in favour of encoding to the same image pixel frame size as the source..... But purely for the purpose of testing!

It's just my opinion. I apologise if I upset you!

Cheers

[twist3d]

Maybe in the next codec comparison it would be useful to share the workload? I think there would be many volunteers in this forum to try different settings and bitrates/resolutions.

Also "real" comparison should use abx-tests using ~100 participants, but this is completely useless as we should first encode video, then convert it to raw .avi for "blind" testing (too large filesizes).

[Doom9]

@twist3d: As you said yourself, blind testing for video faces major problems, mostly on the bandwidth side. I know of another site to try this but they never finished.

And as for different people trying different settings, that's once again unrealistic. Who reviews the results? Who decides which samples are taken? Every review clip takes hours and hours, for creation, muxing, review, screenshots and writing. Therefore, I'll rather not do a comparison at all than test the same codec with 5 different settings (and multiply that by the codecs to be tested).. it's simply not feasible. In video encoding, we simply have too many variables to compare. Thus instead of taking "some" settings I chose to ask the people who create codecs. They should be familiar with the options they offer to the user and should be able to make an educated guess as to which options might give a good result. It might not be the best possible, but I'm convinced that the results are in the "better" area of all the possible results you can get

@SMD: You, and everybody else who just can't keep their mouth shut about what they think is better should have a look at this thread and specifically colordog's posts: http://forum.doom9.org/showthread.php?s=&threadid=67634. Everyone complaining reduces the chance that I'll ever bother making another comparison. I don't need suggestions that I have already considered, or stuff that's simply useless. Last time there were a few good suggestions at slashdot like PNG images and using tables to present the results. Those things really make sense and they have been integrated. But I really don't need people telling how to do my comparisons.. that is at the least disrespectful of my work. If you disagree with the FAQ, do as colordog says.. keep it to yourself because if you bother me with it, it pisses me off and I take it as a personal insult.

It's also my opinion that some codecs/applications handle cropping and resizing better than others,Cropping and resizing is done in AviSynth for every single test so they have zero influence on the result. On the contrary, with anamorphic encoding you'd actually be evaluating the quality of the codec built-in resizing filter.

[dragongodz]

jeez this is still going ?

for those that missed my commenst after the previous codecs test i guess i can say them again.
listen carefully for i will say this only. once(name the show if you dare)

if you are unhappy with what Doom9 has done and would do it differently then do it. do the tests, put them up on a website(plenty of free ones around) and post the link here. then wait and see all the flak you get. then maybe you will appreciate how Doom9 feels. :devil:

[mf]

Originally posted by Doom9
@SMD: You, and everybody else who just can't keep their mouth shut about what they think is better should have a look at this thread and specifically colordog's posts: http://forum.doom9.org/showthread.php?s=&threadid=67634. Everyone complaining reduces the chance that I'll ever bother making another comparison. I don't need suggestions that I have already considered, or stuff that's simply useless. Last time there were a few good suggestions at slashdot like PNG images and using tables to present the results. Those things really make sense and they have been integrated. But I really don't need people telling how to do my comparisons.. that is at the least disrespectful of my work. If you disagree with the FAQ, do as colordog says.. keep it to yourself because if you bother me with it, it pisses me off and I take it as a personal insult.
I'm risking myself here defending SeeMoreDigital, but I'm going to do it anyway. Suggesting is not complaining. I think I understand how you feel as as soon as I show people a webdesign I've been working on I get overloaded by suggestions on readability, colors, etc, while I really did mean all those things to come out like they did. Though they might feel as complaints I have had to understand that they are not, they are simply suggestions on what would improve it for them (which, I must emphasize, is almost never what would improve it for everyone). The best thing to do then is just weigh all the suggestions down by general opinion and weigh in your own (heavier than the rest ;)), and then come to a decision whether you want to change anything. Nobody forces you to anything, and I think it's really best to try not to see any of these suggestions as offensive. After all, they're only suggestions, and if you're not in the mood of explaining why you didn't do things (Senior members and moderators can always back you up on things like the FAQ anyway, I wasn't as sharp as to notice that you actually had a FAQ, but other people will, I think), you're free to ignore them. Also, as a semi-free discussion board it's nothing bad to speculate on how things could be different, when the tendency is not to force anyone to change those things. My own suggestion, though covered by the FAQ (read above about how I totally didn't notice any FAQ, I usually skip to the test itself), also wasn't meant with any pressure in mind. I didn't think that it would actually happen because of me (after all, Futurama has now been in for 2 or 3 tests now), but I did think that if enough people would be of the same opinion, it would be worth considering. Well, if not, pity. No harm done. Or so it should be. Imho. Of course I could still just keep it to myself, but why? As most people here are used to having some kind of freedom in speech, it'll just cause less trouble if you take things friendly ("Better out than in I always say" -- Shrek).
Like SeeMoreDigital, I will also end with my respect to you. I don't wish to upset you, I just want peace in this thread :). I hope my thoughts came across as I intended them through the keyboard and language barrier. Posting like this is always an RGB->YUV->RGB conversion ;).

[bilu]

@mf
You almost made me cry. :goodpost:

@Doom9

Do you think there are ways to share your workload? Encoding, etc.?
I won't be able to provide hosting, but I´m sure lots of guys would be helpful in what they can. You'd do the unreleased codec encodings, of course (such as ND's VFW version). It would be nice if we could at least spare you from some CPU and trouble :) while trying to do things in a impartial way as you have done.

Just trying to help and motivate.

Bilu

[Doom9]

Suggesting is not complaining.There are different ways you can make a suggestion. There's the productive one: measure X would have the following advantages, and the following disadvantages. And you have to be honest about it. I do have well documented reasons why I do things a certain way and as you insist, I come up with even better ones (I think I have to excent the FAQ now). So in the end what you do is counterproductive.. I've come up with even more reasons against anamorphic encoding in this very thread. I have also come up with more ammo for Q1, Q11 and Q12 based on discussions in this very forum.

Take the tables for the numbers: without tables it is hard to read (everybody agrees with that), adding tables requires only a small amount of work while improving readability considerably. Now that is a positive suggestion (and it does appreciate that changing something actually involves work).

However, had I previously written that I cannot be bothered to add a table then it would be useless to even mention it. It's like people telling len0x to add WMA audio or WMV video to AutoGK. He has previously stated that he won't do that so asking is just a waste of time. And most people don't appreciate if you waste their time.

I know what you're trying to do.. nag away until I change my mind. But that can have two endings, one being that the sheer mention of change makes me shut my ears. For those who care: my MBT type is INJT, so that means I think a lot before I write and thus the answers to my FAQ questions are well thought out and chances of my reversing my opinion are slim. Heck, I even offered the possibility of increasing resolution and I was thinking about anamorphic encoding for a 1 CD source but this constant pestering is really getting on my nerves so that I'm seriously considering to drop that idea just to get back at all the people who've gone on my nerves.

@mf:I was accused of writing unreadable posts as well so I try hard to improve... a few more linebreaks would also do your post some good.

Oh, and not reading an FAQ is a rule 1 violation :devil:

[SeeMoreDigital]

Originally posted by Doom9
@SMD: You, and everybody else who just can't keep their mouth shut about what they think is better should have a look at this thread and specifically colordog's posts: http://forum.doom9.org/showthread.php?s=&threadid=67634. Everyone complaining reduces the chance that I'll ever bother making another comparison. I don't need suggestions that I have already considered, or stuff that's simply useless. My dear Doom9, your tone appears to be getting a little hostile!

Nobody is belittling all the time and effort you continually put into 'your' forums and websites. But you cannot expect everybody to share the same views as your good self, and/or go about testing in the same way as you!

I thought one of the aspects of having a democratic forum is to openly discuss matters such as this and to be able to put all opinions forward.

Standard anamorphic (720x480/576) encoding is just the first step in seeing how well todays codecs perform when pushed to their limit!

Given that high-def images are gaining in popularity. I think it's fair to say that there are many of us here, on your forum, who are interested in generating encodes using larger image pixel frame sizes than even standard anamorphic. And would like to know how well the new versions of all the codecs perform!

And already it's clear to me that some codecs perform much better than others in this area. Take WMV9 for instance. It's an established fact that this codec can cope much better with larger image pixel frame sizes than say DivX or XviD, in both encoding and playback. So can RV9 and so can VP6.

In a few weeks time I hope to be upgrading from my WVGA (852x480)42" plasma to an WXGA (1280 or 1366x768) plasma screen.

As a PAL user I currently have to compromise my DVD viewing by squeezing down from 576 vertical pixels to 480 vertical pixels. And anybody who has had the opportunity of viewing say the high-def WMV9 version of T2 Extreme on an WXGA (or even WUXGA) screen, can't help being astonished at the quality of the encoded images. Images that just are no possible to view using less pixels!

In conclusion Doom9. Many of us regard you as an independent judge/voice when it comes to video encoding. Many of us look at your test results and obviously use them as a basis to generate and compare against our own encodes. After 25 years in this business I know how easy it is to get upset when somebody says or suggests something you don't like. I can only speak for myself when I say, I have far too much respect for the work that you do to upset you on purpose!


Kind regards

SeeMoreDigital

[dragongodz]

Doom9 may i humbley suggest you not bother to read this thread anymore ? because the stress levels emeninating from your posts have get to the point where you appear to be ready to burst a blood vesel(at least).
now take 10 slow deep breaths and try to relax.

"the aspects of having a democratic forum"
SMD - democratic forums are blood baths of bad language and flame wars. i dont think anyone really wants that here do they ? open discussion is good but going on once a decision has been made is pointless unless you can come up with a very good NEW reason why the decision should be changed.

[SeeMoreDigital]

Originally posted by dragongodz
....open discussion is good but going on once a decision has been made is pointless unless you can come up with a very good NEW reason why the decision should be changed. I thought I had forwarded a NEW reason for requesting a 'codec shoot-out' using larger image pixel frame sizes (whether they be standard anamorphic frame sizes or even larger). But just to underline my reason, it is because of high-def sources.

I really don't fancy the prospect of encoding my high-def 1920x1080 or 1280x720 'full 16:9 frame' images to say 640x368. I also don't fancy the prospect of storing hundreds of GB's of raw Mpeg2 streams on dozens of hard drives. So I would greatly appreciate some quality, hopefully impartial, information about alternative methods/codecs.

And I don't think I am alone in this line of thinking. But as usual I'm willing to put my head on the block by suggesting it!

Cheers

[bilu]

It's time for specialized benchmarks: high-def, anime, interlaced, live-video, and others. There is demand for them.

But Doom9 itself has no obligation in this respect. Others should do them, post the results, and MAYBE Doom9 would be kind enough to host them, if he has spare resources. But specialized stuff should be left in the hands of others - Doom9 has done more than enough already. And has expressed that more than once.


Bilu

[Kedirekin]

Very well said.

[Doom9]

Given that high-def images are gaining in popularity. I think it's fair to say that there are many of us here, on your forum, who are interested in generating encodes using larger image pixel frame sizes than even standard anamorphic. And would like to know how well the new versions of all the codecs perform!

And already it's clear to me that some codecs perform much better than others in this area. Take WMV9 for instance. It's an established fact that this codec can cope much better with larger image pixel frame sizes than say DivX or XviD, in both encoding and playback. So can RV9 and so can VP6.

Woah, now you have actually ticked my interest. Needless to say, the test are DVD encoding tests and for that they make a lot of sense.

Now unfortunately we don't get any HDTV in good 'ole Europe so all I get is lower than DVD bitrate DVB usually having a lower than DVD resolution (I'm not really sure about that statement right now.. I haven't installed my DVB card yet, I only know about the bitrates). So I guess I'd need a legit HDTV stream source. However, I do have a handful of XviD files that were taken from HDTV sources (don't worry... I have bought the DVD since so I have at least one license for that content), at resolutions that go beyond that of a DVD and I didn't see any problem with XviD.

I do have my honest doubts about your established facts though, but just for that reason I'd like to test (but not in a form of a codec comparison.. that's way too much work) and disprove such statements.

Just guess why there are no comparable tests available anywhere? Who is willing to sacrify one month of free time for something you don't even get paid for? Did you notice that I never commented on other comparisons in this very forum? There are two reasons for that: a) I prefer to see for myself, and b) I know how frustrating it is if everybody seems to know better than the guy who tested (and you bet your ass I'd find a couple of points I'd do differently). So I'd rather bite my tongue than to make somebody else feel the way certain people make me feel.

[SeeMoreDigital]

Yes, agreed.

But I think in order for the tests to be fair, each encode will have to be assessed and the results/findings announced from one impartial voice.

I feel that Doom9 (if he can spare the time) would be the ideal voice for such an undertaking.

Now we all know that generating high-def full movie encodes are not going to fit onto an single 700MB CD-R. So maybe a 4.7GB (4470MB approx) DVD~R would offer more realistic encoding bitrates and give us all something to aim for. Especially as both the DVD burners and DVD blank media are getting cheaper with each passing month!

Anybody else have any thoughts about this that we could move onto another thread?

Cheers

EDIT: Doom9 you generated your response while I was writing the above. I'm sorry if we got off on the wrong foot. I guess I just assumed that you knew what I was eluding to!

I hope that both your good self and other members now understand my enthusiasm for the tests now and why I (in the main) encode using such large image pixel frame sizes. It's not because I think they look better (as they obviously do not) it's because that's where my current interest lies and where I think we will be heading in the near future!

As you may also be aware 'large' frame sizes are unfortunately limited to just 720x576 pixels (in hardware) because the current crop of decoding chipsets are limited to output only 414,720 pixels.... bummer!

Which is why I'm interested in both the proposed new Chinese VP6/DVD recorder/players and BluRay.... I wonder what chip sets they will be using and who is manufacturing them. As each technology will need to both encode and decode using more than just 414,720 pixels!

Cheers for now

[Doom9]

Now we all know that generating high-def full movie encodes are not going to fit onto an single 700MB CD-R. So maybe a 4.7GB (4470MB approx) DVD~R would offer more realistic encoding bitrates and give us all something to aim for. Especially as both the DVD burners and DVD blank media are getting cheaper with each passing month!
Obviously the bitrate would have to be higher, or else we'd be back to the dreaded 300kbit/s RV9 tests from a bits/pixel standpoint. And while using a DVD makes a certain amount of sense I think it might be a bit too high. Just as a guess I think 2-2.5mbit might be enough for a HD source, and that normally doesn't fill an entire DVD-R if I remember my CCE -> DVD-R bitrates correctly.

[SeeMoreDigital]

Originally posted by Doom9
... Just as a guess I think 2-2.5mbit might be enough for a HD source, and that normally doesn't fill an entire DVD-R if I remember my CCE -> DVD-R bitrates correctly. Yes I think we could get away with the bitrate you suggest. But then there will always be people who will want to squeeze two (or more) 120min HD encodes onto a single DVD~R. In much the same way people now like to put a 120min movie onto a single 700MB CD~R.

For me, the logic behind putting a 120min movie onto a single 700MB CD~R is because I can't be arsed, getting off my fat arse, to change over to another disc. I suspect others might feel the same!

Anyway the HD tests I've generated used the 1440x816 image pixel frame size of the T2 high-def trailer as a source (availble from the M$ website). All of which I have converted to DivX, XviD, RV9 and VP6.

My tests have so far revealed that 1440x816 (which is all image/no matte) is a little too much for both DivX and XviD (1280x720 is about their limit) but there is no such problem with a 2pass RV9 and 1pass VP6 encode!

Cheers

[Soulhunter]

Originally posted by Doom9
Just guess why there are no comparable tests available anywhere? Who is willing to sacrify one month of free time for something you don't even get paid for?
I would do it...

If I had the bandwidth for it... Uploads with a 56k connection sucks !!! :(

Also, I would need some HD sources... I live also in good 'ole Europe !!! ;)

Bye

[SeeMoreDigital]

For all those who may be interested, I've started a new thread about the possibilities of creating HD lossy encodes (using RV9, WMV9, Mpeg4, VP6 etc) from an HDTV source here: -

http://forum.doom9.org/showthread.php?s=&threadid=68292

Cheers

[mf]

Originally posted by Soulhunter
If I had the bandwidth for it... Uploads with a 56k connection sucks !!! :(
I have only double your upload. And I also considered doing an anime shootout (same style same rules as doom9's), but came to the conclusion I'm not really interested in how other codecs perform since the only thing that's a challenge to xvid (high detailed backgrounds from PROPER raws instead of the smoothed stuff that fills 90% of the raw population) is something I'm certain of (especially after reading doom9's test) that other codecs won't even be able to reproduce authentically. The only interesting thing left then is low bitrates but as a quality freak I wouldn't even want to test those (besides, Deen() should be as good as those crazy in-loop filters for anime). So I decided against it. It would still be interesting to read such a test though :D.