doom9's codec shootout

[twist3d]

from the codec test:

I contacted each codec maker 2 weeks prior to the start of encoding to give them a chance to suggest the best settings for the movies used in this test

So,

what do you think of the results and settings used on codecs?

ie. would've the results changed if there were lanczos resize instead of bicubic resize & divx 5.1.1 slow setting vs. multipass normal with qpel etc.etc.

comments are welcome.

[spectra]

yeah it would be interresting if you could comment on the makers settings. altogether that is again a great shootout and the png with zooming feature was worth the hard work for getting the screenshots.

[Atamido]

It would be nice to get a BitTorrent up of the pages with all of the pictures as the comparison has already slowed to a crawl. And with a hundred or so people seeding, you could get it distributed pretty fast.

[Atamido]

One thing, it doesn't look like your PNGs have been optimized to get the smallest size. If you run something like PNGCrush on them, you will probably save 2-5% in size, without any quality loss.

Or, if you want to dedicate a few hours to it, then try using this script (http://www.hydrogenaudio.org/index.php?showtopic=16167) from HA and you will likely get 5-7% reduction in file size. It may not seem like much, but that is a lot of cumulative bandwidth.

[bond]

thanks again to doom9 for again a great test, which would make it once again clear howto judge the quality current codecs provide :)

Originally posted by twist3d
what do you think of the results and settings used on codecs?doom9 used the settings he was told from the codec manufacturers to use (the avs script is the same for all)!
believe me they have no interest in telling him settings that wouldnt result in the optimum...

as i am a qpel nerd, i would also have used it, but you cant blame doom9 for what the codec manufacturers suggested...

[jeremymacmull]

not to put a damper on this but im sure loads of people would have been more interested to see if the standard mode is as good as divx 5.05 slowest or better as that is the most likely mode to be used by most people

[bilu]

Originally posted by jeremymacmull
not to put a damper on this but im sure loads of people would have been more interested to see if the standard mode is as good as divx 5.05 slowest or better as that is the most likely mode to be used by most people That's an "Idiot Guide To Codecs" approach, and would require doing every benchmark with default settings. Remember that a lot of codecs have recommended settings that are completely different from outdated default settings.

Bilu

[jeremymacmull]

Not really im thinking more of time taken to encode a clip

the difference in speed between the slow/slowest mode and the standard mode is ridiculous (not sure of the spelling!! :P)

so a lot of people who were and still are quite happy with divx 5.05 will now encode in 5.1.1 in standard mode if they use it.

i myself will be switching to xvid as the speed differnce is not much and the quality diff is a lot

Basically this is for people to compare or try and compare the difference between the old 5.05 slowest mode and the 5.1.1 standard mode which are meant to be equivalent. so people can see if they want to upgrade

JEREMY

[Doom9]

Basically this is for people to compare or try and compare the difference between the old 5.05 slowest mode and the 5.1.1 standard mode which are meant to be equivalent. so people can see if they want to upgrade
What question do I ask on page 5? Is it "should you upgrade from DivX 5.0.5 to DivX 5.1.1?" or is it "which codec looks best?"?

Feel free to make another comparison that answers other questions but if you start trying out different codec builds and different settings for the same codec, I won't expect your results until the next decade. You really have to have done this once to appreciate the effort.. the amount of time required is huge.

[bilu]

Originally posted by jeremymacmull
Not really im thinking more of time taken to encode a clip

the difference in speed between the slow/slowest mode and the standard mode is ridiculous (not sure of the spelling!! :P) People looking for speed should go for Nero Digital.

[jeremymacmull]

I dont want to cause offence the new codec shootout was most enlightening hence the reason im switching to xvid as my puny machine does not like 25 hours of encoding in a row it much prefers 10-12 hours of encoding :P

I think your work doom9 is top notch and i dont know what id do with all my spare time without this site.

ive conducted a few tests with 5.1.1 standard mode on a couple of films but they werent conclusive.

thanks for all the work and if you dont mind me asking how much time did the codec shootout take from start to finish ???

thanks again

JEREMY

[Doom9]

thanks for all the work and if you dont mind me asking how much time did the codec shootout take from start to finish ???
I don't mind at all. I started contacting codec makers in late November. They were given two weeks (a few days past the deadline was still acceptable) to provide suggested settings, and updated builds if applicable. I didn't contact everybody at the same time, so the whole process took about 3 weeks until I started encoding. Encoding started on December 13th and was going on pretty much 24/7 until Sunday the 21st (I had to redo the vvfwf stuff as I've mentioned in the article). Some time I also had to delay the start of encoding to ask back (the usual inclarities when it comes to settings).

The week of Christmas I was speed-testing, muxing and taking screenshots. Taking screenshots actually takes more time than writing the article because frame numbers don't match. Frame 5000 in DivX5 isn't equal to frame 5000 in 3ivX (as an example), so for 2/3s of the codecs I had to search for frame offsets to take screenshots (and then those offsets are not always constant, making it really hard to match frames if there's no visually distinguishing mark on screen.. you have to flick between images just like you can in the comparison now). I also quite a few problems taking the screenshots, with overlay and such. Eventually, the VMR9 renderers (thanks DarkAvenger) worked out fine, but they added a 2 pixel border around everything which had to be removed again. And obviously, you screw up a couple of shots which you have to redo in that last minute (yesterday).

This week I then wrote the article (with proof-reading that took perhaps 6 hours), and spent a lot of time with the image switcher / zoomer (talking online to the programmer, bugfixing, adjusting, you name it), and then I had to turn the static image comparison (looking like the previous one) into a dynamic one.. that also took a few hours.

And now I'm having a png optimizer running in the background and I'll create a jpg based comparison as well for people with low bandwidth (all matrix shots weigh more than 10mb.. I can't imagine how long it takes for a 56k user to view them all).

[dragongodz]

Doom9 - an interesting read as usual.

only 1 complaint, damn switching pictures on 56k is painful. ye i know you already mentioned we 56k users, all i can say is you really dont want to know how long. :(

[mf]

I find the animation test still not testing the codecs very well. The DVD source itself shows artifacts, Futurama is a show without much detail (and also with thick lines that don't show the individual sharpness of the codecs), and what I heard from superdump is that the codecs don't even have all that much difficulty with it either. And last but not least the DVD is a fine example of crappy authoring, as the source was blurry which was covered up with contrast sharpening (unsharp mask), and of course in interlaced tradition sharpened stronger vertically than horizontally. Not that it actually helped.
As much as you might not like it, asian animation is a bigger test to the codecs, is being encoded by more people (so the results have more significance in the community), and is often authored much better.

As for "I personally haven't seen an animated feature that I liked yet", what have you seen already, then? Can we recommend anything?

[snowcrash]

I'm disappointed that WMV9 wasn't included in the shootout. It is becoming a mainstream codec with Microsoft behind it, despite the fact that the videophiles around here like to ignore it.

[jeremymacmull]

if you read doom9s rec he says the reason wmv9 was not included is because no new features or builds etc have been made since the last codec shootout so anyone who wants to compare should look at the last codec shootout images after all its the same movies and frames

same reason why 3.11 is not going to be included after this i believe

JEREMY

[bilu]

Originally posted by mf
As much as you might not like it, asian animation is a bigger test to the codecs, is being encoded by more people (so the results have more significance in the community), and is often authored much better. It probably would make RV9 win by a larger margin, I've heard that XVID cartoon mode is not good with real anime.

http://forum.doom9.org/showthread.php?s=&threadid=65938


Bilu

[Doom9]

to the anime crowd: I forgot to add this to the FAQ (will be added when I upload the modem friendlier jpg version):

Why don't you use an anime source instead of Futurama?

Despite what the anime crow (which is really loud) wants you to believe, more people watch regular animated features than Anime. Look at all the cartoons, and mainstream Hollywood movies (Disney anyone), that attract a much wider audience than Anime (which is a special interest genre).

Furthermore, I just don't get Anime and I won't use a source that I don't like.And those FAQ items are my last word on the subjects. I know all the complaints that come every time and I've dealt with them accordingly in the FAQ because I don't want to waste my time with this here.

[bilu]

IMHO it would be better to invite anime encoders to do their own benchmarkings. I honestly don't like the tone of your FAQ proposal :sly: - Japan is a wide anime market and not a less important one. Of course US, Europe and others do more cartoon then anime, but I've seen lots of anime posting by French users, for example.

Recent article on Japan's cultural growth:
http://slashdot.org/article.pl?sid=03/12/28/1623216&mode=thread&tid=168&tid=186

Bilu

Note: I don't do anime.

[mf]

Spirited Away was released to the west as a Disney release, and that said, even the original Disney releases are more of a challenge to codecs than Futurama.

[Doom9]

I suggest you turn on the TV and flick through the channels, then revise your statement. We have millions of kids every day watching good old fashioned cartoons on TV. How many popular movies that were shown in a theater are Anime and how many are regular animated features? I go to the movies quite often but the last Anime flick I remember was Ghost in the Shell and I could give you a long list of normal animated features that has been shown in the past few years.

Come on, don't try to decieve yourself.

[bond]

Originally posted by snowcrash
I'm disappointed that WMV9 wasn't included in the shootout. It is becoming a mainstream codec with Microsoft behind it, despite the fact that the videophiles around here like to ignore it.it wasnt ignored, it simply didnt change since the last comparison, no need to repeat (and waste bandwidth) for showing that wmv9 provides less details than xvid again :D

btw: i am sniffing a lot around on video related sites, but i dont see where wmv9 is becoming a "mainstream codec" (i guess the reason for this is cause m$ is behind it ;) )
some people prefer to use wma9 because m$ pushes standalone support, but many independant (!) and scientific listening tests showed that wma9 cant cope with vorbis or aac (at as good as all bitrates)

sorry :p

[bilu]

@Doom9

I agree with you about most of the world except Japan.
But if we think of anime as a harder cartoon, then it would make more sense for cartoon benchmarking, much like SPR or Matrix scenes were chosen.

Now if cartoon has hard parts that anime doesn't have, it becomes a completely different question ...

Bilu

[mf]

Originally posted by Doom9
I suggest you turn on the TV and flick through the channels, then revise your statement.
Okay.
I flick on Yorin, I see Toonami. Transformers Armada. Anime.
I flick on Fox Kids, I see Crayon Shin Chan. Anime.
I flick on Nickelodeon, I see Bassie en Adriaan. That's not cartoon so it doesn't count.

[Kedirekin]

I'm a huge anime fan, but (even though I hate to admit it) Doom9 is right.

Despite the surge of popularity of anime in the past 5 years (the selection at my local Best Buy has gone from half a shelf to half a wall), it just doesn't have the same kind of penetration as cartoons.

We've seen the recent theatrical releases of Rugrats, Thornberrys, Kim Possible, Hey Arnold, Recess, Lilo & Stitch, and how many others. The *only* recent anime release I can recall is Spirited Away, and that was a limited release at that.

[Teegedeck]

Just wanted to say thanks; very meticulous work, must have cost a lot of time. Great layout, too!

[sysKin]

Just to add my two cents to the anime part of the thread:
Originally posted by Doom9
Furthermore, I just don't get Anime and I won't use a source that I don't like.Sorry but that settles it. I love anime, test every xvid expermient with it and enjoy doing so. If Doom9 says he doesn't enjoy it, I won't mention anime ever again.
This stuff is made for fun, period.

Radek

PS. at least Futurama is the kind of cartoon that works with xvid's cartoon mode :devil: Well Miyazaki's movies would probably also work, but try it with GITS or even modern series (Naruto lol, or SAC) and it b0rks. So I'm not going to complain.

[trbarry]

I like these shootouts. :)

- Tom

[bilu]

Originally posted by Kedirekin
The *only* recent anime release I can recall is Spirited Away, and that was a limited release at that. Ghost in the Shell, Princess Mononoke, Dragonball Z were the most noticeable in my country in the recent past.

Do we want to forget Pokemon and Digimon? :D

EDIT: @syskin: like your sig ;)

Bilu

[bilu]

Originally posted by sysKin
Just to add my two cents to the anime part of the thread:
Sorry but that settles it. I love anime, test every xvid expermient with it and enjoy doing so. If Doom9 says he doesn't enjoy it, I won't mention anime ever again. I agree with that, Doom9 is just a guy like the rest of us and he does what he likes. Everyone is free to do their own benchmarks, and I'm sure that anime specialized benchmarks would be interesting - mf, MasterYoshidino,Gizmotech,Sirber and others are inside the subject and could do it if they're willing to. If not, they won't :D .

I just disagreed with Doom9's FAQ proposal, the text itself seemed like saying that anime is less important than cartoon for everyone. That's not true to everyone - but it is to Doom9 and that is what really matters. This is HIS benchmark done over what HE feels that is important - and EVERYONE is invited to do the same :)

EDIT: But if someone did an anime benchmark, maybe Doom9 would put a link ;)

Bilu

[mf]

I'm the type of person that just encodes stuff if it's HQ (the only exception is movies like Titanic or Pearl Harbor (:rolleyes::D @ CruNcher), which I won't ever touch for obvious reasons). I'm gonna do ex-driver the movie because the DVD is to lick your fingers for. So I guess I'm not very representative in the anime or cartoon thing. So maybe you might like to have someone else do a subchapter of the "animation" part, that covers anime, to fit in with the subculture thing (subculture->subchapter)? ;)

[BoNz1]

Thanks doom9 for the excellent codec comparison. Each time you manage to outdo your last comparison. The switching between screenshots and zooming rocks IMO. Thanks again :)

[Kedirekin]

We anime lovers should probably take this elsewhere, or at least out of this thread.

I just wanted to point out that Doom9 is correct; anime isn't as widely popular as we fans might wish. Of all the people I know face-to-face, I'm the only one I know that really likes anime. I have to go online to find other fans.

Even within the population in these forums (which most likely is heavily weighted compared to the outside world), anime is probably not as popular as we fans mights like to think.

BTW bilu: we're using different definitions of recent. I was thinking of the last 2 or 3 years.

[sysKin]

Hi again,
If I might, I'd like to point out a single small 'flaw' in the comparison (which actually helps XviD so I shouldn't mind, but I'll tell anyway).
It's the way speed is measured.
The average fps of 1st and 2nd pass seems right but isn't. Take a hypothetical codec that makes 1st pass at 50fps and second at 0.1fps. The average - 25fps - looks good but doesn't really indicate that it takes 20 days for a two-hour movie.

This example is extreme but it does show an advantage given to any codec that can make any pass faster.

The 'better' way would be to measure time of both passes, average the time, and then convert to fps.

BTW Happy New Year everyone,
Radek

[EDIT] As explained by Doom9 himself a bit later, I was wrong. I probably shouldn't write any posts at 4am on 1st of January... I promise I won't do that again, at last for a year :D :p

[twist3d]

I'm not complaining about the comparison, it's pro and big ups for Doom9 that he spent so much time in it.

I intended this thread to be discussion about what users would've wanted for codec settings, resizing filters etc. that'd effected the final scores of the codecs.

I'd still like comments about "what settings i would have used for encoding" from divx/xvid/3ivx/etc. users

[Doom9]

@syskin: I'm not sure if I follow but I think your assumption is wrong. I encoded a part of matrix, 10'000 frames long. I set up everything so that the two passes are done consecutively done, then the time it took (for both passes) is divided by the number of frames (in both passes).

Thus, in your example let's say we have a 25fps source that's 10'000 frames long, so it takes 200 seconds for the first pass, and 100'000 seconds for the 2nd pass, total 100'200 seconds divided by 20'000 encoded frames = 5 fps, not 25 fps.

[KpeX]

Big thanks to Doom9 for the shootout. An excellent read :).

[jggimi]

I for one would like to thank Doom9 for all his terribly hard work. Each comparison has been as fair as possible, has been meticulous, and has been a lot of work to produce. Each has also been an improvement on his previous comparisons.

This latest one, with it's new display format, is by far the easiest to read and comprehend. And I don't even mind that my most commonly used codec didn't fare nearly as well, in his opinion, as others.

The key is these comparisons were his opinion. We do not see the resulting video, only still images, and are free to make our own comparisons. There is sufficient information in the comparisons that we can duplicate his research, if we wish, or, of course, to do our own. With our preferred settings, too.

Doom9, you've helped me understand your interpretation of the limitations and benefits of the codecs I use, and have given me plenty to think about with codecs I have not yet tried. I appreciate it, and all the work you've put into these comparisons.

Thank you.

[Nazgul]

Just a quick question, Doom9. Did DivxNetworks mention any particular reason why they suggested Slow rather than Slowest(or Standard, for that matter)? Did they figure it was a good middle ground between quality and speed?

Personally I'd really be interested if someone took a closer look at the differences between the new P/Q options in Divx5.11, but I fully understand that these shoot-outs are no minor task and appreciate the work you've put into them. They're always interesting reads.

[Soulhunter]

Big THANKS for this shoot-out, really great work !!!

A happy new year ahead... ;)

PS: Could there be also a zip version of it to download ???

Bye

[larsc]

Originally posted by jggimi
I for one would like to thank Doom9 for all his terribly hard work. Each comparison has been as fair as possible, has been meticulous, and has been a lot of work to produce. Each has also been an improvement on his previous comparisons.

AOL and halleluja.

Soulhunter: There's a torrent link here (you'll need a bt-client):
http://forum.doom9.org/showthread.php?s=&threadid=67855


And I have a question too. It's about Q11 in the FAQ-section.
'Q11: Why did you use postprocessing to review the clips?'

I'm using ffdshow (just straight out of the box, I haven't touched any settings) for DivX/XviD/whatever playback on a box I've got setup for a home entertainment centre (no encoding to be done ever), am I missing out on something?
Should I install seperate decoders instead? I'd really prefer not to. :)
(sorry if this has been asked and answered)

[Doom9]

Should I install seperate decoders instead? I'd really prefer not to. Well.. I like ffdshow as a generic purpose solution but I think you should give the codec specific DS filters a shot.. having seen the effect those filters can have firsthand it might not hurt trying them out.

@syskin: I redid the calculations in my head when driving to new year's dinner (I was sitting in the back so I had plenty of time).. and even if you have a 200k image movie, the 2nd pass in your example is the only one that would matter so you'd have 200'000*0.1 = 2 mio seconds divided by 2x200'000 frames = 5fps as in my previous example so I think I'm doing the speed measurements is fair.

[sysKin]

Originally posted by Doom9
@syskin: I redid the calculations in my head when driving to new year's dinner (I was sitting in the back so I had plenty of time).. and even if you have a 200k image movie, the 2nd pass in your example is the only one that would matter so you'd have 200'000*0.1 = 2 mio seconds divided by 2x200'000 frames = 5fps as in my previous example so I think I'm doing the speed measurements is fair. Yes Doom9, I'm sorry bout the fuss - I misunderstood the way you measured speed. Everything's ok :) the comparison is great.

Radek

[sh0dan]

Great comparison - very detailed and very thorough!

:thanks:

A few minor points I noticed:

*) Scripts - I'd personally prefer lanczosresize - it seems like the codecs can handle the added complexity by now. SPR seems like it's better off with bicubic though.

*) Futurama - I'd personally prefer Donalds KernelDeInt over FieldDeinterlace. It provides better detail and a smaller amount of artifacts. My personal preference is KernelDeInt(threshold=4).

*) I noticed a slight color and luma shift in the Real Video, especially in the futurama toward slightly too bright/saturated images.

Still missing WMV9 in the comparison, even though nothing has changed since the last comparison. I would much rather have it than DivX3.

[Doom9]

Still missing WMV9 in the comparison, even though nothing has changed since the last comparison. I would much rather have it than DivX3.
I think the reasons are well documented. Until recently DivX3 was my reference codec and you obviously want to compare other codecs against the reference. Now that DivX3 is no longer the reference it will never again be included in a codec comparison (along with every other codec that is not improved). I'd rather swallow a bullet than test a codec twice. It is more than enough work to test new codecs and I am way too underpaid and busy to restest something I have already tested.
I have an estimate of how WMV9 would've rated though.. better than 3ivX and ND but worse than the rest of the codecs. Do you really want to bother with such a codec? At least ND is fast.

[Teegedeck]

Perhaps it would make sense to keep SBC in as an 'anchor'.

[sh0dan]

@d9: I completely agree - but has compression settings changed since the last test? If not, could the images be included in the very excellent comparison you've put up this time?

I'm not expecting you to change anything in the current shootout - just a suggestion for next time.

[avih]

:)
amazing job doom9.
congrats for yet another great codecs comparision.
the thumb-zoom is very cool.

cheers mate
avih

[Doom9]

No, I will not add the wmv9 screenshot. I just seem to unable to get the message across that images are useless. Maybe I should just not bother with them anymore but then nobody reads the comparison anymore.

People: screenshots are useless in 2/3rd of all cases (at the last)You don't see what I see when I review the clips. Adding images is completely useless if the codec in question has a) not been reviewed, and b) not been compared to the other participants.

I didn't just add the old divx3 screenshots.. I actually reviewed the clip again and compared it against the other codecs. If I hadn't done that I'd never have added the images (and I never will). Without review no images.

Oh and I don't know where any luma shifts might be coming from.. might have something to do with the VMR9 renderer... feel free to investigate. I'm really done with the comparison.. I don't feel the need to invest more time into this (and I didn't have any complaints from the Real people either). One month is more than enough time for something I don't even get paid for.

What I find really interesting is that the people who actually have to gain / loose something because they made the codecs, seem to be the ones with the least amount of complaints in the end. I guess it's vanity to try and satisfy everyone but I can't get that into my head. Maybe I need to bang it against a wall a couple of times...

[KAMiKAZOW]

Great work, Doom9. Thanks a lot.

Originally posted by bond
some people prefer to use wma9 because m$ pushes standalone support, but many independant (!) and scientific listening tests showed that wma9 cant cope with vorbis or aac (at as good as all bitrates) http://audio.ciara.us/test/128extension/results.html shows that WMA9 is quite good (better than Vorbis according to this test).