Doom9's "Codec shoot-out 2004" test results are out!

[SeeMoreDigital]

Yep,

True to his word Doom9 has been hard at work, and come up with the goods in the form of another comp test.

This one is the biggest yet, with 10 different codecs.... For more information look here: -

http://www.doom9.org/index.html?/codecs-104-1.htm



Cheers and many thanks Doom9


EDIT: Thread title changed to match Doom's.

[dvd_maniac]

Page Not Found Error...

[SeeMoreDigital]

Originally posted by dvd_maniac
Page Not Found Error... You're not wrong mate.... It's gone :eek:

A case of.... there it wasn't :D

It was there... coz I've still got it in my "temporary internet files" folder!


Cheers

[Doom9]

welcome to the effects of round robin.. not all the servers are updated at the same speed, and when we're talking about 43MB of image data (damn you PNG), it just takes a while. But by now all the servers are fully up-to-date.

[BoNz1]

Many thanks doom9, I haven't read it but your tests are always first rate. Maybe we could set up torrent for the ultra high bandwidth to help with bandwidth?

[nicco]

DivX Fusion 5.9
:confused:
What is it?

[dvd_maniac]

OK, There it "IS" again.

I just went through it and it is very informative. I am at the begining of encoding my entire dvd collection onto Hard drive. I want to create a media server using SageTV as a frontend. I have been trying to decide between Xvid and ND AVC for a while now and have done many tests. I do not seem to get good results using the AutoGK tool and giving the multitude of DVDs I am doing (1000+) I do not want a lot of steps. So I decided to do action, sci-fi movies in ND AVC. I still did not choose for slower paced, less scenic movies, either Xvid or ND ASP.

I was wondering (@Doom9), You were referring to ND Main Profile as giving you around 31 FPS?
Also, ateme's Main Profile encoder delivered a good 31.40 fps, which is very respectable for an AVC codec
It took me around 8+ hours to encode Matrix 3 in ND Standard AVC Profile. 1rst pass was quick around 70 minutes, the second pass was at around 6 FPS. What is the difference between Main & Standard?Should I be getting better FPS?

My machine is:
p4 3.0 800FSB
Asus P4C800-E Deluxe
ATI AIW 9800 Pro
2x512=1GB Cosair DDR500
2x120 Sata Raid-0 on-board raid controller
2x250 IDE Raid-0 Promise tx2 controller card
Sony DRU-530a & DRU-710 dvd-+rw drives.

[Doom9]

@dvd_maniac: I'm afraid I've never used AVC in Recode... I'm only familiar with the encavc application. But there's one thing.. in Recode you have the whole shebang, MPEG-2 decoding, resizing, audio encoding. And I'm not sure what kind of input preprocessing you had done.. all of that can have an influence. I used a very simple AviSynth script, and the encoding speed is obviously for the video only.

DivX Fusion 5.9It is a video codec by DivXNetworks that you cannot currently have. I suppose the official beta program will start shortly. And in the end.. what comes after 5.9? ;)

[krackato]

Looks like Nero Recode and their AVC codecs won this shootout. Congrats to the Nero team for their excellent codec.

[dvd_maniac]

Thanx Doom.

I want to thank you for all your hard work in helping me, and I'm sure many others, in deciding what codecs to use.

Thanx

[IgorC]

there are some words like

ºIf anybody claims any codec can deliver DVD quality at 700MB for a 2h movie, have him encode Matrix3 and compare it to the DVD. And if he still thinks so, send him to an eye doctor.º

i´ve ripped Matrix3 by Nero H.264 (714 Mb 1Cd rip). NTSC 720x304, 23.976 FILM fps, 760 kbit/s + HE-AAC 52 kbit/s. Of course it isn´t a DVD quality.
But quality is great and big part of details was preserved.

i think it will be (less or more) corect to conisder rip ´loseless´
2-3 movies per DVD.
Very interesting article. Great job. and of course NERO AVC is great winner. Waiting for new versions of XVID 1.1 Divx 5.9 and anothers.
Good luck to all and HAPPY NEW YEAR to all members of doom9!!!

[Emp3r0r]

Thanks Doom9 for the hard work you endured while putting together the new codec shootout. Great job!

I'm looking forward to testing some of those other codecs when they are released to the public. I'd like to see them perform with higher bitrates and HDTV resolutions. XviD works perfectly in this niche for now.

[KpeX]

Great job Doom9, especially previewing so many up-and-coming codecs.
If anybody claims any codec can deliver DVD quality at 700MB for a 2h movie, have him encode Matrix3 and compare it to the DVD. And if he still thinks so, send him to an eye doctor.Ditto.

many player makers specifically advertise XviD playback as a feature - which shows that an open source project, with no advertising budget whatsoever, can clearly make a significant impact in today's world).Good point. AFAIK this was never even a goal of the xvid developers - it has reached this popularity purely by providing excellent quality.

if I were a broadcaster or potential user of HD DVD/ Blu-Ray, seeing the result of WMV9 and ateme's AVC codecs, I'd definitely go for the AVC standard. In terms of current implementations, AVC definitely blows away WMV9 - hopefully HD-DVD producers will make decisions based on quality and not influenced by marketing/corporations.

[CruNcher]

Yep the outcome of the test is personaly not a suprise but it's well done nice work doom9 thx for it, it also confirms my own tests relating to the Speed/Quality Ratio of XviD vs ND AVC.
For me the HDX4 results are very interessting it doesn't look that bad for the start :)

@ KpeX HD-DVD uses very high bitrates you can't take the results from this shoot-out and apply them to HD-DVD.

[Leak]

Originally posted by Doom9
And in the end.. what comes after 5.9? ;)

5.10, as always.

Why do you ask? :D:D:D

[Latexxx]

I would have liked to see wmv 9 advanced aka vc-1 high profile. It comes with wmp10 and you encode it using wme 9. And another thing. If I'm not completely wrong, wmv9 vcm doesn't have b-frames whereas Windows Media Encoder does them.

[Latexxx]

Somebody should merge this thread and http://forum.doom9.org/showthread.php?s=&postid=586284 .

[darth rosenberg]

How can you actually "judge" some rate control mechanism of a codec that can be put into avi? What is to be used as "reference"?

Example:
"normal" AVI: 1,094,551 kB
remuxed with low overhead settings: 1,086,580 kB
(103min video, 1x mp3-vbr)

[Teegedeck]

Thanks for another great test, I'll have fun reading it back and forth for a day!

And congratulations to the guys at Ateme; as said before, you did a very good job!

[dragongodz]

Doom9 - thanks for another interesting read.

again if people want to start saying "i would like to see this" or "you should have done it this way" etc. please DONT.
that happens after every shootout and Doom9 has said many times if you think something should be different then you either come up with an amazingly convincing reason or you do the tests yourself.

[darth rosenberg]

again if people want to start saying "i would like to see this" or "you should have done it this way" etc. please DONT.In other words, suggestions are not welcome, unless you can read someone else's mind? Pointing out that some information could be misleading isn't supposed to be done either? Aha

[Doom9]

@Latexxx: The advanced profile contains two major improvements: interlaced encoding, and transport stream independence (they call it something else I think).. both are completely uninteresting for DVD backups (well, interlaced encoding might be but I only encode progressively). But that's basically it.. we're not talking about quality improvements and such. And, you can easily use B-frames in the VCM.. you can enable them by making an appropriate registry entry. same thing goes for the loop filter (which was activated for the encoding). I was told not to use B-frames. The subject of using VCM or WME also came up, it was the decision of the Microsoft rep I was talking with prior to the test to use the VCM rather than the WME.

For those interested, here's how you activate the LoopFilter and B-frames for the WMV9 VCM:

Loop Filter:

This can be enabled by creating the following DWORD reg key (HKEY_CURRENT_USER\Software\Microsoft\Scrunch\WMVideo\Force LoopFilter) and giving it a value of 1. It can be disabled by setting it to 0.



B Frames:

This can be enabled by creating the following DWORD reg key (HKEY_CURRENT_USER\Software\Microsoft\Scrunch\WMVideo\NumBFrames) and giving it a value from 1 to 7. I usually use 1-3. It can be disabled by setting it to 0.

[slavickas]

and using b frames u'll get delay as much b frames used, even in wme

[Doom9]

@dragongodz: I have a 16 question FAQ for just that purpose so I don't waste my time replying.

How can you actually "judge" some rate control mechanism of a codec that can be put into avi?That question makes no sense to me. What does the container have to do with anything? The container overhead can be calculated, so can the size of the audio file. As long as you use the same settings and tool, the overhead should be constant. And if you add video size + audio size + overhead you have your final size accurately regardless of the container. I could even argue that without ateme's exact overhead calculations for MP4, they'd have a considerable disadvantage because so far nobody came up with an accurate formula.

And, just to make you feel happy, I do have all my source files and I checked for Matrix3: video size + audio size + 7 MB (constant for all codecs) = final size.

Just because there is a low overhead mux mode in AVI Mux GUI (which can create problems as the author noted), doesn't mean using the same tool and settings gives a different overhead. Now I could get into KBs as well, but clearly when we're talking about the considerable oversize or undersize we've seen, that's the rate control. Same goes for extreme image degradation as the action rapidly picks up.. that's something I know from past experience with codecs.

Pointing out that some information could be misleading isn't supposed to be done either? Not when it is without merit.. then it's just trying to make me look bad.

[darth rosenberg]

The container overhead can be calculatedThat's the point. It can't, with the only exception being OGM. With AVI and MKV, there is a bunch of possible things you can do to get the overhead down to 1/3 or 1/4, compared to the "worst" settings.

Of course, in your comparision, some codecs have missed the desired size by more than 20 MB, which is a clear failure, but especially those which are off by not more than 5 MB are within the range the container and some settings can cause for a 1 CD rip.

1482000 kB instead of 1433000 kB is a complete failure (hell, that is even beyond what you could overburn on 2 normal CDs), full ack, but 1440000 imho isn't

[celtic_druid]

Doom9 did all the encoding himself though so he would know what settings and therefor the overhead. We aren't talking about some random files here. If you really wanted you could dump the video to raw streams. No overhead issues then.

[darth rosenberg]

That's exactly the point. The codec can impossibly know what container and with what settings the user is going to store the file. Thus, the codec should only be blamed for not hitting the desired size if it is off by far, like 1482000 instead of 1433000.

Another possible criterion would be if it is oversized by more than 1.2% (because 1.2% is the overhead of OGM and the highest one, compared to others). Then, it is clear that the codec produced an oversized file and that using other containers and/or setting wouldn't "fix" it

[Koepi]

Thank you for the great comparison, Doom9! Sure much work again.

DarthRosenberg:
Encoding has been done through vdub into avi for ages. This is a known procedure. It is the application with it's avimultiplexer which gets used most widely. So there many codecs can do their scaling stuff properly. It doesn't matter if you mux that into other containers (i.e. "low overhead avi"=new container). That wasn't asked in this test. It's about codecs. Not a container shootout. You're mixing things up there %)

I consider Doom9's observations valid.

Regards
Koepi

[celtic_druid]

Well no I think that you are missing the point. You enter a bitrate taking into account the overhead of the container and audio, then if the codec can't hit it your final size will be off due to the codecs rate control. The size would also be off if you expected the codec to take into account container overhead when inputting the bitrate.

The codec doesn't need to know the container/settings, the bitrate doesn't take that into account, that is upto the user.

[SeeMoreDigital]

dvd_maniac

Don't forget when it comes to "encoding speed", the fps rates increase as the encoded pixel frame size decreases!

As you may have read elsewhere on the forum, I recently encoded Terminator 3 to an 720x576 (1:1) 800MB file - using all of NeroDigital's maximum quality Mpeg4/AVC "bells and whistles", complete with 6Ch AAC-HE audio @ 128kbps VBR.

The second pass encoding time was very slow and never seemed to go above 4fps. But would have been significantly quicker if I had encoded to an 640x272 square pixel frame.

But there's no denying that the finished encode looks (and sounds) astonishing, even during the "Demolition Derby" chase scenes. Even when watching on a big screen it's virtually impossible to tell that you're not watching the DVD source!

I can't wait for Mpeg4/AVC hardware players arrive!


Cheers

[temporance]

First of all thanks doom9 for all your hard work. I intend to add my view to this thread, just as I did last time around. Hope that's OK ;)

I intend to replicate at least some of these encodes (meaning a trip to Blockbuster later today!). I remember that in the last comparison, the bit-spend in the movies' credits varied so much between codecs that the bitrate during the movie was affected significantly +-15% iirc.

I'd like to find out the actual bitrate of the segments that were scrutinized (will be different for each codec) and also the frame types and sizes in the region of the screenshots.

Anyway, thanks again for the comparison... I'll be working on some considered constructive feedback.

[darth rosenberg]

Koepi: muhahahahahahaha I knew you'd show up when mentioning OGM that way.

If you knew anything about containers, you'd know that for example MP3 CBR causes almost no overhead in AVI, whereas MP3-VBR causes a lot, at least with VDM/NanDub (you want to use VD(M)/NanDub-AVI as reference).

So if the codec wants to hit the desired size of the final file accurately with an offset of less than 2 MB, it would have to know if the user is going to use VBR or CBR for MP3. All that happens totally indepentant from some muxing settings... those only would make it even worse

[Tommy Carrot]

Darth, please create a separate thread for your problem in the containers forum, it has really nothing to do with Doom9's codec test and it's largely uninteresting.

[darth rosenberg]

Read before replying, and you'll see that it does. It addresses the problem of target file sizes with codecs, especially the 2.5 MB limit Doom9 has chosen because a 700 MB disc has 702,83 MB of space without overburning, and the problem that variations in size within that limit (or even not within that limit) can be caused by influences the codec can't know about or even control, like if the user is going to use vbr or cbr mp3 (unless it asks, of course)

[Koepi]

Read again: the codecs were asked to hit a certain video size. Without any audio.

-> Troll <°|)^))<

[darth rosenberg]

I used a 128kbit/s ABR audio track created in lame 3.93 (using --alt-preset 128 setting) for all movies ... As you may know, not every rate control mechanism is perfect so here are the final movie sizes I got: .... <= exactly there. Yellow / Red "cards" were given if the final file size was off by too much, not for the plain video sizes.

-> Troll <°|)^))<Nice moderators :) Will i be striked if I call you a troll? You obviously have not even read the first page and seriously try to discuss it :oo

I'm saying that the limit of 2.5 MB for getting a "red" in the intro is too strict, because it can still be "fixed" without reencoding to some extent, and all you can say is "troll" :)

[Koepi]

If you quote, quote correctly. The sentence following your quote is the important one:

The size of the audio track was 117'263 KB, 157'455 KB and 20'582 KB respectively. This resulted in an effective video bitrate of 621kbit/s for Matrix, 1016kbit/s for SPR and 987kbit/s for Futurama (here kbit = 1000bit).

There you have the bitrate. The bitrate isn't hit by the marked codecs. Simple as that.

Maybe we can turn over to "real issues" concerning the codec comparison?

Thank you.

Cheers,
Koepi

[Doom9]

actually, I love if people give me an opportunity to beat them with their own facts. dart rosenberg suggested my size findings were irrelevant because the AVI overhead could not be accurately calculated. Well, I present to you the full stats of the entire codec comparison, wherever the AVI container was used:

Codec Final Size Vid size Audio size Overhead

Matrix3

3ivX: 735'541'248 607'709'184 120'077'312 7'754'752
DivX: 734'840'832 607'008'768 120'077'312 7'754'752
HDX4: 734'300'160 606'468'096 120'077'312 7'754'752
VP62: 760'213'504 632'379'392 120'077'312 7'756'800
VSS : 746'326'016 618'491'904 120'077'312 7'756'800
WMV9: 719'798'272 591'966'208 120'077'312 7'754'752
XviD: 733'988'864 606'156'800 120'077'312 7'754'752

delta overhead: 2048 bytes

SPR


3ivX: 1'467'699'200 1'296'295'936 161'233'640 10'169'624
DivX: 1'469'247'488 1'297'844'224 161'233'640 10'169'624
HDX4: 1'468'522'496 1'297'119'232 161'233'640 10'169'624
VP62: 1'518'467'072 1'347'063'808 161'233'640 10'169'624
VSS : 1'487'163'392 1'315'762'176 161'233'640 10'167'576
WMV9: 1'474'691'072 1'303'289'856 161'233'640 10'167'567
XviD: 1'468'395'520 1'296'994'304 161'233'640 10'167'567

delta overhead: 2057 bytes

Futurama

3ivX: 182'532'096 160'155'648 21'075'704 1'300'744
DivX: 183'488'512 161'112'064 21'075'704 1'300'744
HDX4: 183'347'200 160'970'752 21'075'704 1'300'744
VP62: 183'177'216 160'800'768 21'075'704 1'300'744
VSS : 169'793'536 147'417'088 21'075'704 1'300'755
WMV9: 172'410'880 150'034'432 21'075'704 1'300'744
XviD: 183'416'832 161'040'384 21'075'704 1'300'744

delta overhead: 0 bytes

As you can see, we have a whopping overhead difference of 2KB for Matrix3, 2KB plus 3 bytes for SPR, and 0 bytes for Futurama. So that ends this argument once and for all. The numbers clearly speak for themselves and they allow to draw only one conclusion: Knowing audio size and the final size, Gordian Knot allows to derive an exact video bitrate and final video size, and the only reason for an over/undersize of the final video is if the video bitrate is not kept (and thus the size of the video without audio is off).

can be caused by influences the codec can't know about or even control, like if the user is going to use vbr or cbr mp3 (unless it asks, of course)But you can configure that in Gordian Knot.. do you really think I'm that stupid not to take that into account? GKnot can not only accurately calculate the overhead for CBR MP3, VBR MP3, anc AC3, but also a two audio stream combination thereof. Sure, if people do not properly calculate their bitrate, then you have a problem, but being in the business for 5 years now you should give me a little more credit than to commit one of the most basic newbie mistakes.

[darth rosenberg]

As you can seeTo be more precise: As I can see now ;)

OK, now something more interesting: Did you test which of the codecs allow to be run on 2 instances at once, e.g. on a HT or DualCPU system? I remember some old xvid alpha version having issues with that (issues = crash)
But you can configure that in Gordian Knot.. do you really think I'm that stupid not to take that into account?Do you know how many people out there actually would in your place? I can't smell things you didn't write down....

[Doom9]

I'd like to find out the actual bitrate of the segments that were scrutinized (will be different for each codec) and also the frame types and sizes in the region of the screenshots. I can tell you that the frames were all non I-frames but that's about it.. not every codec even offers the facility to find out the frame type. And, in the end it should not matter and you don't really have an influence on whether a frame is a P-frame or a B-frame (I-frame as well.. but at least there's a good change with I-frames that you have one at a scene change). I'm afraid cutting out the reviewed scenes would be too much work and I don't really see the reason behind that.. the point of taking scenes throughout the movie is to ensure that if rate control for a codec give significantly less bits in a scene, this would be compensated for in another scene and on the average you should catch that.. of course, only a small sample was taken, but I doubt that we'd see huge variations.

I see the merit in those findings but IMHO it's way to little to justify the significant additional effort. In the end, as long as the rate control hits the size target, the user should not worry about where the rate control attributes the bits but assume the rate control knows what it does.. and if there are significant problems (like in VSS), my samples should include it at some point.

To be more precise: As I can see nowWell, but you had to first accuse me of delivering improper results, without so much as any proof whatsoever. To put it midly, that's not very nice.

Do you know how many people out there actually would in your place? I can't smell things you didn't write down....Let me answer that with a quote from the articleI used GordianKnot for the bitrate calculations (it can calculate the effective overhead of using VBR MP3 audio).
So there you have it.. it can effectively calculate the overhead of using VBR MP3 audio. Do I have to use 20pt fonts in bold red? And GKnot also considers the raw AVI overhead even if you don't add any audio, not just the audio muxing overhead.

[Koepi]

You need to ask the codec coders for getting that information, it is hard to test a codec for beeing thread safe if you don't know the internals.
The test for that will be huge: you'd need to start 2 encoding instances with _all different_ settings. Then you need to use the same settings on encodes with only one instance at a time. Then you can do a diff on the results.
There you'd see if the encodes were affected by global variables or shred memory usage. But you couldn't tell _where exactly_ the error is (well, this part wasn't really your question though).

Those errors culd sneak into every codec at any stage (and be solved the next update) without anyone even noticing this (I mean the developers here).

I suggest you do such tests and report the results?

Cheers
Koepi

[darth rosenberg]

I can only say that it appears to work flawlessly with xvid 1.0.1. One problem: It doesn't lock the video.pass files for write access. You can use the same filename in both instances and get 2 bad files (that's of course an error only the user is responsible for, but nevertheless one the codec could catch) :D Though i'd have to retest with the latest build

[babayaga]

Originally posted by SeeMoreDigital
The second pass encoding time was very slow and never seemed to go above 4fps. But would have been significantly quicker if I had encoded to an 640x272 square pixel frame.
Doom9 was given a faster version than the one currently in Recode (about 20% faster) and we did not choose the settings that give the absolute best quality but a tradeoff between speed and quality like Xvid for instance.

Regarding the quality settings (in-loop deblocking set to -2 and psycho-visual enhancements set to 2), it is a tradeoff between blocs and detail preservation. Unfortunately, I did not realize that Doom9 was going to watch the movies on an LCD screen where blocs are much more annoying (I would not trade my old CRT :)

[Doom9]

I would not trade my old CRTAgainst a 23" 16ms 16:10 TFT with a native HDTV resolution? Come on. Everybody who ever comes into my room gets really jealous.

As far as HyperThreading goes, the Athlon can't do it, and my P4 is too old to offer Hyperthreading , and those are all the machines I have. Now that I think about it, I have a W2K3 server at work with a 3.2 GHz HT CPU that I could've taken home over the holidays and get the encoding done even quicker.. but it's too late now. But there'll be dual core Athlon64 next year, so..

[bond]

doom9, thanks a lot for this great comparison!

i hope next time we will see a ready to rumble x264 in there too, its already very good imho and already doesnt need to hide itself :)

[SeeMoreDigital]

Hi babayaga,

20% faster for the new release!

Before the new release comes out, what settings do these equate to in the current existing release?


Cheers

And I'll stick with my current plasma!

http://img134.exs.cx/img134/1847/smdbigscreen6to.jpg

[Doom9]

i hope next time we will see a ready to rumble x264 in there too, its already very good imho and already doesnt need to hide itselfI made an addendum to Q6 that I consider very important, but that's probably often missed.

[neo75903]

@doom:
Thanx for the great codec shootout.

@seemoredigital:
wow nice plasma, you are probably the one who can clearly see the differences between the codecs :)

[Doom9]

wow nice plasma, you are probably the one who can clearly see the differences between the codecsActually, LCD screens offer a better picture than Plasma (and I think they're supposed to last longer as well).. hence the price difference between a plasma and an LCD TV of the same size. And for that kind of diagonal, I'd like a higher than HDTV resolution ;)

[babayaga]

Originally posted by SeeMoreDigital
Before the new release comes out, what settings do these equate to in the current existing release?

It's the same since we've not shown new features for this comparison.

BTW it's unclear to me that 2 references are so usefull on the 2 movies.
Originally posted by SeeMoreDigital
And I'll stick with my current plasma!

For watching movies, I'm not sure yet but i'm old fashioned :) Anyway, flat screens are for sure the trend ...