Inventive Software Codec Test 2005 (Results are coming in)

[Inventive Software]

OK, I've decided to take the plunge and do a codec test. The plan is located here (http://s6.invisionfree.com/ivs_online/index.php?showtopic=3), and is also attached. The document is in Word format, and can be open in WordPad. For now, here's the lowdown:

Codecs to be tested:

3ivx D4 4.5.1
DivX 5.2.1 Pro
DivX Fusion build 1528
XviD 1.1.0-beta2
On2 VP6 6.2.6
On2 VP7 7.0.6
Microsoft Windows Media Video 9 VCM
Microsoft MPEG-4 V2
x.264 rev 270
Xiph Theora 1.0-alpha4 (via ffdshow)


The system to be tested on:

See my signature for the system, or visit the planning page.

I hope this will be accurate. Anybody who wants their codec in this test, PM me or mail me, and if I approve, gimme a build that works for my machine and with some pre-testing I may include it in the test. This is my attempt at the ultimate codec test, albeit on the not-so ultimate system. Please support my attempt. I have been planning this for the last couple of months, and have decided to bite the bullet and go for it. :)

Expect the rest of the results to be posted within the next 3 or 4 weeks.

If you find any errors or have any queries worth mentioning, please either PM or e-mail me, or post them here. I will try and get back to you as quickly as possible. I don't have 24-7 internet access, and can only get on the internet a couple of times a week.

Also, with regard to results, do you want them as they come, so you can rant at me if I'm doing it wrong, or should they all be posted at once when they're finished. Be warned that if the second option is preferred, the test is likely to take about a month to complete, and if the first is preferred, you can marvel at just how slow my system really is. Bold is the comparison speed.

RESULTS

DVD: Simpsons Roasting On An Open Fire

Codecs

3ivx D4 4.5.1 1st Pass: 6.669 FPS
3ivx D4 4.5.1 2nd Pass: 7.287 FPS
3ivx D4 4.5.1 Average Encode Speed: 6.964 FPS
DivX 5.2.1 Pro 1st Pass: 7.028 FPS
DivX 5.2.1 Pro 2nd Pass: 7.709 FPS
DivX 5.2.1 Pro Average Encode Speed: 7.353 FPS
DivX Fusion 1528 1st Pass: 6.249 FPS
DivX Fusion 1528 2nd Pass: 6.171 FPS
DivX Fusion 1528 Average Encode Speed: 6.210 FPS
Microsoft MPEG-4 V2 Single Pass: 9.278 FPS
Microsoft WMV9 VCM 1st Pass: 5.007 FPS
Microsoft WMV9 VCM 2nd Pass: 1.867 FPS
Microsoft WMV9 VCM Average Encode Speed: 2.720 FPS
On2 VP6 6.2.6 1st Pass: 2.494 FPS
On2 VP6 6.2.6 2nd Pass: 2.588 FPS
On2 VP6 6.2.6 Average Encode Speed: 2.540 FPS
On2 VP7 7.0.6 1st Pass: 7.059 FPS
On2 VP7 7.0.6 2nd Pass: 5.062 FPS
On2 VP7 7.0.6 Average Encode Speed: 5.896 FPS
x264 rev 270 1st Pass: 3.628 FPS
x264 rev 270 2nd Pass: 3.586 FPS
x264 rev 270 Average Encode Speed: 3.607 FPS
Xiph Theora 1.0-alpha4 1st Pass: 3.078 FPS
Xiph Theora 1.0-alpha4 2nd Pass: 3.614 FPS
Xiph Theora 1.0-alpha4 Average Encode Speed: 3.324 FPS
XviD 1.1.0-beta2 1st Pass: 7.607 FPS
XviD 1.1.0-beta2 2nd Pass: 6.467 FPS
XviD 1.1.0-beta2 Average Encode Speed: 6.991


DVD: Spooks Episode 1

Codecs

3ivx D4 4.5.1 1st Pass: 7.942 FPS
3ivx D4 4.5.1 2nd Pass: 8.127 FPS
3ivx D4 4.5.1 Average Encode Speed: 8.034 FPS
DivX 5.2.1 Pro 1st Pass: 6.651 FPS
DivX 5.2.1 Pro 2nd Pass: 6.760 FPS
DivX 5.2.1 Pro Average Encode Speed: 6.705 FPS
Microsoft MPEG-4 V2 Single Pass: 11.401 FPS
Microsoft WMV9 VCM 1st Pass: 5.507 FPS
On2 VP6 6.2.6 1st Pass: 4.159 FPS
On2 VP6 6.2.6 2nd Pass: 4.366 FPS
On2 VP6 6.2.6 Average Encode Speed: 4.260 FPS
On2 VP7 7.0.6 1st Pass: 8.351 FPS
x264 rev 270 1st Pass: 2.943 FPS
XviD 1.1.0-beta2 1st Pass: 8.998 FPS
XviD 1.1.0-beta2 2nd Pass: 6.467 FPS
XviD 1.1.0-beta2 Average Encode Speed: 7.525 FPS


DVD: Bill Bailey Part Troll

Codecs

3ivx D4 4.5.1 1st Pass: 7.100 FPS
3ivx D4 4.5.1 2nd Pass: 7.208 FPS
3ivx D4 4.5.1 Average Encode Speed: 7.165 FPS
DivX 5.2.1 Pro 1st Pass: 6.252 FPS
DivX 5.2.1 Pro 2nd Pass: 6.579 FPS
DivX 5.2.1 Pro Average Encode Speed: 6.411 FPS
Microsoft MPEG-4 V2 Single Pass: 9.372 FPS
XviD 1.1.0-beta2 1st Pass: 7.697 FPS
XviD 1.1.0-beta2 2nd Pass: 6.564 FPS
XviD 1.1.0-beta2 Average Encode Speed: 7.086 FPS


DVD: Pirates Of The Caribbean

Codecs

3ivx D4 4.5.1 1st Pass: 7.882 FPS
3ivx D4 4.5.1 2nd Pass: 7.906 FPS
3ivx D4 4.5.1 Average Encode Speed: 7.894 FPS
Microsoft MPEG-4 V2 Single Pass: 9.821 FPS


Anything above 6 FPS is fast on my system.

To give you a real world example of how fast the codecs really are I'd have to reconduct the test on a super-powerful system. For now, to give a rough indication, 6 FPS is real-time.

[edit]Removed some of the codecs. I had teething problems with RealVideo 10, so have dismissed it this time round. VP3 is not going to be tested as Theora is basically VP3. XviD 1.0.3 has been removed as well.

[edit]Some of the planning has been refined somewhat. Just the day-to-day test will be carried out now. x264 was updated because 2-pass was broken in the build I had. Also, many more results have been done, just not posted yet.

Inventive Software

[Sirber]

Could you add RealVideo 10? Thanks!

[Nic]

Shouldn't you really test VP7 as well if you are going to test the other VP versions?
http://www.on2.com/register_trial.php?class=VP7

Good luck,
-Nic

[Gannjunior]

and Nero Digital AVC?

[Sharktooth]

If you had carefully read what inventive software wrote on his forum you would have spotted this:
Nero Digital, RealVideo 10 and Windows Media Video 10 will also be included in the next test, as will all the other codecs listed above. If any more come to my attention worthy to be grilled later in the year, I shall consider including them.

[Manao]

I wonder why you want to test VP3, since you're testing both VP6 and theora ( which can be seen as the closed source and the open source evolution of VP3 ).

[stephanV]

Just nitpicking:

Container Format: AVI 2.0 Legacy
Giving a Legacy AVI the number 2.0 is a bit weird, call it AVI (Legacy) or AVI 1.0 :)

Audio / Video Interleave Gap: 500 ms (0.5 seconds)
Just wanna make clear that this is not the standard setting, so do not forget to set it. ;)

One question though... why not NanDub instead of MS-MPEG4 v2? it should at least enable you to do a 2-pass.

Good luck!

[kwtc]

A final graph of all the encoding performances will tell just how each codec performed. The table will be done in Excel, so the graph will be easier.


Your test is well prepared; however I fail to see what performance measure you will use to compare the results.

Will you measure performance with a metric, with a personnal note given to each codec, with speed ?

Then each codec offers a tradeoff between encoding speed and quality. As I understand you will select the codec quality level; in that case most comparisons involving speed is irrelevant. On the other hand, you may also select the speed and choose the best parameters for that speed (if possible) for each codec. This method has its advantage and drawbacks; MPEG-4 AVC codecs will always be slower than MPEG-4 ASP ones...

[Inventive Software]

If anybody can point me to a binary of RealVideo 10 I'll happily test it this time round. :D

Your test is well prepared; however I fail to see what performance measure you will use to compare the results.

Will you measure performance with a metric, with a personnal note given to each codec, with speed ?


If you've read the other codec tests that Doom9 has done, you'll see he uses the encoding FPS to measure. I'll do much the same.


This method has its advantage and drawbacks; MPEG-4 AVC codecs will always be slower than MPEG-4 ASP ones...


Noted, and will be mentioned in the results. Still, it's still a valid comparison, to see just how good the picture quality really is with AVC compared to ASP.

Can somebody PLEASE point me to a recent ffdshow build, PLEASE???

[celtic_druid]

http://celticdruid.no-ip.com/xvid/ <-- my ffdshow compiles.
https://helix-producer.helixcommunity.org/downloads <-- helix producer, but you really should check out the real video forum here for a GUI.

[On2Tech]

A couple of notes on your test.

Note that a personal use version of Vp6 is now available at our online store.

I suspect your suggested VP6 settings are not optimal. In particular I would suggest a higher MinQ value as a starting point. Perhaps 2-4 for the 1.5Mb/sec clip and something in the range 4-8 for the other two. Also I would suggest your VBR settings do not provide much scope for VBR. For other tests we have done recently we have been defaulting to 70 variability and 10-25 and 200 for min and max section rates.

I am sure we can get you a copy of the latest version of VP7 due to be released in the next day or so. This version has improvements in the first pass and is thus less sensitive to the value you give to MinQ. You might want to set a MinQ of 2 for the 1.5Mbit clip but otherwise the defaults for MinQ, MaxQ, and other vbr settings should be OK.

The latest version of VP7 also offers a speed slider when good quality mode is selected. This enables you to adjust the speed vs quality trade off. You might want to compare the quality vs encode time for a couple of the good quality modes as well. At the slow end, good quality is roughly twice as fast as best quality. It is usually within about 0.1db on PSNR and for SSIM it actually beats the best quality mode of the previous release. With the speed slider at the opposite end of the scale, on my test machine (a 3.2Ghz P4) it encodes the HPII test clip used by Sagittairre in his tests at over 34fps and still beats the likes of WM9, XVID and DIVX on both psnr and SSIM.

On the timing front I notice that you say you are only going to time the second pass. This is slightly unfair as there is a trade off between how much work is done in the first pass and the second pass. For example VP7’s first pass is blindingly fast, but other codecs may gather a lot of additional information that can be used to speed up the second pass. In such cases the first pass may be as slow or even slower than the second. I would suggest you look at combined first and second pass encode time. This also makes comparison with one pass encoders fairer.

One final warning. If you do metric tests on VP7 you will need to add an entry strCpuFree to the registry for VP7 and set it to 0. The current versions auto-adapt post processing according to cpu loading. This is designed to help insure smooth playback on lower end machines but of course when you are running a metric there is loads else going on and post processing may get turned off. The strCpuFree 0 disables this feature such that the chosen level of post processing is used all the time. I would recommend strPostprocessingLevel of 2 for most situations.

On2Tech

[Yong]

Originally posted by stephanV
...One question though... why not NanDub instead of MS-MPEG4 v2? it should at least enable you to do a 2-pass.

Good luck!
iirc ffvfw MSMPEG4-V2 also can do 2-pass(and less buggy than nandub...), mencoder can even do nth pass encoding~...:D

@Inventive Software:
EDIT: i wonder why still include the old MSMPEG4-V2 in the codec test...
Why don't include libavcodec's SNOW?:D
though it is still an experimantal codec.:)

[virus]

Originally posted by Inventive Software
x.264 rev 215
Beware that revision 215 has a bug when using B-frames as references that may show bad artefacts and force you to drop your testclip altogether.
So either keep B-refs off (personally, I'd recommend to leave them off in any case for natural material right now) or update to rev. 216 where the problem has been fixed.

[LordRPI]

What about the latest beta codec from DivX Labs (http://labs.divx.com/)?

You'll also find the writer of Spooks and I share the same name. No, we are not the same person.

[guada 2]

Which serious on behalf of all one each one.
And if by chance, all these tests proved to be negative or strange.
Which attitude would adopt:disappointment or interpellation?

[planet1]

Though I don't understand why Microsoft MPEG-4 V2 should be included (DivX 3.11 isn't used anymore, so why go preDivX ), why not unlocking Microsoft MPEG-4 V3 ?

http://www.undercut.org/msmpeg4/


IMHO a common container for the MPEG video codecs should be used, unless there is someone here who thinks AVI is the desired storage platform for H.264. :sly:

[Inventive Software]

OK, after careful consideration, VP7 and RealVideo 10 will be included.

@On2Tech: Thanks for the info and it will be taken on board. I think I will change how the timing is measured, possibly an average of the two. I dunno, I'll sort something out.

@Yong: The original MS-MPEG-4 V2 will be used because it's the original. Surely the question is why not? It's a fast codec and yields good quality, which is what we're after here, right?

@LordRPI: I assume you are referring to the DivX Fusion codec? If so, that will be directly compared to DivX 5.2.1. If you have a relatively stable build of DivX 6 available, that can be included and tested if you wish. Or is that too much to ask? ;)

@Celtic_druid: Thanks for the links for ffdshow and Helix Producer. They will be pre-tested well.

[SeeMoreDigital]

When testing the Mpeg4 codecs.... given that 3ivx D4 4.5.1 does not offer ASP encoding implementations such as B-VOP, GMC and Qpel, may I ask whether it's intended to generate comparison tests with DivX 5.2.1 Pro, DivX Fusion build 1528, XviD 1.1.0-beta2 and even NeroDigital, without using their ASP encoding implementations too?


Cheers

[Sagittaire]

IMO you must test NDAVC

If you want this test could be better ...

MPEG4 ASP class codec test
3ivx
DivX Fusion
XviD
HDX4
NDASP
...

MPEG4 AVC class codec test
NDAVC
x264
...

Proprietary class codec test
WMV9
RV10
VP6
VP7
...

Open source class codec test
x264
xvid
Theora
Snow
...

best codec for each class and for global class ...

[vlada]

I would also suggest testing LAVC codec from Mencoder. I heard it is even better then XviD, so it might be the best MPEG-4 ASP codec. But I never saw this one in a comparison. I wonder why.

Regards,
Vlada

[Inventive Software]

@vlada: Probably because it's not well known enough to be used as much as XviD et al. I'd never heard of it.

I'm seriously considering Nero Digital AVC. Only problem is, it's a 30 MB download, and I don't have time to wait for 30 MB to download.

@Yong: About the SNOW codec, it's still very experimental, and it only supports quality settings. To be accurate enough ideally I need bit-rate support.

@Sagittaire: Nice thinking, and it'll probably be done that way next time, but for now, the way I've described is the way it's gonna be done. Sorry. I know that the AVC codecs are gonna be slower than the ASP ones, especially with the high quality settings, and that's just the way they're designed. Saying that, NDAVC looks pretty quick on Doom9's tests. Maybe there's a way out for AVC.

[LordRPI]

Originally posted by Inventive Software

@LordRPI: I assume you are referring to the DivX Fusion codec? If so, that will be directly compared to DivX 5.2.1. If you have a relatively stable build of DivX 6 available, that can be included and tested if you wish. Or is that too much to ask? ;)

I'll see what I can do. Check your PM's for my email addy.

BTW in the US we call "Spooks" "MI-5"

[bobololo]

Originally posted by Inventive Software
@vlada: Probably because it's not well known enough to be used as much as XviD et al. I'd never heard of it.

I'm seriously considering Nero Digital AVC. Only problem is, it's a 30 MB download, and I don't have time to wait for 30 MB to download.

If your problem is just a download issue, I can send you a CD with the demo version of Recode. Just PM me you address.

[Inventive Software]

BTW in the US we call "Spooks" "MI-5".


A bit confusing, but that's the American way I suppose. ;)

[Inventive Software]

Results are coming in, it just takes a while to update them.

[Inventive Software]

Right, awards time! The tests aren't quite finished yet, but this is my indication so far, and it may change with future results, though it's unlikely. :devil:


Outright Fastest Codec: Microsoft MPEG-4 V2
Outright Slowest Codec: On2 VP6 6.2.6 / Microsoft WMV9 VCM (On2 VP6 6.2.6 is faster at the higher bitrates but WMV9 is shockingly slow on the 2nd Pass, when it works)

Fastest MPEG-4 SP/ASP Codec: 3ivx D4 4.5.1
Slowest MPEG-4 SP/ASP Codec: DivX Fusion 1528

Fastest Proprietary Codec: On2 VP7 7.0.6 (at least when it works)
Slowest Proprietary Codec: On2 VP6 6.2.6 / Microsoft WMV9 VCM

Best Codec: Can't make a decision now, as I need more results
Worst Codec: Microsoft WMV9 VCM (it's terrible speed doesn't justify the average quality)
Best Newcomer: x264 rev 270 (needs some speed increases, but the quality was outstanding considering it's youth)

If you can't accept these, say so! This is only my opinion and like I said, it may change.