SSIM: H.261 vs. Theora +MPEG2, H.263 @CIF, 24fps, 300kbps

[Shapierian]

EDIT: It seems that ffmpeg's H.261 encoder is very broken for low bitrates. I've switched to 261cap and have new results below (http://forum.doom9.org/showthread.php?p=1096342#post1096342). The H.261 results seen in this post are not to be considered representative. Apologies to all.




Introduction:

I've done some SSIM comparisons of H.261, MPEG-2, H.263 and Theora as in light of claims raised by Stephen Wenger of Nokia [1]. The standards I wanted to compare were H.261 and Theora. But I also threw in MPEG-2 and H.263 to try to see how far apart H.261 and Theora were.

Source Material:

The Source Material used was Elephant's Dream [2] downscaled from 640x360 to CIF. (I wanted to run the frame at 640x360 but H.261 only supports CIF and QCIF sized frames so the input stream was rescaled to CIF.)

Encoders:

The only H.261 encoders I was able to find were 261cap [3] and ffmpeg/libavcodec [4]. 261cap did not give me enough bitrate freedom to actually use it. I also used ffmpeg for MPEG-2, H.263, and theora via it's libtheora wrapper. Since It has been claimed that the libtheora wrapper of ffmpeg does not give the theora encoder enough freedom/proper tunings I also tried ffmpeg2theora [5] which is an encoder designed by gluing an ffmpeg front end to a libtheora backend. I also used TMPGEnc [6] as a second MPEG-2 encoder.

Decoding:
All output was decoded to YUV via ffmpeg.

Comparison:
SSIM was calculated via the MSU Video Quality Measurement Tool [7].

Encoding Details:

A target streaming bitrate 300kbps was chosen as a sample low bandwith streaming bitrate. I was willing to lie to teh encoders to get it closer to the target than they generted with a request of 300kpbs.

FFmpeg version SVN-r11143, Copyright (c) 2000-2007 Fabrice Bellard, et al.
configuration: --enable-gpl --enable-pp --enable-swscaler --enable-pthreads --enable-liba52 --enable-avisynth --enable-libamr-nb --enable-libamr-wb --enable-libfaac --enable-libfaad --enable-libgsm --enable-libmp3lame --enable-libnut --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxvid --cpu=i686 --enable-memalign-hack --extra-ldflags=-static
libavutil version: 49.5.0
libavcodec version: 51.48.0
libavformat version: 52.1.0
built on Dec 3 2007 17:42:57, gcc: 4.2.2 (mingw32)

TMPGEnc-2.524.63.181-Plus-EN-Installer-DL.exe

$ cat ed-288-png.avs
ImageReader("D:\videotools\ed-360-png\media.xiph.org\ED\ED-360-png\%05d.png",1001,2000, fps = 24).BicubicResize(352, 288)

./ffmpeg -i ed-288-png.avi -vcodec mpeg2video -vb 290k -flags trell -me_method full -mbd rd -g 100 ed-288-.m2v -pass 1 -passlogfile mylog
./ffmpeg -i ed-288-png.avi -vcodec mpeg2video -vb 290k -flags trell -me_method full -mbd rd -g 100 ed-288-.m2v -pass 2 -passlogfile mylog

./ffmpeg -i ed-288-png.avi -vcodec h261 -vb 285k -flags trell -me_method full -mbd rd -g 100 ed-288-.h261.mov -pass 1 -passlogfile mylog
./ffmpeg -i ed-288-png.avi -vcodec h261 -vb 285k -flags trell -me_method full -mbd rd -g 100 ed-288-.h261.mov -pass 2 -passlogfile mylog

./ffmpeg -i ed-288-png.avi -vcodec h263 -vb 285k -flags trell -me_method full -mbd rd -g 100 ed-288-.h263.mov -pass 1 -passlogfile mylog
./ffmpeg -i ed-288-png.avi -vcodec h263 -vb 285k -flags trell -me_method full -mbd rd -g 100 ed-288-.h263.mov -pass 2 -passlogfile mylog

./ffmpeg2theora-0.20.exe ed-288-png.avi -V 330 --optimize -o ed-288-.ogv

./ffmpeg -i ed-288-png.avi -vcodec libtheora -vb 325k -flags trell -me_method full -mbd rd -g 100 ed-288-.fflibtheora.ogg -pass 1 -passlogfile mylog
./ffmpeg -i ed-288-png.avi -vcodec libtheora -vb 325k -flags trell -me_method full -mbd rd -g 100 ed-288-.fflibtheora.ogg -pass 2 -passlogfile mylog

$ ls -l *.ogv *.mov *.m2v
1562970 Feb 1 20:25 ed-288-293k.tmpg.m2v
1567145 Feb 1 20:08 ed-288-300.9k.fflibtheora.ogv
1567223 Feb 1 19:51 ed-288-300.9k.h263.mov
1569589 Feb 1 19:45 ed-288-301.4k.h261.mov
1572710 Feb 1 19:48 ed-288-302.3k.m2v
1574034 Feb 1 19:55 ed-288-302k.ff2t.ogv

Files available at [8].

Results:

Theora (ff/libth) AVG SSIM: 0.96361
Theora (ff2th) AVG SSIM: 0.96002
H.263 (lavc) AVG SSIM: 0.95866
MPEG2 (TMPG) AVG SSIM: 0.95803
MPEG2 (lavc) AVG SSIM: 0.95271
H.261 (lavc) AVG SSIM: 0.60656

http://vorlon.case.edu/~ajc30/video/v001/withh261.png

http://vorlon.case.edu/~ajc30/video/v001/withouth261.png

Further thoughts:

Other encodes with other settings and other encoders are welcome. I'd like to give each of these formats a fair hearing. I'm especially interested in better H.261 encodes since it faired so poorly.

At this bitrate and framesize with these encoders I would disagree with: "While not competitive with today’s state of the art codecs, [H.261 is] in the author’s personal experience not that far in its performance from Ogg Theora."

References:

[1] http://www.w3.org/2007/08/video/positions/Nokia.pdf
[2] http://media.xiph.org/ED/ED-360-png
[3] http://www.angelfire.com/in/H261/h261capture.html
[4] http://www.ffmpeg.org
[5] http://v2v.cc/~j/ffmpeg2theora/
[6] http://www.tmpgenc.net
[7] http://compression.ru/video/quality_measure/video_measurement_tool_en.html
[8] http://vorlon.case.edu/~ajc30/video/v001/

[Dark Shikari]

Apparently he has his own definition of "not far from."

[Inventive Software]

H.261 is, by design, similar I think to MPEG-1, minus the B-frames, so it's not that surprising to see it outperformed by the other codecs, but to see it do that badly is, well, surprising.

[Dark Shikari]

H.261 is, by design, similar I think to MPEG-1, minus the B-frames, so it's not that surprising to see it outperformed by the other codecs, but to see it do that badly is, well, surprising.Its likely that its just choking on the bitrate--that its not built for bitrates that low.

[Inventive Software]

H.261 is a 1990 ITU-T video coding standard originally designed for transmission over ISDN lines on which data rates are multiples of 64 kbit/s. It is one member of the H.26x family of video coding standards in the domain of the ITU-T Video Coding Experts Group (VCEG). The coding algorithm was designed to be able to operate at video bit rates between 40 kbit/s and 2 Mbit/s. The standard supports two video frame sizes: CIF (352x288 luma with 176x144 chroma) and QCIF (176x144 with 88x72 chroma) using a 4:2:0 sampling scheme. It also has a backward-compatible trick for sending still picture graphics with 704x576 luma resolution and 352x288 chroma resolution (which was added in a later revision in 1993).

So it'll do max 2 Mbit, but designed for ISDN lines? If it's not doing well on 300 Kbit target material....

[Shapierian]

So it'll do max 2 Mbit, but designed for ISDN lines? If it's not doing well on 300 Kbit target material....

What my be going on here is that the ffmpeg's implementation may not scale that low though it is feasible within the general rules of H.261. Watching the video, each P-Frame only updates a small number of macroblocks on each frame, especially in high motion cases, making the video unwatchable. Since H.261 is supposed to scale down to 64kbit/sec (for QCIF), for CIF we should require a minimum of 256kbit/sec so we should still have a small but sizable buffer above the minimum bitrate. That said, H.261 was designed for teleconferencing where low motion is to be expected. I really need a different H.261 encoder to test this theory. Anyone know of any other encoders?

[MfA]

H.261 lacks good skip modes, it can't hope to compete with MPEG-4 and Theora in low bitrate coding with modern resolutions (including the ones for next gen handsets). The difference between H.261 and Theora is far greater than the difference between Theora and the state of the art. Maybe mr. Wenger's is still living in the 90s in the age of postage stamp video? (Where you are basically forced to code every MB every frame anyway if there's significant motion.)

There is no credible alternative to Theora for W3C, without Theora they might as well not bother ... non open standards are the only alternative and those they can not use.

[foxyshadis]

h.261 was designed to be encodable in real time on a 386 or sparcstation, what does that tell you about its power? It's the equivalent of comparing HE-AAC to G.726. There's no need to make excuses for something that outdated, it's just outlived its usefulness. Lots of pricey telecommunication packages still use it, for some strange reason, though many have moved to h.263 by now.

My guess is that the nokia guy may have evaluated Theora several years ago and decided it wasn't up to snuff, or he just meant to say h.263. The only way to find out is to email him.

[Shapierian]

What my be going on here is that the ffmpeg's implementation may not scale that low though it is feasible within the general rules of H.261. Watching the video, each P-Frame only updates a small number of macroblocks on each frame, especially in high motion cases, making the video unwatchable. Since H.261 is supposed to scale down to 64kbit/sec (for QCIF), for CIF we should require a minimum of 256kbit/sec so we should still have a small but sizable buffer above the minimum bitrate. That said, H.261 was designed for teleconferencing where low motion is to be expected. I really need a different H.261 encoder to test this theory. Anyone know of any other encoders?

It looks like this was just the case: I bumped the bitrate to 320kbps and used H.261cap instead of ffmpeg:

Encoder settings:

./ffmpeg -i ed-288-png.avi -vcodec libtheora -vb 347k -flags trell -me_method full -mbd rd -g 100 ed-288-.fflibtheora.ogg -pass 1 -passlogfile mylog
./ffmpeg -i ed-288-png.avi -vcodec libtheora -vb 347k -flags trell -me_method full -mbd rd -g 100 ed-288-.fflibtheora.ogg -pass 2 -passlogfile mylog

261cap q7

./ffmpeg -i ed-360-png.avi -vcodec h263 -vb 305k -flags trell -me_method full -mbd rd -g 100 ed-288-.h263.mov -pass 1 -passlogfile mylog
./ffmpeg -i ed-360-png.avi -vcodec h263 -vb 305k -flags trell -me_method full -mbd rd -g 100 ed-288-.h263.mov -pass 2 -passlogfile mylog

The new filesizes are:
1,670,772 ed-288-320.8.fflibtheora.ogg
152,064,000 ed-288-320.8.fflibtheora.ogg.yuv
1,670,681 ed-288-320.8k.h263.mov
152,064,000 ed-288-320.8k.h263.mov.yuv
1,668,347 ed-288-320k.261
152,064,000 ed-288-320k.261.yuv

The new avg SSIMs are:
Theora (ff/libth) AVG: 0.96518
H.263 (lavc) AVG: 0.96097
H.261 (261cap) AVG: 0.95082

The new SSIM graph is:
http://vorlon.case.edu/~ajc30/video/v002/320k.png

Still where Theora dominates H.261:
http://vorlon.case.edu/~ajc30/video/v002/fr265.png

Still where H.261 fares better:
http://vorlon.case.edu/~ajc30/video/v002/fr322.png

I'd still like to see how a well tuned H.261 encoder performs.

[Dark Shikari]

For comparison, an x264 encode.

SSIM: 0.9876623

Linkage (http://mirror05.x264.nl/Dark/force.php?file=./x264.mkv)

2.82 times as good as Theora
3.40 times as good as MPEG-2
3.16 times as good as H.263
3.99 times as good as H.261
(and, from a small test of mine, 2.21 times as good as Xvid, with SSIM 0.97293)

(Note this isn't 100% fair given I'm using RDRC and AQ and TESA on this, though given the OP's use of "full search," it is mildly justified... ;) )

[Shapierian]

For comparison, an x264 encode.

SSIM: 0.9876623

Linkage (http://mirror05.x264.nl/Dark/force.php?file=./x264.mkv)


Wow! That looks pretty transparent to me. Nice work x264 guys.

[Shapierian]

My guess is that the nokia guy may have evaluated Theora several years ago and decided it wasn't up to snuff, or he just meant to say h.263. The only way to find out is to email him.

It may be based on old data, but he certainly means H.261, his proposal was for the web to standardize on a video codec old enough for all relevant patents to have expired.

[IgorC]

For comparison, an x264 encode.
SSIM: 0.9876623
Linkage (http://mirror05.x264.nl/Dark/force.php?file=./x264.mkv)
....
(Note this isn't 100% fair given I'm using RDRC and AQ and TESA on this, though given the OP's use of "full search," it is mildly justified... ;) )
Seems like RDRC forces 3 bframes though quality per frame is very constant. Very good quality for current test conditions.

The conditions of this test (resolution) limits a lot potential of x264 with all new stuff. Too low resolution to see effect of AQ and other improvements. :confused:

What has happened with gop length? ;)

[Dark Shikari]

Seems like RDRC forces 3 bframes though quality per frame is very constant. Very good quality for current test conditions.Incorrect, I did that manually with no-b-adapt. RDRC doesn't decide frametypes (unfortunately!). Forcing 3 B-frames boosted OPSNR drastically (I think 1db or close to it).

And with GOP length--what is this "GOP" you speak of? We only need I-frames on scenecuts, obviously! ;)

[IgorC]

Yes, I speak about I key frame for scenecuts but there is 6 times when scene is changed and there is only 4 i frames in the stream. Is there something to do with key interval =2000?

[Dark Shikari]

Yes, I speak about I key frame for scenecuts but there is 6 times when scene is changed and there is only 4 i frames in the stream. Is there something to do with key interval =2000?Dunno. Probably.

I just didn't want there to be I-frames at non-scenecuts.

[IgorC]

Or maybe RDRC has problem with new scene detection.

p.s. sorry, I forgot that rdrc doesn't inlfuence on frame type.


I think it's Non adaptive b-frames. Encoder just put 3 bframes each time wihout worrying of scene change.

[Phate]

Hi!
I'm doing my thesis on Open Source codec algorithms, and I was wondering if I could get some help from you.

I need to make a couple of video comparison between theora and h264, specifically for the web.

The videos are the followings:
http://people.xiph.org/~greg/video/ytcompare/comparison.html
http://people.xiph.org/~maikmerten/youtube/

I know, they use youtube, which does not make the best h264 video, and it's possibly biased since they come from people at xiph, but it serves for the purpose of my thesis, which reflects online video in specific.

I need an SSIM comparison of the videos in SD (first and second link) and HD (720p, only the second link). I see you guys used the MSU Video Quality Measurement Tool, but I don't have any access to a Windows machine, and I'm having trouble compiling the java/c++ implementations on Zhou Wang's website:

http://www.ece.uwaterloo.ca/~z70wang/research/ssim/

The original source file is in matlab, but that only analyses freeze frames and at the moment I can't use matlab. :(

Please, help! >_<

[nm]

I know, they use youtube, which does not make the best h264 video, and it's possibly biased since they come from people at xiph, but it serves for the purpose of my thesis, which reflects online video in specific.
How about examining x264 as an open source encoder in addition to Theora, or are you only interested in royalty-free formats?

I need an SSIM comparison of the videos in SD (first and second link) and HD (720p, only the second link). I see you guys used the MSU Video Quality Measurement Tool, but I don't have any access to a Windows machine, and I'm having trouble compiling the java/c++ implementations on Zhou Wang's website:
You could still grab Windows 7 rc1 for a free one-year trial and run it with VirtualBox or VMWare Server/Player.

Alternatively, use AviSynth's SSIM filter, for example by running avs2yuv with Wine.

The original source file is in matlab, but that only analyses freeze frames and at the moment I can't use matlab. :(
Do you mean you don't know how or you don't have access to it? Looking at the script, I'd say there is a good chance that Octave runs it just fine. Still, you need to know how to handle raw YUV video to feed it properly.

[Dark Shikari]

I know, they use youtube, which does not make the best h264 videoIt's not just "not the best", it's worse than H.263 in many cases.and it's possibly biased since they come from people at xiph, but it serves for the purpose of my thesis, which reflects online video in specific.Really? Taking a sample video from a phenomenally inefficient encoder used only by a company bleeding tens or even hundreds of millions of dollars a year "reflects online video"?

[Phate]

How about examining x264 as an open source encoder in addition to Theora, or are you only interested in royalty-free formats?

Actually, I started off with x264 as OSS alternative to the de facto standard H264, only then I added theora to the picture, mostly due to the HTML5 controversy.


You could still grab Windows 7 rc1 for a free one-year trial and run it with VirtualBox or VMWare Server/Player.

Alternatively, use AviSynth's SSIM filter, for example by running avs2yuv with Wine.


Thank you for the tip, I think I'll go with wine and AviSynth's SSIM filter. I hope it works, the release on my darwinports is 1.0.1.


Do you mean you don't know how or you don't have access to it? Looking at the script, I'd say there is a good chance that Octave runs it just fine.

Still, you need to know how to handle raw YUV video to feed it properly.
OK, what would be the correct iter to follow in order to test x264 and theora videos on my mac, using wine, AviSynth's SSIM, ffmpeg and/or mencoder?

[Dark Shikari]

Actually, I started off with x264 as OSS alternative to the de facto standard H264, only then I added theora to the picture, mostly due to the HTML5 controversy.What? This doesn't even make sense. x264 is an "alternative" to H.264 like a Civic is an alternative to cars.

[Phate]

Really? Taking a sample video from a phenomenally inefficient encoder used only by a company bleeding tens or even hundreds of millions of dollars a year "reflects online video"?

YouTube is losing less money that they though (http://latimesblogs.latimes.com/entertainmentnewsbuzz/2009/06/report-youtube-losing-less-money-than-thought-and-happy-that-hollywood-doesnt-know-it.html) it would, plus it has a market share of ~73% (http://www.readwriteweb.com/archives/youtube_dominates_video.php) and was awarded as the best HD video service on the Web (http://news.cnet.com/8301-17939_109-10150445-2.html). Yes, it does reflect online video.

In any case, I decided to test both those h264 videos made by youtube and local h264 compiled on my machine using HandbrackCLI/mencoder, to get more significant results and comparisons.

[Phate]

What? This doesn't even make sense. x264 is an "alternative" to H.264 like a Civic is an alternative to cars.

OK, let me rephrase then.
x264 is a free library for encoding H264/AVC video streams.

Anything more I can say without you being angry at me?

[Dark Shikari]

YouTube is losing less money that they though (http://latimesblogs.latimes.com/entertainmentnewsbuzz/2009/06/report-youtube-losing-less-money-than-thought-and-happy-that-hollywood-doesnt-know-it.html) it wouldthe site's losses this year will be a much more narrow $174.2 million.174 million dollars is a narrow loss? Is this a joke?, plus it has a market share of ~73% (http://www.readwriteweb.com/archives/youtube_dominates_video.php)No it doesn't, because those numbers completely omit Facebook, which probably has more uploads than Youtube does, albeit is probably still second in views. and was awarded as the best HD video service on the Web (http://news.cnet.com/8301-17939_109-10150445-2.html). Yes, it does reflect online video.And yet you contradict yourself again--that test didn't use the crappy "HQ" H.264 encoding mode; it used the "HD" encoding mode, which uses x264.

And further biased against Facebook, it used the low quality Facebook video encoding mode, despite the fact that every page has a gigantic "HD" link on it!

Facebook's maximum video bitrate is higher than that of Youtube's, though as far as I know they both use similar x264 encoding settings for their "HD" mode, so there should generally be little difference. Youtube tends to be more willing to throw its maximum bitrate at videos that don't need it, however.In any case, I decided to test both those h264 videos made by youtube and local h264 compiled on my machine using HandbrackCLI/mencoder, to get more significant results and comparisons.Use x264 directly, and use appropriate settings.

If you want to maximize SSIM and don't care about encoding speed, use --preset placebo --tune ssim. You should probably also use --ssim; it's a good idea to make sure that the SSIM from whatever plugin you're using matches x264's measurement, to ensure that you have no error due to mismeasurement.Anything more I can say without you being angry at me?I'm not angry; I'm just trying to help clear up misconceptions.

[Phate]

174 million dollars is a narrow loss? Is this a joke?


Nope. It was expected to lose half a billion dollars, which it didn't. Furthermore, it has been stated several times that the value of youtube for Google is not a mere commercial value, and certainly is not a short-term project.

Also, I don't see how relevant this would be with regards to who has the biggest market share.

And further biased against Facebook, it used the low quality Facebook video encoding mode, despite the fact that every page has a gigantic "HD" link on it!

Who? Are you referring to the comparative study on cnet? No, it doesn't.

From the article:

HD video on Facebook is gorgeous when viewed in the service's built-in player, but it does not scale well at all when viewed in full screen.


Or, were you referring to something else?


No it doesn't, because those numbers completely omit Facebook, which probably has more uploads than Youtube does, albeit is probably still second in views.And yet you contradict yourself again--that test didn't use the crappy "HQ" H.264 encoding mode; it used the "HD" encoding mode, which uses x264.


I will test both the HQ and HD encoding.


Use x264 directly, and use appropriate settings.


OK.


If you want to maximize SSIM and don't care about encoding speed, use --preset placebo --tune ssim. You should probably also use --ssim; it's a good idea to make sure that the SSIM from whatever plugin you're using matches x264's measurement, to ensure that you have no error due to mismeasurement.


OK, I will, thanks :)


I'm not angry; I'm just trying to help clear up misconceptions.

OK, so should I call x264 "a free library for encoding H264/AVC video streams" every time a refer to it? Isn't there a shorter name I can give it to indicate that I want to use an Open Source encoder instead of MainConcept or other commercial H264 implementations? Doesn't that make it an alternative solution to the problem of encoding H264/AVC video streams? :confused:

[nurbs]

You saidI started off with x264 as OSS alternative to the de facto standard H264
That sounds like you think x264 doesn't produce streams conforming to the H.264 standard. This is what Dark Shikari wanted to correct. You just call x264 that and the other implementations whatever they are named.
It's the same as with DivX and XviD. Both are encoders that implement the MPEG-4 ASP standard, none of them provides an alternative to the standard.

[Phate]

You said
That sounds like you think x264 doesn't produce streams conforming to the H.264 standard. This is what Dark Shikari wanted to correct. You just call x264 that and the other implementations whatever they are named.
It's the same as with DivX and XviD. Both are encoders that implement the MPEG-4 ASP standard, none of them provides an alternative to the standard.

OK, then I was misunderstood.
In the thesis I go into details in explaining how various implementations of the MPEG-4 Part 10 exist, each of which implements some of the specifics. As of July 2009, x264 implements more features than any other H.264 encoder (http://en.wikipedia.org/wiki/H.264#Software_encoder_feature_comparison). :)

[Dark Shikari]

OK, then I was misunderstood.
In the thesis I go into details in explaining how various implementations of the MPEG-4 Part 10 exist, each of which implements some of the specifics. As of July 2009, x264 implements more features than any other H.264 encoder (http://en.wikipedia.org/wiki/H.264#Software_encoder_feature_comparison). :)Of course, it depends how you define "features".

You could probably define "features" in a dozen different ways in order to make whatever encoder you wanted win ;)

(Some notable things x264 doesn't support: Explicit weighted prediction, PAFF/field coding, adaptive MBAFF, Periodic Intra Refresh, SVC, MVC, Extended Profile, Things In Baseline That Aren't In Main, 4:0:0 [because it's useless], 4:2:2, 4:4:4, constant frame-size, MMCO, multi-level b-pyramid, long-term references, HRD, open-GOP, sub-8x8 B-partitions, direct-8x8=0 [because it's useless, I removed support for it], adaptive deblocking filter strength, constrained intra prediction, multiple slices, and a few dozen more I can't think of off the top of my head)

[benwaggoner]

I did a couple of test runs at this you're welcome to look at:

Here's some encodes comparing Greg's samples with x264 and VC-1. I'm sure Dark Shikari could squeeze yet more quality out of these, but we can clearly see x264 outperforms Theora in per-pixel quality even at a much larger frame size:
http://cid-bee3c9ac9541c85b.skydrive.live.com/browse.aspx/.Public/BBB%7C_Compare

And these samples compare 1-pass CBR encoding where GOP=buffer duration and with roughly the same encoding time, in order to match Theora's rate control limitations. Obviously a 2-pass VBR encoder would do a lot better, but since Theora doesn't have one this should be more of an apples-to-apples of what Theora is capable of versus other encoders (although it's probably only as far as apples-to-pears here, compared to the apples-to-eggplants above):
http://cid-bee3c9ac9541c85b.skydrive.live.com/browse.aspx/.Public/Island%20Compare

Obvoiusly these are only snapshots of where x264 and Theora were at the time; they were all encoded with the current daily builds as of their file creation timestamp. The BBB VC-1 samples were done with the VC-1 Encoder SDK build used in EEv3, and The Island with Win 7 RC.

And I actually have permission from DreamWorks to distribute encodes of that trailer for codec testing purposes, so you can take screen shots from it etcetera.

[Dark Shikari]

I did a couple of test runs at this you're welcome to look at:

Here's some encodes comparing Greg's samples with x264 and VC-1. I'm sure Dark Shikari could squeeze yet more quality out of these, but we can clearly see x264 outperforms Theora in per-pixel quality even at a much larger frame size:
http://cid-bee3c9ac9541c85b.skydrive.live.com/browse.aspx/.Public/BBB%7C_Compare

And these samples compare 1-pass CBR encoding where GOP=buffer duration and with roughly the same encoding time, in order to match Theora's rate control limitations. Obviously a 2-pass VBR encoder would do a lot better, but since Theora doesn't have one this should be more of an apples-to-apples of what Theora is capable of versus other encoders (although it's probably only as far as apples-to-pears here, compared to the apples-to-eggplants above):
http://cid-bee3c9ac9541c85b.skydrive.live.com/browse.aspx/.Public/Island%20Compare

Obvoiusly these are only snapshots of where x264 and Theora were at the time; they were all encoded with the current daily builds as of their file creation timestamp. The BBB VC-1 samples were done with the VC-1 Encoder SDK build used in EEv3, and The Island with Win 7 RC.

And I actually have permission from DreamWorks to distribute encodes of that trailer for codec testing purposes, so you can take screen shots from it etcetera.Theora already has 2-pass ratecontrol; it was added a few days ago.

[benwaggoner]

Theora already has 2-pass ratecontrol; it was added a few days ago.
Yes, but not when I did those tests. I'll update when I've got some headspace.

...October at this rate...