Codec Shootout 2007?

[zambelli]

I just noticed Doom9 will be gone during the holidays. Does that mean this year's codec shootout will have to wait until January or February?

[vlada]

Anyway I think the time has come to start a new discussion what sources and encoders should be included in next year's comparison. What are your ideas?

I think one of samples might be the movie "Elephant's Dream". It won't be a DVD backup but an absolutely clear HD video source. Or do we still have to concentrate on DVD backups?

As for the used codecs/encoders - is there anything new regarding Snow and Dirac, which would make it worth trying them again? On the other side I heard quite a lot of work has been done on Theora. Of course VC-1, XviD AVC, x264, Ateme and MainConcept h.264 shouldn't be missing. I'm not sure if it makes sense to test XviD ASP and DivX again.

Any other suggestions?

[shon3i]

My opinions

DivX
Ateme ASP
Ateme AVC HP
XviD ASP
XviD AVC HP (if possible)
x264 HP
Mainconcept HP or Elecard HP (since use same encoder)
WMV VC-1

Same films like usual, maybe will be more interesting if will be compressed with full anamorphic resolution (720x576) with target size @ 700mb

All codec in mp4 container except WMV, for all 5.1 audio around 112kbs He-aac, for wmv aslo 5.1 audio.

@zambelli, it is Codec Shootout 2006 not 2007, that will be next year ;)

EDIT: i don't saw vlada's post but we should have some HD test beside DVD backup.

[Inventive Software]

@vlada: Grabbing the lossless source of Elephants Dream will be a challenge, as I think it's about 5 to 6 GB, maybe more! :eek:

I think having different types of codecs will help. When I do mine next year, I'll have a variety of ASP, AVC, and Proprietary codecs, then the winners from each can have a crack at Elephants Dream. :D

[Sirber]

isn't XviD AVC just vaporware?

[shon3i]

isn't XviD AVC just vaporware?
Is what??, sorry but i don't know what this word means.

It will be nice to see what devs or XviD work last year.

[buzzqw]

vaporware: (also called vapourware) is software or hardware which is announced by a developer well in advance of release, but which then fails to emerge, either with or without a protracted development cycle. The term implies deception, or at least a negligent degree of optimism; that is, it implies that the announcer knows that product development is in too early a stage to support responsible statements about its completion date, feature set, or even feasibility.

en.wikipedia.org/wiki/Vaporware

i think too is vaporware

BHH

[shon3i]

i think too is vaporware

Well will see next year what is happend, aslo we have half of month maybe XviD devs suprise us with new version, like last year

[Doobie]

The Codec Shootout doesn't seem very important anymore. Most movies fit nicely on a single CD, using any of several common codecs. But, CDs themselves are obsolete, given the price of DVDs.

[mahsah]

I think codec shootouts are worthwhile... heres what codecs I think:
(from shon3i)
DivX
Ateme ASP
Ateme AVC HP
XviD ASP
XviD AVC HP (if possible)
x264 HP
Mainconcept HP or Elecard HP (since use same encoder)
WMV VC-1

And I, personally, would like to see Theora, VP7, and a Wavelet codec (either snow or dirac).

[check]

Unless the shootout was moving up to HD resolution, there would be little advantage in testing all the old ASP files again. I'd be a lot more interested in a DVD5 size movie with a full bitrate AC3 track (384 or 448) for the HD three, ie AVC, VC1 and MPEG2. This gives you just over 4.5mbits, which is quite a bit of bitrate, a a longish movie would be needed, and of course a nice HD source will need to be found.

[Sirber]

+1 for HD.

About quality but also decoding speed :)

[Chainmax]

I think a much more useful codec shootout would be an MPEG2 one with all eight combinations between low motion--high motion, low bitrate--high bitrate, progressive--interlaced content. IMO DVD creation/authoring is still more prevalent than HD encoding.

[ChronoCross]

I think the codec shootout should focus on two divisions this year rather than 1. In the previous year it was a combination of the two. It should also only use codecs available to the public

H264/VC-1 (New Age codecs)
x264 (High Profile)
Nero (High Profile)
VC-1

ASP/Similar codecs
Xvid 1.1.2
Divx 6 Latest
etc
etc

[zambelli]

I think one of samples might be the movie "Elephant's Dream". It won't be a DVD backup but an absolutely clear HD video source. Or do we still have to concentrate on DVD backups?
I absolutely agree that the focus should move to SD and HD resolutions. Note that previous codec shootouts didn't even go up to SD. The Matrix encodes were done at like 640x272. I think one 480p/576p source and one 1080p source would be sufficient for each round of testing.

All codec in mp4 container except WMV, for all 5.1 audio around 112kbs He-aac, for wmv aslo 5.1 audio.
There's no point in adding audio soundtracks - the Doom9 shootout has always been a video codec test. They'd just take up space.

it is Codec Shootout 2006 not 2007, that will be next year
Next month IS next year. ;)

About quality but also decoding speed
Agreed. If it's a "codec" comparison, both encoders and decoders should be compared.

[shon3i]

Next month IS next year. Yes, but this is Codec Shootout 2006 not 2007, lastest been 2005, so this is 2006

There's no point in adding audio soundtracks - the Doom9 shootout has always been a video codec test. They'd just take up space.
Have point, only for simulation of encoding in real conditions.

I absolutely agree that the focus should move to SD and HD resolutions. Note that previous codec shootouts didn't even go up to SD. The Matrix encodes were done at like 640x272. I think one 480p/576p source and one 1080p source would be sufficient for each round of testing.
I Agree

[Inventive Software]

Doom9 has always stated previously that the codec tests were a test of backing up a DVD movie, audio included. ;)

[Morte66]

I'd be very interested in another Doom9 shootout, in case there are any surprises ("hey, VC1 is looking pretty decent these days and it's easier to decode than h264" or whatever). It saves me a lot of testing. Yes, I'm selfish... ;)

As for material... There's so much that could be tested, probably more than Doom9 could handle and remain sane. But I'd like to see any of the following:

- Anamorphic PAL 720x576 DVD backup at, oh, 1500kbps. A typical fairly high-ish quality backup to put multiple files on DVD5 or media server.

- I think "CD sized movie backup" really means "for filesharing" these days. But Some sort of iPod/PSP encode from DVD would be a meaningful reduced-resolution test.

- Reference 1920x1080p encode from clean source, e.g. Elephant's Dream or the Taurus Media samples. And again from film with grain, e.g the Swedish 65mm samples. This is mostly of academic interest.

- Denoise and encode typical (i.e. poor) MPEG2 HDTV stream movie at about 6GB/hour to DVD5. This is a real application, of growing interest.

[shon3i]

- I think "CD sized movie backup" really means "for filesharing" these days. But Some sort of iPod/PSP encode from DVD would be a meaningful reduced-resolution test.But aslo with Anamorphic PAL 720x576 or Anamorphic NTSC 720x480 resolution

[vlada]

I think that anamorphic resolution shouldn't be used at all. It should have never been introduced.

Why are you worrying that much about resolution? It doesn't make any change whether the movie is 640x272 or 1920x1080. There shouldn't be any difference for today's encoders AFAIK.

[weaver4]

I'd be very interested in another Doom9 shootout, in case there are any surprises ("hey, VC1 is looking pretty decent these days and it's easier to decode than h264" or whatever). It saves me a lot of testing. Yes, I'm selfish... ;)

What tool do you use that makes it "easier"? AutoMKV is pretty easy?

[shevegen]

"But, CDs themselves are obsolete, given the price of DVDs."

DVDS are obsolete, given the price of external HDDs.

[SeeMoreDigital]

I think that anamorphic resolution shouldn't be used at all. It should have never been introduced.Well I've always been of the opinion that it should..... As it provides the most effective way of more accurately comparing the source to the encode.... By all means crop away the black mattes though ;)


Cheers

[Doobie]

DVDS are obsolete, given the price of external HDDs.

If DVDs are obsolete, that strengthens my argument that a new Codec Shootout is of much less importance now than in years past.

But, DVDs are far from obsolete, given the price of external HDDs. DVDs are 20 times cheaper per GB. And, DVD disks also don't crash, taking your whole collection with them.

[zambelli]

I think that anamorphic resolution shouldn't be used at all. It should have never been introduced.
Anamorphic is irrelevant in a codec comparison because non-square pixel playback is a property of the renderer, not the codec (and certainly not the encoder).

It doesn't matter whether you use 720x480 4:3 or 16:9 or square pixel - the encoded video will be the same resolution and compared to the source, which is what matters.

Why are you worrying that much about resolution? It doesn't make any change whether the movie is 640x272 or 1920x1080. There shouldn't be any difference for today's encoders AFAIK.
It does. The length of motion vectors is important in video encoding. We can't just assume that it scales correctly in all codecs.

[check]

oh yes, there's plenty of codec performanc differences between SD and HD encoding. Different codec options are used to different degrees for one thing.
As for anamorphic not being used... well, I think you're crazy, but in practical terms it would be better to have the most complex video test possible to test today's encoders.

[vlada]

zambelli
I know how anamorphic movies work and that it doesn't have anything to do with encoding/decoding. I just think that the introduction of 720x*** resolution was the 2nd worst idea after interlacing in the last 100 years :-) I spent many hours trying to explain that you don't have to letterbox video to get 16:9 picture. I think that 50% of people will never understand how is it possible. It only causes confusion IMO.

According to SD/HD - if you think there might be a difference, I say, let's try it, why not. But I think that it is much more important to define the sources. The possibilities are:
1) A digitally created clean source
2) An uncompressed HD movie (but I don't think anyone could get it)
3) Blu-ray/HD DVD
4) HDTV broadcasting
5) HDV camcorder

[zambelli]

zambelli
I know how anamorphic movies work and that it doesn't have anything to do with encoding/decoding. I just think that the introduction of 720x*** resolution was the 2nd worst idea after interlacing in the last 100 years :-) I spent many hours trying to explain that you don't have to letterbox video to get 16:9 picture. I think that 50% of people will never understand how is it possible. It only causes confusion IMO.
It's not the worst idea. Nobody just pulled 720 out of thin air. The digital representation of a standard def analog video signal comes out to about ~702 horizontal pixels. In the digital world, that's already anamorphic regardless of whether you're representing 4:3 or 16:9. The DVD standard simply tried to stay as compatible as possible with the existing technologies.

[Doobie]

It's not the worst idea. Nobody just pulled 720 out of thin air. The digital representation of a standard def analog video signal comes out to about ~702 horizontal pixels.

Television resolution resolution isn't even near 702x.

[check]

whether or not a TV can display 720 pixels is irrelevant to this topic.
Her's my proposal:
o test 720p encoding of a 'standard' movie (ie, one that follows the previous years' picks), from the best available source with the following codecs:
o x264
o apple264
o elcard
o MS vc1
o MPEG2 (which encoder?)
o xvid
Codecs to be rated on maximum quality, but decoding speed to also be measured. Decoding speed should not be a judging criteria because it is a non issue for practically all new computers available today and will become even less important as time goes on.
Judging of visual quality should rely purely on subjective means, objective tests not being able to provide an objective measure in line with human perception. The codecs should be compared relative to one another rather than to an absolute baseline (the term is Q-sort if i recall correctly), and the scoring should be recorded for both a small group of selected viewers (forum mods, or similar) and for the population at large (obviously samples will have to be made available if this is the case).

[Morte66]

What tool do you use that makes it "easier"? AutoMKV is pretty easy?

Easier (i.e. requires less CPU) to decode, not encode.

Now that I make some h264 HD encodes, I can only play them on my A64 X2 3800+ Athlon because I bought a special codec (CoreAVC Pro). They may not play on other machines (e.g. that convection cooled silent HTPC I thought about). There are plenty of other formats that are easier to decode, but AFAIK nothing else comes close to the quality/bitrate of x264 encoding.

It's nice to read a codec shootout once per year, in case things like this have changed while I wasn't looking.

[vlada]

The DVD standard simply tried to stay as compatible as possible with the existing technologies.

Backward compatibility is the biggest problem of all invention. But you have to break it once if want to move further. But this is off-topic here.

I also think it would be interesting to add an MPEG-2 encoder. For example the company Digigami (http://www.digigami.com/megapeg/) claims, that their MPEG-2 encoder is better then h.264 codecs.

[shon3i]

I also think it would be interesting to add an MPEG-2 encoder. For example the company Digigami claims, that their MPEG-2 encoder is better then h.264 codecs.Oh please come on, another BS, from such this companies. This Mpeg2 encoder is probably out from all mpeg2 standards.

We should have standard DVD Backup and HD backup

[SeeMoreDigital]

I also think it would be interesting to add an MPEG-2 encoder.MPEG-2 is not in the same high-performance compression league as say, WMV, MPEG-4, RM, VP etc

In my opinion it would be better to have a separate "MPEG-2" shoot out ;)

[vlada]

In my test a video from miniDV camcorder (noisy source) looked very similar in MPEG-2 (ProCoder) and MPEG-4 (XviD) at 3 Mbps. But it was just an idea...

[shon3i]

In my test a video from miniDV camcorder (noisy source) looked very similar in MPEG-2 (ProCoder) and MPEG-4 (XviD) at 3 Mbps. But it was just an idea...
On 3mbs, every encoder will show amazing quality. So what is point on doom9 test, is to show what encoders can do in low-mid bitrates such 700-900kbps as usual cd1 rip. In that bitrate mpeg2 dont have any chance.

[SeeMoreDigital]

In my test a video from miniDV camcorder (noisy source) looked very similar in MPEG-2 (ProCoder) and MPEG-4 (XviD) at 3 Mbps. But it was just an idea...Let's not forget, miniDV captured sources are pure interlaced. So if your're intending to generate comparitive MPEG-2/MPEG-4 encodes, they'll have to be interlaced too... That is, unless you want to introduce software de-interlacers into the encoding food chain!


Cheers

[vlada]

shon3i
720x576@25 and 3 Mbps results in 0,3 bpp (bits per pixel). That isn't too much for an interlaced noisy source.

[shon3i]

For MPEG4 is, for MPEG2 maybe, but depends of duration. For 1-1.5 hours i good enought, for my taste

[zambelli]

Decoding speed should not be a judging criteria because it is a non issue for practically all new computers available today and will become even less important as time goes on.
Are you serious? Have you tried decoding H.264 CABAC-coded 720p/60 video? It's far from an easy task. I'd guess over 85% of computers in people's homes today couldn't decode that flawlessly even with CoreAVC.

[check]

Yes, plenty of computers around today will be unable to decode a HD 720p stream, anything slower than a 3ghz will have no hope. What I meant to say (and what I think I did! :-P) is that the majority of new computers released today will be able to play the videos -- in other words, decoding speed is still an important issue now, but probably not in 6 months.

More points:

o the codec comparison has traditionally been a look at the video quality, and nothing else. However, no other codec generation has been as CPU intensive as the current one. Is it better to change the goal of the comparison or or to keep it the same? Only doom9 knows :)

o on platforms other than windows, decoding performance is far more important (something I didn't consider before).

o the 'optimal' balance between CPU usage and video quality is a) subjective and b) highly variable between users. It is not neccessarily just (subjective quality score)/(mhz required), because that assumes both values are of equal importance.

This last point is very important. If decoding were to be included in the scoring, a point system would need to be worked out, and I can guarantee everyone's ideal system for decoding requirements would be { =< my_cpu_speed:10 , else 0} :rolleyes: woohoo made up pseudocode!.
For me, my A64 3000+ can handle most 720p h264, and that which it can't is being carefully filed away until I get the chance to buy my spanking new system hopefully before march. I would prefer pure quality numbers because I am already aware of the decoding requirements and am far more interested in pure output quality.

PS, added xvid in my above post

[vlada]

This thread has been very quiet for a long time. Does anybody know if a new codec comparison is in the works? I think this year it might be about Blu-ray (or HD DVD) to 720p DVD-5 (4,7 GB) ripping. It gives about 0,2 bpp, so it is difficult enough to make a comparison.

[Inventive Software]

I plan to do one in the summer (June / July / August time). Stay tuned. ;)

[shon3i]

Blu-ray (or HD DVD) to 720p DVD-5 (4,7 GB) rippingAnd 1080p to DVD-9. Anyway comparision of standard DVD backup to 1 CD for me will be more interesting than backuping HD, 3 AVC, 3 ASP encoders, 2 comercial and one opensource in each group. To see is opensource solutions still bst.

[Inventive Software]

My plan is something along these lines...

Test several codec's SD encoding performance and quality in both "high" and low bitrate scenarios with some DVDs; Spooks Series 1 to a DVD-5 and an undecided 2 hour film to 700 MB CD.

Test the HD encoding performance and quality of codecs with the lossless version of Elephants Dream. This may well be to a CD. The lossless SD version will be encoded to 1/4 CD.

Test decoder performance. This is key, since a movie encoded has to play on certain hardware. My test-bed for the SD decoding is my Celeron 800 MHz desktop, and my Dell Inspiron 1501 AMD64 X2 laptop if the Celeron can't handle it. The laptop will also handle the HD decoding.

Specs of the test are nowhere near finalised, but this is sort-of a basic rundown of what I plan to do. Open and closed codecs will be used, MPEG-4 ASP, MPEG-4 AVC, Propriety are the categories the codecs are under. VC-1 comes under Propriety in this case.

[zambelli]

Test the HD encoding performance and quality of codecs with the lossless version of Elephants Dream. This may well be to a CD. The lossless SD version will be encoded to 1/4 CD.
The Elephants Dream source is a fairly easy encode. Besides that one running sequence at the beginning, the rest of it is fairly simple. In lack of something better, it'll do, but it won't really challenge the encoders to the max.
Also, note that there are banding issues present in the PNG source itself. BenWaggoner pointed them out on AVS Forum (can't find the thread link atm). Not a deal breaker since obviously the same source would be fed to all codecs, but just another thing to consider - it's not a perfect source.

Test decoder performance. This is key, since a movie encoded has to play on certain hardware. My test-bed for the SD decoding is my Celeron 800 MHz desktop, and my Dell Inspiron 1501 AMD64 X2 laptop if the Celeron can't handle it. The laptop will also handle the HD decoding.
Agreed. I pointed out the importance of decoding after the last codec shootout and it's even more important now in the HD era. I'm assuming you will test decoders independently from encoders? For example, Nero's decoder won't necessarily be tested together with Nero's encoder, right?
Another thing to consider with decoders is whether you want to include app integrated decoders into the test. Not all decoders are system-wide DirectShow components.
DXVA decoding support should be another consideration.

VC-1 comes under Propriety in this case.
Ummm, why? It's no less of a standard than MPEG-4 is.

P.S. The adjective form of the word is "proprietary".

[Inventive Software]

There's a lack of VC-1 codecs, (currently only WMV9) hence why it's under Proprietary. I'll compare it to AVC though in the report. ;)

Thanks for the grammar tip BTW. :)

[Wilbert]

That is just plain wrong. You are trying to confuse codecs and standards. There is a difference between those two. What you should do, is consider three standards: MPEG-4 ASP, MPEG-4 AVC, VC-1. All of those three are open. Each standard has open source and proprietary encoders (well, VC-1 has no open source encoders, as you note).

I think you will agree that Nero's MPEG-4 encoder is as proprietary as WMV9.

[Doom9]

The Matrix HD DVD kit is currently making its way over here.. that's all I can say for now.
But since the lowest performing system in my posession is an E6700 C2D (and the better system a Q6700), decoding performance really is of no concern to me. If you get a HD capable player, you not only pay for the blue laser diode, but also the CPU horsepower to decode high def video.. why should it be any different for a PC?
And especially AVC decoding is highly decoder dependant.. if you go with a free single threaded variety (e.g. ffdshow), you'll need more CPU horsepower than if you go with a commercial multithreaded version (e.g. coreavc) - now which should be considered in the evaluation? If you're cheap, you're going to bitch about decoding performance because you're not willing to go beyond what's free, yet I don't think that this is particularly fair.. HD software players all have multithreaded decoding and I bet there's a few more multithreaded decoders out there these days.
For example, Nero's decoder won't necessarily be tested together with Nero's encoder, right?As far as AVC or VC-1 is concerned, this shouldn't be of any consequence since the postprocessing part is part of the standard (not like ASP where it's optional and hence you have different implementations). Baring any implementation errors, decoder X should yield the same results as decoder Y.

And since DVD still owns 99% of the market, it would be highly inappropriate if any comparison didn't reflect upon that and only went for HD.

[phædrus]

Has anyone done a good MPEG-2 encoder shootout recently? I've been playing around with HDTV2DVD, which uses ffmpeg encoder I think. It is fast, but seems to add a certain "texture" to the output. I wonder how it would actually perform in a controlled test against other options, though. I have searched on google for such a shootout, but I only found old tests.