The 2009 MSU H.264 encoder comparison: a complete crock (or not?)

[Dark Shikari]

It seems that MSU is interested in fixing this one; see this post (http://forum.doom9.org/showpost.php?p=1281774&postcount=45) for the latest information. Good on them for caring about the accuracy of their results. Note I made some mistakes in writing this post as well, partially due to the way in which the report was designed--see the previous link.

I recently received a draft copy of MSU's 5th H.264 encoder comparison (http://forum.doom9.org/showthread.php?p=1268192#post1268192) results. Overall, the tests themselves were well done and the raw data rather interesting. They had a number of interesting extra tests as well, including tests of motion estimation efficiency and a measure of quality in the "tail area" behind moving objects. Of course, there were many flaws in the report as well. But one particularly flaw is so serious that it compromises the accuracy of the entire comparison.

I emailed MSU immediately after receiving the draft with a full run-down of problems with the report. I gave MSU the benefit of the doubt, but it has been one full day since I contacted and I have received no response--not even an acknowledgment--despite having a dedicated contact there who provided immediate responses during the initial entry period. So I'm posting this here.

Optimizing for PSNR and SSIM at the same time is impossible: SSIM is effectively a weighted version of PSNR, so optimizing for SSIM inherently lowers PSNR. Moreso, optimizing for SSIM lowers PSNR a lot more than it raises SSIM (this is critical). So, I asked MSU which I should optimize the x264 settings for. Originally, when I asked MSU which metric they were planning to use (and thus which I should optimize for), they told me:

DS> In addition, what about the fact that encoders have different
DS> settings optimized for PSNR, SSIM, and visual quality,
DS> respectively? x264 can optimize for PSNR (AQ off, psy-RD off),
DS> SSIM (AQ on, psy-RD off), or visual quality (AQ on, psy-RD on).

We will use SSIM and recommend optimization for this measure for
comparison.

DS> It would be surely be unfair to ask for an encoder to try to
DS> optimize for all three with one set of settings--one would either
DS> end up with settings that are suboptimal for all measurements or
DS> settings that are extremely good in one but bad in the others.

Sure! Let's use SSIM, such approach allow to test many sequences
for several usecases as well.

So I followed their instructions and optimized for SSIM.

Recently, they sent me the draft results, which I looked through. The results were roughly as follows (source graphs (http://img201.imageshack.us/img201/5017/x264ssim.png), I think I got them all):

Clear win for x264: 11 (Wendys, Indiana Jones, State Enemy, Troy)
Slight win for x264: 4 (Battle, Troy)
Practically a tie: 6 (Mobile Calender, Stockholm, Troy)
Slight win for Mainconcept: 3 (Pedestrian Area, Mobile Calender)
Clear win for Mainconcept: 1 (Pedestrian Area)

But then I noticed they had a ton of PSNR graphs too... which unsurprisingly had x264 quite a bit lower, given that I had optimized for SSIM, not PSNR, as they told me to! And remember--optimizing for SSIM lowers PSNR much more than it raises SSIM. I also noticed that later on in the report that they seem to have completely forgot what I told them about a dozen times--that x264 this time is optimized for SSIM, not PSNR:

"There is also one interesting fact: if to use Y-PSNR as quality metric then x264 does not show the good progress as can be seen from Figure 244 or Figure 245. "

They wonder why x264 has gotten worse PSNR since 2007... despite being told a half-dozen times why. Does the left hand not know what the right hand is doing?

... and then the clincher. They declare Mainconcept the winner (purely in terms of encoding quality) because it won the PSNR tests.

Yes, that's right. Despite telling me they would use SSIM for the comparison, and that I should optimize for SSIM, they awarded victory to the encoder that lost most of the SSIM tests.

Hanlon's Razor (http://en.wikipedia.org/wiki/Hanlon%27s_razor) would suggest this is a problem of incompetence, but I simply cannot believe that the MSU are this stupid, so I'm tempted to go with malice here.

(Of course, if they are willing to fix the report, so much the better--but perhaps there needs to be a bit of embarrassment before they understand that their publications need to actually have a basis in reality.)

[Chengbin]

I just read through the whole post. This is a serious mistake they made. I hope they correct it.

[Audionut]

There seems to be very little ssim difference between the High quality and High speed presets.

edit: Could you post the command lines?

And personally, I probably would have given them more than 24hrs to respond. But your right, left hand isn't talking to right hand.

[Dark Shikari]

There seems to be very little ssim difference between the High quality and High speed presets.

edit: Could you post the command lines?One of the more minor flaws in the comparison (one which is rather hard to fix after-the-fact) is the fact that their speed targets were as a whole far too low, forcing me to be extremely bruteforce in choosing settings. That is, even the "high speed" settings used very slow encoding modes.

Here's the settings I have from my own notes (might be slightly off as we tweaked them a bit):

x264 r1114M

For all presets, do:
x264 -o outfile.264 infile.yuv $WIDTHx$HEIGHT --fps $FPS --bitrate $BITRATE --no-psnr --no-ssim --bframes 3 --b-pyramid --mixed-refs --keyint 500 --threads auto --psy-rd 0

additional options
HD 3fps:
--pass 1 --subme 7 --direct auto --b-adapt 2
--pass 2 --subme 10 --direct auto --ref 4 --8x8dct --me esa --trellis 2

HD 10fps:
--pass 1 --subme 1 --direct auto --b-adapt 1 --partitions none
--pass 2 --subme 7 --direct auto --ref 6 --8x8dct --me umh --trellis 1 --merange 24

SD 12fps:
--pass 1 --subme 7 --direct auto --b-adapt 2
--pass 2 --subme 11 --direct auto --ref 6 --8x8dct --me umh --trellis 2

SD 40fps:
--pass 1 --subme 1 --direct auto --b-adapt 1 --partitions none
--pass 2 --subme 7 --direct auto --ref 4 --8x8dct --me umh --trellis 1

CIF 50fps:
--subme 9 --ref 5 --aq-strength 1.25 --8x8dct --me umh --merange 12 --trellis 2 --b-adapt 0 --scenecut -1 --ratetol 10

CIF 100fps:
--subme 9 --ref 4 --aq-strength 1.25 --8x8dct --me umh --merange 8 --trellis 1 --b-adapt 0 --scenecut -1 --ratetol 10

"subme 10" is subme 9 combined with a 7x7 exhaustive qpel RD search. "subme 11" is subme 9 combined with an 11x11 exhaustive qpel RD search. All settings include a modified qpel search decision process which I may add to the regular x264 soon (it gained ~0.5-1% efficiency at a small speed cost).

[Chengbin]

subme 9 for CIF 100fps. WOW!

[poisondeathray]

Do you know if the other entrants submitted modified/optimized builds, and if they were give the same instructions?

How do you think a generic non-optimized x264 build would have done in this testing scenario?

[cogman]

Man, that is disappointing dark. Hopefully this gets corrected. They should have realized something was wrong when after a year nothing appeared to change in x264.

[Dark Shikari]

Do you know if the other entrants submitted modified/optimized builds, and if they were give the same instructions?I would assume that everyone involved submitted their latest internal builds rather than what was publicly available, but obviously I can't be sure. I asked explicitly whether I could submit a modified build and they said it was fine; the original purpose of this was to include Holger's optimizations, but those were committed in time for the final submission and so that ended up being unnecessary.How do you think a generic non-optimized x264 build would have done in this testing scenario?1-2% worse at most; the primary purpose of subme10/11 was to provide a slightly better speed/quality tradeoff at Absurdly Slow Speed Settings than --me tesa does.

[Audionut]

is the fact that their speed targets were as a whole far too low

I'm sure they've relied more on the speed of the encoder in previous comparisons. Which leads me to believe it's a biased comparison.

edit: SD 12fps. need I say more.

edit 2: A target ssim + bitrate showing resulting speed of each encoder would make sense to me.

[akupenguin]

A target ssim + bitrate showing resulting speed of each encoder would make sense to me.
That would work as long as the target ssim+bitrate is within the capability of all the encoders. Looking at the graphs, there is no point that all of the encoders can meet with some speedwise tradeoff.
Target speed+bitrate and measure ssim (as was supposed to be done), is the more flexible.
Target speed+ssim and measure bitrate is theoretically possible, but most codecs don't have a target ssim ratecontrol mode, so it would take many retries.
Most informative would be a target bitrate and measure lots of presets to graph a whole speed-vs-quality curve.

[benwaggoner]

Do you have access to the sources used? It would be interesting to rerun the test with PSNR-optimized settings.

That said, I don't think either PSNR or SSIM is that well suited to the LCD era, as artifacts in low luma are much more visible than in high luma. Perceptually uniform gamma is a lovely idea, but it died as a realistic assumption with the CRT.

I'm glad that plasma appears to be on the way out at least; it had different but also bad low luma issues, and it stank to have to sweat both.

What was their platform for perf testing this time around? A couple of years ago it was some ancient single-core box that simply wasn't a realistic optimization target anymore.

[LoRd_MuldeR]

...the primary purpose of subme10/11 was to provide a slightly better speed/quality tradeoff at Absurdly Slow Speed Settings than --me tesa does.

If I read this correctly, you are saying that "--subme 10/11" provides a better speed/quality trade-off than "--me tesa".

Since "--me tesa" is included in official x264, wouldn't it be logical to include "--subme 10/11" too?

[RunningSkittle]

... Since "--me tesa" is included in official x264, wouldn't it be logical to include "--subme 10/11" too?...
Answered already:
...All settings include a modified qpel search decision process which I may add to the regular x264 soon (it gained ~0.5-1% efficiency at a small speed cost)...

[Sagittaire]

@ benwaggoner

SSIM can use luma masking and other mask if you want ...


@ Dark Shikari

Well unfortunaly use particular setting to optimized each test for x264 is perhaps not good for egality if you don't make that with the other codec too (I have serious doubt about that). I think that by definition developper (not only you) are not objective to evaluate test procedure and result. Like always best solution IMO is tradeoff: soft AQ and soft psy (but real visual gain) with really good PSNR (less than 0.5 dB delta with best possible result). Dark Shikari know certainely x264 profil for produce that.

[ChronoCross]

This is not really shocking. Guess we know who will get a grant for research this year.

[IgorC]

I know I criticize a lot but speaking seriously:

MSU's comparisons had very serious mistakes of concept in past. And they did again. For example they compared Divx with postprocessing filter enabled to Xvid withouth PP. It sounded like a joke but it's hapened. A lot of graphs, explanations and other stuff are all useless when a rude mistake is done.

And, of course, people know about unreacheble (by any other commercial encoder) quality of x264 . Even more, ssim and psnr have nothing to do with outstanding x264's psy optimiziations.

[Dark Shikari]

Well unfortunaly use particular setting to optimized each test for x264 is perhaps not good for egality if you don't make that with the other codec too (I have serious doubt about that).They didn't allow particular settings per test--only per resolution and framerate. They also did not tell us which sources would be used beforehand.

Like always best solution IMO is tradeoff: soft AQ and soft psy (but real visual gain) with really good PSNR (less than 0.5 dB delta with best possible result). Dark Shikari know certainely x264 profil for produce that.Except that they specifically told me to optimize for SSIM, not PSNR. Yes, that's what I would have done if they had told me to use both, but I decided that I was going to, you know, follow their instructions? :rolleyes:

Also, "soft psy" doesn't exist in x264--psy-rd always decreases PSNR and SSIM at any strength.

[kosmonaut]

This is disturbing news. Codec comparisons are so hard to do in any semblance of a fair way, even if everybody is acting in good faith, which is now in question for the MSU test.

[CruNcher]

Indeed that seems to have been a perfect example of a internal communication failure (one hand doesn't know what the other want's), but not sure if it's that bad most know the problem between PSNR & SSIM relation so one that sees the results and looks @ them carefully is gonna realize the difference that x264 is actually leading @ the SSIM results while being behind @ PSNR :)
I guess we can mirror the Mainconcept Results 1:1 onto the last DivX Encoder, or did they used their AQ and Grain processing ?

[Sagittaire]

Also, "soft psy" doesn't exist in x264--psy-rd always decreases PSNR and SSIM at any strength.

well in my test really soft psy setting --psy-rd 0.50:0.0 --aq-strength 0.50 --aq-mode 1 produce similar result than Mainconcept SDK with best PSNR setting. Perhaps really small visual gain (I don't make comparison) but you have really good PSNR with AQ and psy.

[kolak]

.....
I guess we can mirror the Mainconcept Results 1:1 onto the last DivX Encoder, or did they used their AQ and Grain processing ?

The whole beauty with Mainconcept is in the AQ settings and grain optimisation, which does work very well in Cinevision.


Andrew

[benwaggoner]

It's not known what instructions other participants received, is it?

It seems like the right solution to this is to allow two sets of settings: PSNR and SSIM, and publish results for both.

They're different metrics, so it'd be interesting to compare and contrast. It'd be interesting to see the correlation between them (although the scales are quite different; it'd be hard to do it with much mathematical rigor).

I think people overestimate the "objectivity" of PSNR as it is. There's a whole lot of psychovisual assumptions already baked in, from the 8-bit scale with perceptually uniform gamma to using 4:2:0 chroma subsampling. A .yuv is already quite tuned to human vision before it ever hits a codec. It's probably look really weird to other species.

[Dark Shikari]

It's not known what instructions other participants received, is it?

It seems like the right solution to this is to allow two sets of settings: PSNR and SSIM, and publish results for both.That was one option I suggested in the email--that would actually make sense though, we can't be making sense here can we ;)

Also, it's been almost 3 days now; no response.

[video_magic]

Okay, as an amateur I have to wonder as often - do you guys ultimately rely on the human eye as a judge on "what is best" or do you want a judge on what is most pleasin to those who want their result to look most like (in their eyes) the original footage? (no smoothing etc)

Sorry, for being an amateur user - but sometimes I feel so left out.

[Dark Shikari]

Okay, as an amateur I have to wonder as often - do you guys ultimately rely on the human eye as a judge on "what is best" or do you want a judge on what is most pleasin to those who want their result to look most like (in their eyes) the original footage? (no smoothing etc)

Sorry, for being an amateur user - but sometimes I feel so left out.The job of an encoder is to match the properties of the original footage as best as possible. If someone wants to smooth it, they can preprocess the footage.

Also, this is not really the thread for this kind of question.

[video_magic]

I'm sorry,I am frustrated as to what looks the best (as in most accurate) to my eyes and in what looks the same at say 1800 KBPS at the original VHS capture - I am frustrated, and sorry.

[Dark Shikari]

And it's now been nearly a week with no response...

[Who Am I ?]

it ultimately comes down to this : either they are stupid or blind ( no offence to the blind folks here ) not to read emails or they're absolutely biased which seems to be the case here ..

i wont like to cuss main concept here, but this serves as a major example off how the corporate oriented money making software's ( and eventually the people judging them ) and machine's get biased against the *absolutely free* open source community ..

i really hope this gets sorted dark shikari , if there's a grant to be given on such reports , no one deserves it more then the x264 team .

[IgorC]

As far as I remeber Ateme's developers also had problems with MSU. MSU asked for money to Ateme "for not disclose the results where Ateme wasn't winner". It's fact, not opinion.

Bad, very bad.

[yesgrey]

I'm tempted to go with malice here.
Except that they specifically told me to optimize for SSIM, not PSNR. Yes, that's what I would have done if they had told me to use both, but I decided that I was going to, you know, follow their instructions?
They let you pick your poison, and you did...;)

Don't worry, we will continue to use x264 for the best (sorry for using the word:D) two reasons:
-It's better
-It's cheaper
:)

[Comatose]

But this test could sway big corporations into using x264 :\ It would be sad to see it lose when it really shouldn't.

[CruNcher]

As far as I remeber Ateme's developers also had problems with MSU. MSU asked for money to Ateme "for not disclose the results where Ateme wasn't winner". It's fact, not opinion.

Bad, very bad.

Sure that's not nice but from where came the request, i mean if someone decides to participate and the results don't are what he expected doing like the test never existed is a little questionable, of course responding "give us money and we do as these results never existed" is also questionable (moral point of view).
On the other side you shouldn't forget they also make money with every test result the more attractive (more participants) the more money they get back for their testing efforts with the full payable report so it's somehow from a business point of view clear why they asked Ateme for money for their request.Their testing service is for free but only as long as you don't have extra wishes and they can use the results in any way they seem appropriate to cover the costs for it :) you agree to this by participate (not sure if as a participating party you have to sign a service agreement currently)

[Dark Shikari]

But this test could sway big corporations into using x264 :\ It would be sad to see it lose when it really shouldn't.Perhaps. But the two biggest players (or close to it) in terms of amount of content encoded already use x264 (Youtube and Facebook).

[Sagittaire]

Perhaps. But the two biggest players (or close to it) in terms of amount of content encoded already use x264 (Youtube and Facebook).

Perhaps but nobody use x264 for produce bluray. Conclusion here can be exact opposite. Youtube, Facebook and warez use x264 not because it's the best but simply because it's free.

For the initial thread: IMO MSU should work with only one profil for x264 and IMO good profil for large majority of content in general case (high, medium, low bitrate, anime, grain ... etc) is profil with soft psy setting (--psy-rd 0.50:0.0 --aq-strength 0.50 --aq-mode 1).

[Dark Shikari]

Perhaps but nobody use x264 for produce bluray. Conclusion here can be exact opposite. Youtube, Facebook and warez use x264 not because it's the best but simply because it's free.Well, I doubt warez encoders care about cost... ;)

[onesloth]

Perhaps but nobody use x264 for produce bluray. Conclusion here can be exact opposite. Youtube, Facebook and warez use x264 not because it's the best but simply because it's free.

Seems most blurays are simply encoded at whatever bitrate fills the available space on the disc: resulting in encodes with hugely excessive bitrates for the source quality. At bitrates that high, it shouldn't really matter which of the popular encoders you use. Why wouldn't you use the free one?

[Who Am I ?]

Well, I doubt warez encoders care about cost... ;)

100 % right , why ? cuz they never pay for the software's they use in the first place ( afaik ) :rolleyes:

[Chengbin]

Perhaps but nobody use x264 for produce bluray. Conclusion here can be exact opposite. Youtube, Facebook and warez use x264 not because it's the best but simply because it's free.

I think YouTube uses x264 because it is a hell of a lot faster than the other H.264 encoders, and therefore saving them a lot of electrical power.

[Dark Shikari]

I think YouTube uses x264 because it is a hell of a lot faster than the other H.264 encoders, and therefore saving them a lot of electrical power.Not that either. Google apparently doesn't care due to the absurd size of their compute farm, which is why they used Skal's home-grown encoder for so long (and still do for fmt=18).

They use it because I told Skal that his encoder, giving barely better results than FLV1 and worse in some cases, was surely not going to give remotely sane results at 720p@2megabits. I was right.

[CruNcher]

What i find somehow crazy about these pre-results is to see Elecards Encoder so heavily behind personally i can't agree with the results from a Visual point of view and also my beta experience with it is a different one then what reflects in these mathematical results :( also that Elecard was involved in the Mainconcept Encoder in one way or another (remember when all their stuff was Mainconcept branded) to some degree let it seem unreal somehow.

[Dark Shikari]

What i find somehow crazy about these pre-results is to see Elecards Encoder so heavily behind personally i can't agree with the results from a Visual point of view and also my beta experience with it is a different one then what reflects in these mathematical results :( also that Elecard was involved in the Mainconcept Encoder in one way or another (remember when all their stuff was Mainconcept branded) to some degree let it seem unreal somehow.I think there are a couple things going on here:

1. Elecard's options weren't very optimized (theirs was way above the target speed, and they only submitted a High Speed preset).

2. I agree that Elecard's encoder gives significantly better results than the latest Mainconcept Reference demo from the Mainconcept website.

but ...

3. I keep getting the feeling that all the Mainconcept tests thrown about recently are on some magical build of Mainconcept that I have yet to see, because the Mainconcept Reference on the site is so mediocre that I cannot imagine that it is the one ranking so high in so many tests.

Also, I will be leaking the full draft report if a full week passes without me hearing a response from MSU (i.e. a day or two from now).

[kolak]

Seems most blurays are simply encoded at whatever bitrate fills the available space on the disc: resulting in encodes with hugely excessive bitrates for the source quality. At bitrates that high, it shouldn't really matter which of the popular encoders you use. Why wouldn't you use the free one?

There are lots of other factors than qulity with encoders, which are important for professional authoring studio. x264 is good, but it lucks lots of features, which are must have for an authoring studio. It's also rather optimised for low bitrates <15Mbit, which is way below ones found on the Blu-ray discs.

Your statement about bitrate is also not true. Reference quality encodes need as high bitrate as possible and even with 40Mbit there are sometimes problems. I've had a source which was killing encoder even at 35Mbit.


Andrew

[Chengbin]

What i find somehow crazy about these pre-results is to see Elecards Encoder so heavily behind personally i can't agree with the results from a Visual point of view and also my beta experience with it is a different one then what reflects in these mathematical results :( also that Elecard was involved in the Mainconcept Encoder in one way or another (remember when all their stuff was Mainconcept branded) to some degree let it seem unreal somehow.

How did you get your hands on the results?

[Gabriel_Bouvigne]

Youtube, Facebook and warez use x264 not because it's the best but simply because it's free.

I'm not sure that Facebook or YouTube choosed x264 because of its cost. Unless the codec vendor is totally unreasonable, once your transcoding nodes are running 24/7, codec cost is not that high anymore, and you then really care about other points. (quality, speed, ease of integration within your workflow,...)

[Dark Shikari]

And I finally got a response one week later, so I'm not leaking anything ;)

Turns out that they seem to be interested in not making this a crock, and there were some misunderstandings!

1. Their results were based on SSIM, not PSNR--it was just very not clear that this was the case (they're going to clear it up in the final report). The problem was rather that they screwed up in making their "conclusion" graphs, resulting in x264 losing in places it shouldn't have.

2. I completely missed one series of tests: the series of CIF "videoconferencing" tests. Mainconcept won many of these, especially at low bitrates. My suspicion is that because it was 1-pass encoded and the clips were extremely short, x264 chose too high a QP for the first I-frame, which was sufficient to butcher quality at ultra-low bitrates in nearly-static clips like Akiyo. Unfortunately these clips don't actually represent reality, since in reality, you generally encode more than 300 frames at a time with a whole lot more than one I-frame. Because their scoring system counts large wins so highly, a couple huge Mainconcept wins at low bitrates here was sufficient to budge the overall score a surprising amount in its favor.

Overall, it seems to be a case of insufficient information in the report and incompetence rather than malice. Let's hope the final report is better.

Edit: suspicions confirmed about Akiyo:

17:24 < Alex_W> 1 pass:
17:24 < Alex_W> x264 [info]: SSIM Mean Y:0.9676780
17:24 < Alex_W> x264 [info]: PSNR Mean Y:39.636 U:42.036 V:43.483
Avg:40.416 Global:39.396 kb/s: 62.23
17:24 < Alex_W> 2 pass:
17:24 < Alex_W> x264 [info]: SSIM Mean Y:0.9790610
17:24 < Alex_W> x264 [info]: PSNR Mean Y:41.145 U:43.939 V:44.836
Avg:41.976 Global:41.953 kb/s: 62.19
17:25 < Alex_W> in the 1 pass encode the I-frame has an average QP of 41.87 while in the 2 pass encode it has an average QP of 21.63

[CruNcher]

Akiyo isn't that the News Reporter Scene ?
It makes me wonder how the difference of Mainconcept vs x264 1pass would have looked @ riverbed being non static @ all

[Schrade]

Are the encoding parameters/settings for all the encoders made public for these tests?

[benwaggoner]

I've had a source which was killing encoder even at 35Mbit.
What's that source like?

The HBO logo that fades in from white noise still gives me nightmares...

[kolak]

What's that source like?

The HBO logo that fades in from white noise still gives me nightmares...

It's a live recording of the opera- 1080 60i. There is noise, lots of details and high contrast. It just kills every encoder.
It was DNRed before encoding- there was no other way.


Andrew

[Sharktooth]

that's what you get when you trust metrics as a measure of quality.
you get cheated. you cant simply state encoder X is better than Y coz of a higher metric score...
as a matter of fact, as DS said, optimizing for SSIM lowers the PSNR... and probably optimizing for other metrics results in the same situation...
so who's right? obviously your eyes... not some stupid numbers that represents NOTHING.