VP8 vs x264

[Emanem_]

Hi all,

I've done a simple comparison between VP8 and x264 computing SSIM and PSNR for some fragments of HD (720p video).
It's here:
http://qpsnr.youlink.org/vp8_x264/VP8_vs_x264.html

Please let me know what you think!
Cheers,

Ps. I've posted this in the alternative codecs section as well, I hope it's ok, otherwise please close one of them and redirect to the other :-)

[LoRd_MuldeR]

You don't show your x264 command-line explicitly. I only see "default settings". So I have to assume you didn't use "--tune psnr" or "--tune ssime", right?

If that was the case, then you had Psy-optimizations enabled (x264 default) and thus the whole PSNR/SSIM comparison is worthless, as Psy-optimizations massively hurt PSNR and SSIM.

Needless to say that metrics, such as SSIM or even PSNR, can only give a very rough idea about the perceived quality anyway...

[Emanem_]

You don't show your x264 command-line explicitly. I only see "default settings". So I have to assume you didn't use "--tune psnr" or "--tune ssime", right?

If that was the case, then you had Psy-optimizations enabled and thus the whole PSNR/SSIM comparison is worthless, as Psy-optimizations massively hurt PSNR and SSIM.

No definitely didn't have those on.
The point is, if I leave those on, the codec will cheat trying to better adapt PSNR and SSIM instead of performing what is expected to do (i.e. trimming the data our eye should not be able to understand).

Then I used PSNR and SSIM in order to provide a number as well, not only a subjective test.

I see this flag as when the ATi drivers were detecting that quake3 was running and degraded the quality of textures in order to get better FPS...

I'm not trying to look for a better number to satisfy PSNR and/or SSIM, but to satisfy the eye, and then provide some numbers as well.

Btw if there were such optimizations for PSNR and/or SSIM for VP8, I'd leave them off as well...

Cheers,

Ps. I know that even psy optimization can't be good for everyone, that for example our sensitivity to red is different, and even shapes, ...

[LoRd_MuldeR]

No definitely didn't have those on.
The point is, if I leave those on, the codec will cheat trying to better adapt PSNR and SSIM instead of performing what is expected to do (i.e. trimming the data our eye should not be able to understand).

What you say makes no sense. If you decide to use a specific metric to judge quality, but you leave options enabled the intentionally hurt those metrics, you screw up the results. x264 will not "cheat", if you pass "--tune psnr" or "--tune ssim". It simply will disable certain "optimizations" that are known to greatly improve the perceived quality, but inherently hurt quality metrics because of the way they work. This also is another prove how one can be fooled when relying to much on quality metrics. Sometimes you need to make decisions that hurt those metrics in order to get the visually improved result.

After all it's your test and I only was giving suggestions. If you want to keep a worthless and misleading test on your web-site, then it's your decision...


I'm not trying to look for a better number to satisfy PSNR and/or SSIM, but to satisfy the eye, and then provide some numbers as well.

Then do the visual comparison with default settings (i.e. Psy-optimizations enabled) and do the comparison of PSNR or SSIM numbers with the appropriate settings (i.e. Psy-optimizations off).

[Emanem_]

What you say makes no sense. If decide to use a metric to judge quality, but you leave options enabled the intentionally hurt those metrics, you screw up the results. x264 will not "cheat", if you pass "--tune psnr" or "--tune ssim". It simply will disable certain "optimizations" that are known to greatly improve the perceived quality, but inherently hurt quality metrics because of the way they work. This also is another prove how one can be fooled when relying to much on quality metrics. Sometimes you need to make decisions that hurt those metrics in order to get visually improved results.

After all it's your test and I only was giving suggestions. If you want to keep a worthless and misleading test on your web-site, then it's your decision...




Then do the visual comparison with default settings (i.e. Psy-optimizations enabled) and do the comparison of numbers (PSNR or SSIM) with the appropriate settings.

First of all thanks for you suggestions.

Nevertheless probably I wasn't clear enough.
From my point of view I follow this logic:
1) A video codec has to digitally compress the video stream to provide best quality for the eye.
2) If I want to test a video codec, I have to test against the settings that video codec is supposed to be used in day to day usage.
3) I can't just provide a subjective test and I need some numbers. Again asking the codec to tune its output to satisfy a test makes the test useless.
4) As seen as a number is better than no number, I provide SSIM and PSNR. but, due to point (1) and (2) I don't turn on optimizations that would affect quality for the eye to better have a PSNR/SSIM.

Is a bit like if we have to compare trucks.
But then we have a special truck, that has an option to morph into a Ferrari.
Now, let's say we're measuring the speed of trucks on a circuit. We're testing, as trucks, those vehicles.
Everyone that would buy that truck will use it as a truck.
Would you turn Ferrari mode on just for this test?

Cheers,

Btw, to a certain extent I think we're both right.

[Keiyakusha]

Is there any other metrics that can be used so the question with --tune ssim/psnr won't be valid anymore? PEVQ, ITU-T J.247?

[Manao]

Emanem_ : I think everybody will agree on your points 1) and 2). But 1) conflicts with 3). Especially since a visual comparison with x264 at default settings and x264 at default settings + --tune psnr will visually show that the higher the psnr definitely doesn't mean the higher the visual quality.

[LoRd_MuldeR]

Is a bit like if we have to compare trucks.
But then we have a special truck, that has an option to morph into a Ferrari.
Now, let's say we're measuring the speed of trucks on a circuit. We're testing, as trucks, those vehicles.
Everyone that would buy that truck will use it as a truck.
Would you turn Ferrari mode on just for this test?

I think that analogy doesn't capture the facts. But to stay with your "car" analogy, we could think about it like this:


We want to compare cars. And we want to know: Which car drives best ???

As that is very abstract and too hard to compare, we need some "hard" numbers. So we decide to only compare the maximum speed that the cars can reach (on a straight 1/4 mile track).

Now let's assume one car has an "optimization" that greatly improves handling and thus performs extremely well in daily life, but comes at the cost of reduced maximum speed.

In our "hard" number comparison that car will perform worse, if we keep the "optimization" enabled. That's because in our comparison we harshly ignore all aspects, except for the maximum speed.

Still in real life that car would perform better than the others, because maximum speed is worthless without good handling.

Also our comparison, which we have chosen to restrict to one single aspect (maximum speed) is unfair for the one car, as it could perform better in our comparison, but we simply don't allow that.

Instead we keep the "optimization" enabled, although we know that this optimization intentionally (and for good reason) hurts the one aspect we have chosen to compare.


The proper and fair method would be: Tell all car manufacturers that we are going to compare maximum speed only. And give them the chance to setup their cars for this specific contest.

[Emanem_]

Trying to answer to all in one post :P

I know, that's why I tried to be as objective as I could and compare both PSNR and SSIM.
Personally I think PSNR numbers are definitely to take with a grain of salt, and imho I prefer usually SSIM. Or better, I prefer them both.
Sometimes PSNR can capture what SSIM does and viceversa and sometimes both won't capture anything.

Nevertheless (1) and (2) conflicts with (3), but still we need an objective metric used against the product in the same way as day to day use.

Is a bit like VaR/sensitivities in finance, these are numbers that alone don't mean anything, and you have to fully analyze different aspectes of the whole to get an idea of (sort of) what's going on.

And indeed, in my analysis I try to focus when I see big gaps in PSNR and/or SSIM and then I save the frames and visually compare them.
And I watched the segments as well.

Now, I'd rather not turn on psnr and/or ssim optimizations because then if I'd run a sort-of-pyschovisual benchmark on that product, I'd see bad video, not representing the usage.

Btw I disagree on the following:

The proper and fair method would be: Tell all car manufacturers that we are going to compare maximum speed only. And give them the chance to setup their cars for this specific contest.

I could put a rocket on a car and I'd win. Now please tell me you'd buy that car to drive it :)
Again we're comparing what's drives best (to use your analogy)

This is why instead of only PSNR or SSIM I decided to use them both with some subjective analysis (see some screenshots).
I am alone, I can't provide a metric like PEVQ myself, I don't have time and resources to hire a group of persons and ask them to provide evaluation.

If you guys would like to recommend any other objective metric that I can implement in C/C++ in qpsnr I'll be more than happy to extend the software and use that (other than PSNR/SSIM/subjective) to evaluate different options.

Nevertheless, if you read the whole (not too serious) analysis, I don't just say this is better because the PSNR is higher or this is c**p because the SSIM is way below 1.
I provide what I can with my means (that indeed are limited), but I think the analysis is fair.

Cheers guys,
Let me know what you think! :)

Ps. PSNR is being computed again RGB colorspace, while SSIM against Y (of YCbCr)... I try to cover color and shapes.

[LoRd_MuldeR]

Btw I disagree on the following:

The proper and fair method would be: Tell all car manufacturers that we are going to compare maximum speed only. And give them the chance to setup their cars for this specific contest.

I could put a rocket on a car and I'd win. Now please tell me you'd buy that car to drive it :)
Again we're comparing what's drives best (to use your analogy)

Now you are joking. Of course we would compare the cars as they are sold to the customers.

But, as we have have chosen to make a highly specific test, which concentrates on one single aspect and ignores all the rest, we must allow the manufacturers to adapt their cars for the test.

If we don't allow that, our test result will simply be random! That's because the "default" setup of an individual car might be more or less suitable for our specific test - just by chance.

Of course the manufacturers will only be allowed to tweak the setup of their cars. They will not be allowed to add any additional "non-standard" parts (motors, rockets, whatever) to their car.


That's also how proper Codec comparisons are done in reality:

First the testing methodology is publicly announced. For example the comparision can be announced to be about a certain metric (i.e. PSNR or SSIM) only.

Or the test can be announced to be a "visual" comparison only, performed by human subjects only.

And then, once the testing methodology is clearly defined, the Codec developers get the chance to adapt their Codec for the specific test. Finally they submit their test settings/configuration.

Of course they must use the "standard" version of their Codec, the version that will be used by the end-users too. And using special non-standard hardware isn't allowed either.

[Emanem_]

Of course I was joking :)

Anyway, I didn't just use PSNR.
I used SSIM, PSNR and then subjective analysis as well.

Please propose any other object C/C++ implementable objective metric and I'll try to implement and run that as well.

The point of my analysis was to do a more than possible objective review of VP8 and x264, using PSNR/SSIM and actually viewing the results; what do you think about it about the conclusions?

I've quite played around VP8 quite a bit these days.
It's astoundingly slow, really slow. But apparently finally we have something that can be compared with x264.

Have a look and tell me what you think.

Cheers :)

[LoRd_MuldeR]

Of course I was joking :)

Anyway, I didn't just use PSNR.
I used SSIM, PSNR and then subjective analysis as well.

That's not the problem. Doing a subjective analysis and a metric-based one, is a good idea.

The problem is that when you did the SSIM/PSNR test with x264, you tweaked it (or kept it tweaked) against PSNR/SSIM.

Consequently your results for the PSNR/SSIM test were unfair/biased.


Please propose any other object C/C++ implementable objective metric and I'll try to implement and run that as well.

The problem is: The one objective quality metric that perfectly predicts the perceived quality doesn't exit ;)

We are far away from fully understanding how the HVS (human visual system) works. How can we accurately model it in a computer program then?

So whatever metric we use, it will always be a simplifying model and it will always be restricted to certain aspects, leaving out others.

That doesn't mean metrics are useless! Sometimes we must have "hard" numbers and then using metrics is the only way. But we should be aware of the limitations of such a test!

This also means: If we know that a certain optimization while being visually advantageous hurts a specific metric, we don't keep that optimization enabled for that metric.


Some interesting reading about quality metrics:
http://www.ece.uwaterloo.ca/~z70wang/publications/SPM09.pdf

[Emanem_]

Btw I think default VP8 is twaeked for human eye as well :)
Who would encode video in VP8 if default would be tweaked for PSNR and/or SSIM? ;)

Anyway, cool, thanks for the paper, I'll give it a look later on.

Now, back to business, what do you think about the analysis itself and VP8?

Let me know!
Cheers,

[Blue_MiSfit]

I don't think VP8 has any psy optimization currently, If I recall correctly from Dark_Shikari's blog.

A few thoughts

1) --tune ssim and --tune psnr doesn't enable anything. It actually just DISABLES part of the encoder that would hurt the respective metrics. If you don't do this.. well you could easily come away with the impression that x264 isn't as good as it actually is!

2) SSIM and PSNR are valid in certain contexts. If you're going to do a comparison, you must use the proper tuning. However, you should always do a subjective analysis as well, with the psy optimizations turned on. You can do one optimized for metrics as well, and be blown away at how good psy optimizations are :)

Derek

[foxyshadis]

Default VP8 is tweaked heavily for PSNR. That's why it wasn't well accepted here once psy opts made such huge strides in x264. Now Google is undoing that and introducing psy into VP8, at the cost of PSNR. By posting PSNR and SSIM scores, you may as well be throwing up random numbers, because they won't correspond to reality. Google is using something like CWSSIM last I heard now, actually.

You want an a truly valid objective score, use what HydrogenAudio uses: ABX. Get enough people to ABX or judge perceived quality on a 1-10 scale and statistically you have a hard metric. A computer can't judge visual quality, stop making excuses, and don't bother making comparisons if you aren't going to make useful, valid ones that require effort. It just adds to the overall noise level on the topic.

[mandarinka]

You gave a (potentialy big) advantage to VP8 by misconfiguring x264. Your comparison is flawed, it is as simple as that.

The metrics are used for evaluating the encoder's prowess in things like motion search and RDO, stuff that (to simplify it) comes before the encoder begins to use the tricks of psy optimizations. Something like a raw power. If you meassure PSNR or SSIM after psy optimizations do their job, you will get improper information about the thing you wanted to know. As it turns out, when x264 does the tricks to improve visual quality, the raw power as represented by PSNR suddenly seems to be much lower... (With VP8, there are no psy optimizations, so it's results won't get harmed.)

[ricardo.santos]

Please correct me if im wrong, im not an expert on PSNR etc etc but i see that you're trying to test the quality of both encoders and i think you missed soomething very importante:

x264 2k Bitrate: 1556
VP8 2k Bitrate: 2378

x264 2k Bitrate: 2030
VP8 2k Bitrate: 2614

Whatever results those tests show, VP8 has a higher bitrate and that to me means better quality... without knowing you're "cheating" the test.

i've come across that bug with ffmpeg (bitrate, oversizing), so to test both encoders i used Nic's ivfenc version with avs support and tested both through the avs with same bitrate.

[poisondeathray]

nice catch ricardo! did you ever get that sorted out (the bitrate issue for vp8 using ffmpeg version?)

[ricardo.santos]

Hi poisondeathray, no i couldnt find a fix, the only way i could get the exact bitrate with VP8 was to use nic's ivfenc avisynth version.

I suspect a few "good vp8 reviews" are made with ffmpeg and people comment about good quality but are unaware of the bitrate issue.

[Guest]

I've posted this in the alternative codecs section as well, I hope it's ok Not OK. Read and follow our forum rules!

[CruNcher]

Emanem_
if you would really compare it on a psy level VP8 currently has only 1 real Psy like option and that's a very basic deadzone change done via --sharpness=x (which could have been usefull in this case)
also posting non correct frame comparisons with frames that are different between source and encodes doesn't help much (also not informing what frames we actually see their and how far they away from the keyframe isn't nice)

And yeah it's obvious you didn't hit bitrate with ffmpeg and posting this then as valid comparison is blatantly wrong you should know that

read this http://pastebin.org/346660 understand it live it :)

[Dark Shikari]

Emanem_
if you would really compare it on a psy level VP8 currently has only 1 real Psy like option and that's a very basic deadzone change done via --sharpness=x (which could have been usefull in this case)
also posting non correct frame comparisons with frames that are different between source and encodes doesn't help much (also not informing what frames we actually see their and how far they away from the keyframe isn't nice)

And yeah it's obvious you didn't hit bitrate with ffmpeg and posting this then as valid comparison is blatantly wrong you should know that

read this http://pastebin.org/346660 understand it live it :)Come on, don't post my drafts, post the link to the actual blog post (http://x264dev.multimedia.cx/?p=472) :)

[CruNcher]

Yeah sorry credits go to Dark and all the other contributors in #x264 :)

[Emanem_]

Come on, don't post my drafts, post the link to the actual blog post (http://x264dev.multimedia.cx/?p=472) :)
Thanks for your ideas and reccomendations to properly test videos.
How would you objectively test codecs with psy on? Which metric would you reccommend?

OMG, I posted that table with 'real' bitrate and thought that encoders would have used the requested bitrate!

As soon as I have time I'll reupdate and republish it.

Regarding objective measures, do you have any metric to suggest other than PSNR and SSIM?
Cheers,

[Dark Shikari]

Thanks for your ideas and reccomendations to properly test videos.
How would you objectively test codecs with psy on? Which metric would you reccommend?That's inherently not possible -- I'd use screenshots and my eyes.Regarding objective measures, do you have any metric to suggest other than PSNR and SSIM?
Cheers,Not that I know of. There's some fancy ones, but I don't know of any implementations. Or any proof that they're any good.

Here's what I'd do:

1. Pick a metric (SSIM or PSNR).

2. Tune all tested encoders, wherever possible, for these metrics (e.g. --tune psnr in x264). Use the slowest settings and so forth -- get the most out of each encoder.

3. Do an objective chart accordingly.

4. Tune the encoders for psy instead, and use that to generate screenshots.

The downsides of this method are that:

a) The graphs will bias against encoders that have good psy optimizations, like x264. This is because they don't take into account the fact that psy optimizations make the difference between encoders greater than one would measure via raw objective metrics.

b) The graphs will bias even more against encoders that can't turn off their psy optimizations (e.g. if you're optimizing for PSNR, Theora Ptalarbvorm is by-default-and-you-can't-turn-it-off optimized for SSIM). This is to the point where if you can't disable psy opts, objective testing is basically unfair and shouldn't be done.

[Emanem_]

That's inherently not possible -- I'd use screenshots and my eyes.Not that I know of. There's some fancy ones, but I don't know of any implementations. Or any proof that they're any good.


Thanks again! :)

I do understand your points.
But how must be hard to develop a codec and test/verify that what you've done is an improvement if there is no such automated way to do it? Do you authors of x264 have at your disposition a group of experts to help you with psy measurements?

Anyway, about x264 and VP8, I've always used the slowest settings (think during encoding that VP8 in the Shark scene goes 1.5 FPS while x264 30+ on my Phenom II X4 965 B.E.).

Apart the stupid mistake (not taking a look at the table with file size and real bitrates), I've always tried to compare apples to apples (i.e. 4 MBps bitrate) and I tried to do a visual comparison as well.
And, with surprise, now that I look at that table again, VP8 bitrate for 4 MBps is lower than x264! So could it have used more quality? Perhaps those 40 KBps more in the first Avatar scene could decrease distance with x264 HP?

Indeed my comparison didn't pretend to be serious (it's in the title:)).

But at the end of the day I've found out that VP8 is very good quality wise.
Have you ever give it a shot, you know just out of curiosity, to take a peek at the competition? Have you ever encoded a HD video with VP8 and seen it? I'm just curious...

I mean, clearly in the eyes of an expert my comparison lacks of maybe ppm (or png) frames (btw should I be authorized to publish them if I don't own the images?) and the final encodings, plus maintaining a strict filesize between different encodings.
But you see, I've always tried to be more fair than possible.

At the end of the day I look to the better codec as the codec which given the same bitrate will provide more quality or if same quality least disadvantages (i.e. no software patents and/or speed).

Thanks again for your hints,
Regards,

Ps. I'm not against x264. I don't like the fact that H.264 is patented (and generally software patents do exist). Nevertheless I still think x264 is the best and stuff encoded with it works on my PS3 beautifully ;)
I have never taken a look at x264 sources, but I bet the quality is as the software itself, really and truly exceptional. Too bad for patents.

[Dark Shikari]

Thanks again! :)

I do understand your points.
But how must be hard to develop a codec and test/verify that what you've done is an improvement if there is no such automated way to do it? Do you authors of x264 have at your disposition a group of experts to help you with psy measurements?We have our eyes and intuitions ;) But of course I don't claim to have "golden eyes" or any of that type of BS that audiophiles like to spout; I'm as limited as anyone else.

Anyway, about x264 and VP8, I've always used the slowest settings (think during encoding that VP8 in the Shark scene goes 1.5 FPS while x264 30+ on my Phenom II X4 965 B.E.).30+ FPS isn't the slowest ;). Try placebo.

But at the end of the day I've found out that VP8 is very good quality wise.
Have you ever give it a shot, you know just out of curiosity, to take a peek at the competition? Have you ever encoded a HD video with VP8 and seen it? I'm just curious...Yes, I tested it as part of my analysis of VP8. It came out rather lacking, losing by about 20-30% PSNR-wise and even more visually (due to lack of psy opts). It beat x264 Baseline profile by about 8% though, PSNR-wise. Of course, in a visual comparison, even x264 Baseline beat VP8, but that's not surprising.

It's even worse if you pick a non-super-slow mode though; VP8 does incredibly badly on its "fast" modes, which is what really matters for practical use (try encoding a 1080p movie with --best!).

Of course, speed issues are problems with the encoder, not the format. The format itself is a bit better compression-wise than H.264 Baseline, but greatly inferior to Main or High. Speed-wise, it's pretty similar to Main or High, maybe a bit slower.

[LoRd_MuldeR]

Regarding objective measures, do you have any metric to suggest other than PSNR and SSIM?
Cheers,Not that I know of. There's some fancy ones, but I don't know of any implementations. Or any proof that they're any good.

Well, there is the MSU Quality Measurement Tool (http://compression.ru/video/quality_measure/info_en.html) available, which in includes an implementation of VQM (http://compression.ru/video/quality_measure/vqm.pdf) along with some others.

Be aware: I have no experience how useful VQM is or how it compares to SSIM. Just mentioned it for completeness...