Codec Runoff: H.264/AVC vs WMV9 vs Xvid

[Dark Shikari]

This test was done for the EVE Online forums to find out and demonstrate which codecs were best for player-made videos, but the results were impressive enough that I figured I would adapt the post for this forum also.

The source video used for these tests can be found here, compressed in high-bitrate H.264 format in an MKV container. To do the tests, I cropped it to 1680x1040 from 1680x1050, so that it was a multiple of 16 in both directions for optimal compression.

To represent the fact that 0.99 SSIM is has half the deviation from the source material as 0.98 SSIM and so forth, I also created a "Relative Quality Rating" graph, where Xvid at 2231kbps is 1.0 and all the others are the relative quality compared to Xvid 2231kbps. So if Xvid at 2231kbps was 0.90 SSIM, and H.264 at 4029kbps was 0.99 SSIM, the score for the H.264 graph would be 10, because the H.264 encode had 10 times less variation from the source than the Xvid encode.

Some notes about the codecs tested:

Xvid: Unrestricted profile, maximum VHQ, all the best settings basically. Could not, regardless of quality settings, get the bitrate below about 1600kbps, so I did not include a 1000kbps option on the graph. Even 2000kbps was hard, and it came out as 2231kbps when I set 2000 as the target.

WMV9 Main/Complex Profile: These had serious frame skipping problems at 1000kbps; the WMV codec often skips frames to save space when it can't lower the quality much more. This, however, makes it impossible to take an accurate SSIM reading. As a result only 2000kbps and 4000kbps are included. Maximum quality setting was used in the options panel.

WMV9 Advanced Profile: Frame skipping problems even at 2000kbps, and crashed at 4000kbps, so not included.

H.264: No problems at any bitrate. Command-lines used were:

x264 --trellis 2 --me umh --bime --merange 64 --no-fast-pskip --partitions all --8x8dct --pass 1 --bitrate 4000 --subme 7 --bframes 16 --b-pyramid --ref 16 --b-rdo --mixed-refs --direct auto --weightb --progress --threads 4 --deblock 1:-1 -o full_frames_test.mkv read_mkv.avs

x264 --trellis 2 --me umh --bime --merange 64 --no-fast-pskip --partitions all --8x8dct --pass 2 --bitrate 4000 --subme 7 --bframes 16 --b-pyramid --ref 16 --b-rdo --mixed-refs --direct auto --weightb --progress --threads 4 --deblock 1:-1 -o full_frames_test.mkv read_mkv.avs

(yes, I know the first pass has overkill commands, but I didn't bother to change them)

Most of the commandlines have been manually optimized for SSIM in EVE videos by me, simply by trial and error and comparing resulting SSIM values. That's where I got the --deblock 1:-1 from, for example. I used the MSU video quality measurement tool to get the data for the graphs below.

Graphs (time axis is the frames of the video):


A smoothed graph of the relative quality, with Xvid 2231kbps as 1.0.





Conclusion:

H.264 blows away everything else. At 1000kbps, an incredibly low bitrate that the other codecs could not even reach, it beats WMV9 handily at 2000kbps by a wide margin and blows away Xvid. At 2000kbps, it edges out WMV9 at 4000kbps by a significant margin, and at 4000kbps it is by far and wide in a class of its own. Some people say that without CABAC or the like H.264 is hardly better than Xvid or Windows Media, but considering that its quality is two or three times better than the other codecs here at the same bitrate, I would doubt that.

[foxyshadis]

0.005 SSIM is not exactly a "wide margin" or a blowout. Have you tried to ABX them in any way to support your contention that "0.99 SSIM is twice as good as 0.98 SSIM and so forth"? I've never heard that argument before. As always, objective tests point to general trends and areas to research, but only subjective tests can lead to firm conclusions. (And it is a useful datapoint, I'm not trying to deny that, just that the conclusion seems to be much too strong for the data represented.)

[Dark Shikari]

Originally Posted by foxyshadis

0.005 SSIM is not exactly a "wide margin" or a blowout. Have you tried to ABX them in any way to support your contention that "0.99 SSIM is twice as good as 0.98 SSIM and so forth"? I've never heard that argument before. As always, objective tests point to general trends and areas to research, but only subjective tests can lead to firm conclusions. (And it is a useful datapoint, I'm not trying to deny that, just that the conclusion seems to be much too strong for the data represented.)

I'm pretty sure that 0.99 SSIM means half the variation from the source as 0.98 SSIM.

Another way to compare is with the bitrates; 2000kbps H.264 is better than 4000kbps WMV9, for example, which appears pretty impressive.

0.005 SSIM is a "wide margin" when you would probably need 7000, 8000, or more kbps in WMV9 to equal it. I will try testing that and see exactly how much is needed to equal it.

[akupenguin]

+.005 SSIM is a wide margin if it's .995-.990. It's not a wide margin if it's .905-.900. Unlike PSNR, SSIM is not a logarithmic scale, so computing differences doesn't make much sense. .99 SSIM is twice as good as .98 SSIM in exactly the same sense as 45 dB is twice as good as 42 dB.

[Dark Shikari]

Originally Posted by Dark Shikari

I'm pretty sure that 0.99 SSIM means half the variation from the source as 0.98 SSIM.

Another way to compare is with the bitrates; 2000kbps H.264 is better than 4000kbps WMV9, for example, which appears pretty impressive.

0.005 SSIM is a "wide margin" when you would probably need 7000, 8000, or more kbps in WMV9 to equal it. I will try testing that and see exactly how much is needed to equal it.

Results on this are in. 10000kbps WMV9 Main Profile gives 0.99458 Average SSIM, which is still worse than 4000kbps H.264.

[Sagittaire]

Originally Posted by Dark Shikari

Results on this are in. 10000kbps WMV9 Main Profile gives 0.99458 Average SSIM, which is still worse than 4000kbps H.264.

In this case you have problem with your encoding method ...

[Dark Shikari]

Originally Posted by Sagittaire

In this case you have problem with your encoding method ...

I doubt it. Max quality settings were used, using the bog-standard Windows Media Encoder. AviSynth + DirectShowSource were used to read them back for the quality test.

I'm pretty sure its just because WMV9 is, in comparison to H.264, very mediocre, especially with the very intensive H.264 settings I used (16 reference frames!)

I tried VC-1, as stated in the first post, but it skipped frames and crashed and so I couldn't really do a comparison with it. I'm not sure how much better VC-1 is than WMV9 Main Profile.

[Sagittaire]

I doubt it. Max quality settings were used, using the bog-standard Windows Media Encoder. AviSynth + DirectShowSource were used to read them back for the quality test.

You are sure that all your decoder work in the same color space? You have desactived the post process for WMV9 (WMV9 DMO decoder use PP4 by default)?

I'm pretty sure its just because WMV9 is, in comparison to H.264, very mediocre, especially with the very intensive H.264 settings I used (16 reference frames!)

Well 16 ref is completely useless setting like 16 bframes. X264 will never use in practice 16 ref or 16 consecutive bframe.

I tried VC-1, as stated in the first post, but it skipped frames and crashed and so I couldn't really do a comparison with it. I'm not sure how much better VC-1 is than WMV9 Main Profile.

H264 is certainely better than WMV9, but the quality difference is certainely not at 250% ...

[Dark Shikari]

Originally Posted by Sagittaire

You are sure that all your decoder work in the same color space? You have desactived the post process for WMV9 (WMV9 DMO decoder use PP4 by default)?

I didn't know there were any decoding settings for WMV9 at all. How do I set them?

[Sagittaire]

http://multimediacom.free.fr/Video/WMVPostpross.exe

[sillKotscha]

Originally Posted by Dark Shikari

How do I set them?

for example with WMV PowerToy.exe

[Terranigma]

Originally Posted by Sagittaire

Well 16 ref is completely useless setting like 16 bframes. X264 will never use in practice 16 ref or 16 consecutive bframe.

You can force it to use 16 b-frames by increasing the b-frame bias. A setting of 200 will most likely result in usage of 16 b-frames.

[Sagittaire]

Originally Posted by Terranigma

You can force it to use 16 b-frames by increasing the b-frame bias. A setting of 200 will most likely result in usage of 16 b-frames.

Certainely but it's not the best efficiency way ...

[Terranigma]

I agree ;-)

[Dark Shikari]

Originally Posted by Sagittaire

http://multimediacom.free.fr/Video/WMVPostpross.exe

It says that the deblocking is only good for low-quality content, and its postprocessing, rather than inloop deblocking like H.264. Are you sure this would improve quality at 10000kbps?

[foxyshadis]

Originally Posted by Dark Shikari

I'm pretty sure that 0.99 SSIM means half the variation from the source as 0.98 SSIM.

Be precise with your terms then. Distortion is not quality; quality is one of those subjective things which can only be measured by human eyes (as we don't yet have a complete mathematical model for them, only approximations). Unless you define and adhere to an alternate definition of quality, the assumption is human visual quality.

I'd like to try to find a threshold of visual detection of distortion. I wonder how hard such a test would be to create.

[Dark Shikari]

Originally Posted by foxyshadis

Be precise with your terms then. Distortion is not quality; quality is one of those subjective things which can only be measured by human eyes (as we don't yet have a complete mathematical model for them, only approximations). Unless you define and adhere to an alternate definition of quality, the assumption is human visual quality.

I'd like to try to find a threshold of visual detection of distortion. I wonder how hard such a test would be to create.

Well that is the goal of most of these measures. SSIM tries to be more accurate in relation to human perception than PSNR and the like. Obviously its not perfect, but its one of the best out there.

[foxyshadis]

http://www.ece.northwestern.edu/~pap...oks_hvei06.pdf (No implementations are currently available.)

I think the source is naturally slanted to h.264 now that I looked at it (mainly because inloop already had its way with it, apparently), and I question the meaning of distortion against an already distorted source. But of course, put up or shut up, so I'm looking for a better source; currently considering HD elephant's dream and some raw game captures.

[akupenguin]

Originally Posted by Dark Shikari

The source video used for these tests can be found here, compressed in high-bitrate H.264 format in an MKV container.

That does mean "I ran the test on an uncompressed source, here's the compressed version just so you can see what it looks like", right? Because if the source was already compressed with H.264, that easily explains H.264's advantage.

[Dark Shikari]

Originally Posted by akupenguin

That does mean "I ran the test on an uncompressed source, here's the compressed version just so you can see what it looks like", right? Because if the source was already compressed with H.264, that easily explains H.264's advantage.

Just curious, why would that increase H.264's lead though? Would it be because H.264 has a unique pattern of artifacts that are compressed best by itself?