Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 24th April 2007, 15:22   #1  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,689
Codec Runoff: H.264/AVC vs WMV9 vs Xvid

This test was done for the EVE Online forums to find out and demonstrate which codecs were best for player-made videos, but the results were impressive enough that I figured I would adapt the post for this forum also.

The source video used for these tests can be found here, compressed in high-bitrate H.264 format in an MKV container. To do the tests, I cropped it to 1680x1040 from 1680x1050, so that it was a multiple of 16 in both directions for optimal compression.

To represent the fact that 0.99 SSIM is has half the deviation from the source material as 0.98 SSIM and so forth, I also created a "Relative Quality Rating" graph, where Xvid at 2231kbps is 1.0 and all the others are the relative quality compared to Xvid 2231kbps. So if Xvid at 2231kbps was 0.90 SSIM, and H.264 at 4029kbps was 0.99 SSIM, the score for the H.264 graph would be 10, because the H.264 encode had 10 times less variation from the source than the Xvid encode.

Some notes about the codecs tested:

Xvid: Unrestricted profile, maximum VHQ, all the best settings basically. Could not, regardless of quality settings, get the bitrate below about 1600kbps, so I did not include a 1000kbps option on the graph. Even 2000kbps was hard, and it came out as 2231kbps when I set 2000 as the target.

WMV9 Main/Complex Profile: These had serious frame skipping problems at 1000kbps; the WMV codec often skips frames to save space when it can't lower the quality much more. This, however, makes it impossible to take an accurate SSIM reading. As a result only 2000kbps and 4000kbps are included. Maximum quality setting was used in the options panel.

WMV9 Advanced Profile: Frame skipping problems even at 2000kbps, and crashed at 4000kbps, so not included.

H.264: No problems at any bitrate. Command-lines used were:

x264 --trellis 2 --me umh --bime --merange 64 --no-fast-pskip --partitions all --8x8dct --pass 1 --bitrate 4000 --subme 7 --bframes 16 --b-pyramid --ref 16 --b-rdo --mixed-refs --direct auto --weightb --progress --threads 4 --deblock 1:-1 -o full_frames_test.mkv read_mkv.avs

x264 --trellis 2 --me umh --bime --merange 64 --no-fast-pskip --partitions all --8x8dct --pass 2 --bitrate 4000 --subme 7 --bframes 16 --b-pyramid --ref 16 --b-rdo --mixed-refs --direct auto --weightb --progress --threads 4 --deblock 1:-1 -o full_frames_test.mkv read_mkv.avs

(yes, I know the first pass has overkill commands, but I didn't bother to change them)

Most of the commandlines have been manually optimized for SSIM in EVE videos by me, simply by trial and error and comparing resulting SSIM values. That's where I got the --deblock 1:-1 from, for example. I used the MSU video quality measurement tool to get the data for the graphs below.

Graphs (time axis is the frames of the video):


A smoothed graph of the relative quality, with Xvid 2231kbps as 1.0.





Conclusion:

H.264 blows away everything else. At 1000kbps, an incredibly low bitrate that the other codecs could not even reach, it beats WMV9 handily at 2000kbps by a wide margin and blows away Xvid. At 2000kbps, it edges out WMV9 at 4000kbps by a significant margin, and at 4000kbps it is by far and wide in a class of its own. Some people say that without CABAC or the like H.264 is hardly better than Xvid or Windows Media, but considering that its quality is two or three times better than the other codecs here at the same bitrate, I would doubt that.

Last edited by Dark Shikari; 24th April 2007 at 16:56.
Dark Shikari is offline   Reply With Quote
Old 24th April 2007, 16:38   #2  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,416
0.005 SSIM is not exactly a "wide margin" or a blowout. Have you tried to ABX them in any way to support your contention that "0.99 SSIM is twice as good as 0.98 SSIM and so forth"? I've never heard that argument before. As always, objective tests point to general trends and areas to research, but only subjective tests can lead to firm conclusions. (And it is a useful datapoint, I'm not trying to deny that, just that the conclusion seems to be much too strong for the data represented.)
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order.
foxyshadis is offline   Reply With Quote
Old 24th April 2007, 16:48   #3  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,689
Quote:
Originally Posted by foxyshadis View Post
0.005 SSIM is not exactly a "wide margin" or a blowout. Have you tried to ABX them in any way to support your contention that "0.99 SSIM is twice as good as 0.98 SSIM and so forth"? I've never heard that argument before. As always, objective tests point to general trends and areas to research, but only subjective tests can lead to firm conclusions. (And it is a useful datapoint, I'm not trying to deny that, just that the conclusion seems to be much too strong for the data represented.)
I'm pretty sure that 0.99 SSIM means half the variation from the source as 0.98 SSIM.

Another way to compare is with the bitrates; 2000kbps H.264 is better than 4000kbps WMV9, for example, which appears pretty impressive.

0.005 SSIM is a "wide margin" when you would probably need 7000, 8000, or more kbps in WMV9 to equal it. I will try testing that and see exactly how much is needed to equal it.

Last edited by Dark Shikari; 24th April 2007 at 16:52.
Dark Shikari is offline   Reply With Quote
Old 24th April 2007, 16:54   #4  |  Link
akupenguin
x264 developer
 
akupenguin's Avatar
 
Join Date: Sep 2004
Posts: 2,393
+.005 SSIM is a wide margin if it's .995-.990. It's not a wide margin if it's .905-.900. Unlike PSNR, SSIM is not a logarithmic scale, so computing differences doesn't make much sense. .99 SSIM is twice as good as .98 SSIM in exactly the same sense as 45 dB is twice as good as 42 dB.
akupenguin is offline   Reply With Quote
Old 24th April 2007, 17:45   #5  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,689
Quote:
Originally Posted by Dark Shikari View Post
I'm pretty sure that 0.99 SSIM means half the variation from the source as 0.98 SSIM.

Another way to compare is with the bitrates; 2000kbps H.264 is better than 4000kbps WMV9, for example, which appears pretty impressive.

0.005 SSIM is a "wide margin" when you would probably need 7000, 8000, or more kbps in WMV9 to equal it. I will try testing that and see exactly how much is needed to equal it.
Results on this are in. 10000kbps WMV9 Main Profile gives 0.99458 Average SSIM, which is still worse than 4000kbps H.264.
Dark Shikari is offline   Reply With Quote
Old 24th April 2007, 23:22   #6  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,419
Quote:
Originally Posted by Dark Shikari View Post
Results on this are in. 10000kbps WMV9 Main Profile gives 0.99458 Average SSIM, which is still worse than 4000kbps H.264.
In this case you have problem with your encoding method ...
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9
Sagittaire is offline   Reply With Quote
Old 24th April 2007, 23:27   #7  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,689
Quote:
Originally Posted by Sagittaire View Post
In this case you have problem with your encoding method ...
I doubt it. Max quality settings were used, using the bog-standard Windows Media Encoder. AviSynth + DirectShowSource were used to read them back for the quality test.

I'm pretty sure its just because WMV9 is, in comparison to H.264, very mediocre, especially with the very intensive H.264 settings I used (16 reference frames!)

I tried VC-1, as stated in the first post, but it skipped frames and crashed and so I couldn't really do a comparison with it. I'm not sure how much better VC-1 is than WMV9 Main Profile.

Last edited by Dark Shikari; 24th April 2007 at 23:31.
Dark Shikari is offline   Reply With Quote
Old 25th April 2007, 00:38   #8  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,419
Quote:
I doubt it. Max quality settings were used, using the bog-standard Windows Media Encoder. AviSynth + DirectShowSource were used to read them back for the quality test.
You are sure that all your decoder work in the same color space? You have desactived the post process for WMV9 (WMV9 DMO decoder use PP4 by default)?

Quote:
I'm pretty sure its just because WMV9 is, in comparison to H.264, very mediocre, especially with the very intensive H.264 settings I used (16 reference frames!)
Well 16 ref is completely useless setting like 16 bframes. X264 will never use in practice 16 ref or 16 consecutive bframe.


Quote:
I tried VC-1, as stated in the first post, but it skipped frames and crashed and so I couldn't really do a comparison with it. I'm not sure how much better VC-1 is than WMV9 Main Profile.
H264 is certainely better than WMV9, but the quality difference is certainely not at 250% ...
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9
Sagittaire is offline   Reply With Quote
Old 25th April 2007, 01:18   #9  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,689
Quote:
Originally Posted by Sagittaire View Post
You are sure that all your decoder work in the same color space? You have desactived the post process for WMV9 (WMV9 DMO decoder use PP4 by default)?
I didn't know there were any decoding settings for WMV9 at all. How do I set them?
Dark Shikari is offline   Reply With Quote
Old 25th April 2007, 01:24   #10  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,419
http://multimediacom.free.fr/Video/WMVPostpross.exe
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9
Sagittaire is offline   Reply With Quote
Old 25th April 2007, 01:26   #11  |  Link
sillKotscha
what's this for?
 
sillKotscha's Avatar
 
Join Date: Apr 2002
Location: Berlin, Germany
Posts: 993
Quote:
Originally Posted by Dark Shikari View Post
How do I set them?
for example with WMV PowerToy.exe

__________________
latest mix sets: http://soundcloud.com/sill
sillKotscha is offline   Reply With Quote
Old 25th April 2007, 01:53   #12  |  Link
Terranigma
*Space Reserved*
 
Terranigma's Avatar
 
Join Date: May 2006
Posts: 953
Quote:
Originally Posted by Sagittaire View Post
Well 16 ref is completely useless setting like 16 bframes. X264 will never use in practice 16 ref or 16 consecutive bframe.
You can force it to use 16 b-frames by increasing the b-frame bias. A setting of 200 will most likely result in usage of 16 b-frames.
Terranigma is offline   Reply With Quote
Old 25th April 2007, 02:05   #13  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,419
Quote:
Originally Posted by Terranigma View Post
You can force it to use 16 b-frames by increasing the b-frame bias. A setting of 200 will most likely result in usage of 16 b-frames.
Certainely but it's not the best efficiency way ...
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9
Sagittaire is offline   Reply With Quote
Old 25th April 2007, 02:09   #14  |  Link
Terranigma
*Space Reserved*
 
Terranigma's Avatar
 
Join Date: May 2006
Posts: 953
I agree ;-)
Terranigma is offline   Reply With Quote
Old 25th April 2007, 02:37   #15  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,689
Quote:
Originally Posted by Sagittaire View Post
It says that the deblocking is only good for low-quality content, and its postprocessing, rather than inloop deblocking like H.264. Are you sure this would improve quality at 10000kbps?
Dark Shikari is offline   Reply With Quote
Old 25th April 2007, 03:29   #16  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,416
Quote:
Originally Posted by Dark Shikari View Post
I'm pretty sure that 0.99 SSIM means half the variation from the source as 0.98 SSIM.
Be precise with your terms then. Distortion is not quality; quality is one of those subjective things which can only be measured by human eyes (as we don't yet have a complete mathematical model for them, only approximations). Unless you define and adhere to an alternate definition of quality, the assumption is human visual quality.

I'd like to try to find a threshold of visual detection of distortion. I wonder how hard such a test would be to create.
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order.
foxyshadis is offline   Reply With Quote
Old 25th April 2007, 04:12   #17  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,689
Quote:
Originally Posted by foxyshadis View Post
Be precise with your terms then. Distortion is not quality; quality is one of those subjective things which can only be measured by human eyes (as we don't yet have a complete mathematical model for them, only approximations). Unless you define and adhere to an alternate definition of quality, the assumption is human visual quality.

I'd like to try to find a threshold of visual detection of distortion. I wonder how hard such a test would be to create.
Well that is the goal of most of these measures. SSIM tries to be more accurate in relation to human perception than PSNR and the like. Obviously its not perfect, but its one of the best out there.
Dark Shikari is offline   Reply With Quote
Old 25th April 2007, 06:57   #18  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,416
http://www.ece.northwestern.edu/~pap...oks_hvei06.pdf (No implementations are currently available.)

I think the source is naturally slanted to h.264 now that I looked at it (mainly because inloop already had its way with it, apparently), and I question the meaning of distortion against an already distorted source. But of course, put up or shut up, so I'm looking for a better source; currently considering HD elephant's dream and some raw game captures.
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order.
foxyshadis is offline   Reply With Quote
Old 25th April 2007, 07:54   #19  |  Link
akupenguin
x264 developer
 
akupenguin's Avatar
 
Join Date: Sep 2004
Posts: 2,393
Quote:
Originally Posted by Dark Shikari
The source video used for these tests can be found here, compressed in high-bitrate H.264 format in an MKV container.
That does mean "I ran the test on an uncompressed source, here's the compressed version just so you can see what it looks like", right? Because if the source was already compressed with H.264, that easily explains H.264's advantage.
akupenguin is offline   Reply With Quote
Old 25th April 2007, 12:41   #20  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,689
Quote:
Originally Posted by akupenguin View Post
That does mean "I ran the test on an uncompressed source, here's the compressed version just so you can see what it looks like", right? Because if the source was already compressed with H.264, that easily explains H.264's advantage.
Just curious, why would that increase H.264's lead though? Would it be because H.264 has a unique pattern of artifacts that are compressed best by itself?
Dark Shikari is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 10:03.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.