Analysis of Sebastians Multi-Codec Listening Test

[AMTuring]

I believe that the recent Sebastian Listening test at 128 kbps (Sebastian (http://www.maresweb.de/listening-tests/mf-128-1/results.htm)) yields more conclusions that the analysis presented so far.

First a Tukey HSD (Honestly Significant Difference) test applied to the overall results only indicates that Shine mp3 encoder is worse than all the other encoders tested. However there is a problem with applying the Tukey HSD to these data.

The first problem is that the variance or MSE (Mean Squared Error) of each of the codecs is not the same. If there are significant differences in the MSE then another approach is indicated. There are two alternatives. Use the Games-Howell significance test or change the codecs being compared.

The Games-Howell test is no help in this case. It produces a less strong conclusion. Tukey is stronger because it assumes all of the variances are equal and this uses more data in the averages used for comparison.

The correct answer in this case is obvious in the following table.

http://amturing.batcave.net/images/Sebastian_Summary.gif

The problem is indicated by the box. Shine's variance is an order of magnitude higher than the other encoders. It should not be included in the Tukey analysis. The anchor is literally weighing the other encoders down with its high variance.

If the Shine data are dropped and the remaining encoders subjected to a Tukey HSD test, then we get the following result.

http://amturing.batcave.net/images/Sebastian_tukey1.gif

So both Ogg Vorbis (AoTuv) and iTunes (new VBR mode) are significantly better than LAME mp3.

There is another problem with the application of the Tukey HSD to the summary results of the Sebastian Listening Test. Tukey also assumes that the mean values of each experiment (song clips in this case) is represented by a single mean.

It is well known that some songs are more difficult to encode than others, and they result in lower quality encoded files regardless of the encoder used. So the assumption of equal means amongst experiments is violated.

In the case of individual songs where the participants represent the different experiments the assumption of equal means is more reasonable and the Tukey test is arguably appropriate for those comparisons. It is only the summary result where the Tukey test is inappropriate.

A means to get around this problem that I used in my earlier correlation tests is to compare residual scores (i.e. Subtract the mean of each song from each score). A better approach that I used on these data is to assign rank scores to each encoder within a song clip. This just about guarantees equal means and variances between songs.

The rank score I calculated is based on how many encoded clips each clip is better than. If it is the lowest score within a clip the score is zero. If it is better than one and only one other clip the score is one, and so on.

The rank scores are shown in the following table.

http://amturing.batcave.net/images/Sebastian_ranks.gif

One again the Shine encoder variance is not comparable to the other encoders. In this case it is zero, since Shine was always worse than all the other encoders. Shine should be deleted from the Tukey analysis in this case in order to keep from overestimating the HSD.

The following table shows the Tukey HSD applied to the ranks.

http://amturing.batcave.net/images/Sebastian_tukey2.gif

Ogg Vorbis stands out as a clear winner in this comparison. However, it is not significantly better than the iTunes encoder.

In a future post I will show the results of my correlation based approach in calibrating the technique to this Listening test.

Notes: OpenOffice.org spread sheets were used for this analysis. I did not delete the Nero codec from the comparison (as discussed here Nero (http://www.maresweb.de/listening-tests/mf-128-1/miscellaneous/nero.txt)). As a listening test it is still valid and significant. However, there are some issues with Nero and other encoders that I will discuss when I post my correlation posts.

[ff123]

As I believe I have mentioned in a PM to you before, if you want to include all the correlations in the data set into the analysis, perform a bootstrap resampling analysis:

http://ff123.net/bootstrap/

The results will be a table of p-values, adjusted for the fact that multiple comparisons are being made, and in which each comparison has a separate number to indicate significance (rather than one number applied to all comparisons).

The downside, as I indicated, is that the easily digested graphs cannot be constructed with this method.

You are welcome to try the bootrap resampling method for this test, but I doubt that you will get 'new' results from the data.

ff123

[Gabriel_Bouvigne]

Something is puzzling me: how can you attribute ranks to statistically tied contenders?
As an example, let's have a look at the first sample: In the test, Vorbis and WmaPro are statistically tied. Are you sure that in this case you can really attribute a different rank to both?

[AMTuring]

@ Gabriel, I am not claiming that each rank is statistically significant. The ranks are just what they are. I could have just as easly assigned 1st place, 2nd etc. The point is that Ogg Vorbis came out at the high end of the ranking so many times and that is statistically significant.

I assign no significance to the individual "ranks" in the table. Only to the statistical analysis of them.

Again, the Tukey test is designed for populations with equal variance and where each experiment is supposed to have the same mean and variance. The variance between songs present in the overall comparison is masking the significance in the test. The only thing that is supposed to vary is the mean of each codec.

@ ff123, Sorry I have never received a PM from you on doom9. I will look at your link.

[AMTuring]

As I believe I have mentioned in a PM to you before, if you want to include all the correlations in the data set into the analysis, perform a bootstrap resampling analysis:

http://ff123.net/bootstrap/

The results will be a table of p-values, adjusted for the fact that multiple comparisons are being made, and in which each comparison has a separate number to indicate significance (rather than one number applied to all comparisons).

The downside, as I indicated, is that the easily digested graphs cannot be constructed with this method.

You are welcome to try the bootrap resampling method for this test, but I doubt that you will get 'new' results from the data.

ff123

Here are the results from your bootstrap program.

http://amturing.batcave.net/images/bootstrap.gif

There is very little documentation for the program, but I assume the * indicates a significant difference. If that is correct then it indicates that Ogg Vorbis is better than Nero and LAME. Also iTunes is better than LAME.

This is similar to the results of a Tukey test after dropping Shine from the analysis. It does not appear to address the unequal means between audio clips problem.

[ff123]

Re unequal means:

Each different audio clip is treated as a "block," to incorporate the correlations of each particular sample (for example some audio clips are rated higher or lower than the average). It is similar to the way a "blocked" ANOVA analysis would be performed, except using resampling techniques.

The bootstrap resampling program is about as sensitive a method as you're going to get without "cheating" and disregarding p-value adjustment for multiple comparisons. There's basically no magical method that can create significant differences where there are none.

ff123

[ff123]

Actually, in theory you could improve the power of the bootstrap resampling method even further by doing a "restricted p-value adjustment" instead of the "free step-down method" used in this program (both are refinements over a crude Bonferonni). I say in theory because in practice I'm not sure what that would really mean except in pretty weird-looking data sets. In any case I never got around to implementing that last bit of it, but the professional statistical packages (SAS) include it.

For references on the bootstrap program, I recommend "Resampling-Based Multiple Testing, Examples and Methods for p-value adjustment" by Peter H. Westfall and S. Stanley Young

ff123

[AMTuring]

Thanks for the education. I noticed the two papers refrence in the code.

However, you did not confirm what the results of bootstrap mean.

[ff123]

Thanks for the education. I noticed the two papers refrence in the code.

However, you did not confirm what the results of bootstrap mean.

Default analysis for the bootstrap program (asterisks show significant differences) are for a blocked t-test of ratings (vs. rankings) with 10,000 resamples, using free-step down to adjust the p-values for multiple comparisons. A t-test compares the means.

ff123

[AMTuring]

Correlation Analysis of Sebastian's Listening Test

Background

I have been working since 2004 on a method to measure the fidelity of audio codecs with a computer program. Readers of this note need to be aware that this is a controversial concept. Over at Hydrogenaudio (”http://www.hydrogenaudio.com”) it is an article of faith written into their Terms of Service #8, mathematics can not be used to judge audio quality.

The HA belief is not as illogical as it sounds to the uninitiated. There are some aspect of the physiology of hearing that limit what is perceivable by a human being. A computer program or other hardware could detect differences that would be invisible to the human listener. I will discuss these aspects further in a subsequent post.

However, it should be noted that the physiology of hearing indicates that a mathematical method could falsely identify a clip as poor while a human listener would see a good correlation. However, physiology does not imply that a good mathematical measurement is false.

In addition, it I believe it is possible to develop a mathematical description of human hearing that is capable of taking physiology into account. I have developed a physiology based method for measuring the correlation between audio clips that I will present at a later time. This note is essentially background to that method, and represents the ultimate development of my original, simple approach to the problem.

Comparison with the Listening Test

I previously published a paper (http://amturing.batcave.net/papers/calibration.pdf) in an attempt to calibrate my correlation based approach. This calibration was not very satisfying because it was too late to be able to duplicate the codec versions exactly. With the recent Sebastian Multi-Format 128 kbps Listening Test (http://www.maresweb.de/listening-tests/mf-128-1/results.htm) I was able to download the listening test files and exactly duplicate the codec versions that I as unable to do for Roberto Amorim's Listening Test (http://www.rjamorim.com/test/multiformat128/results.html).

This has been invaluable for the correlation work. There is more reliable data available from Sebastian's test to improve the accuracy of the results.

First I ran Pearson correlation coefficients on several moments derived from the running correlation method described in the papers available here (http://amturing.batcave.net/papers.html). The Pearson coefficient measures if there is a linear relationship between two sets of data. 100% indicates a perfect match and low magnitudes indicate a poor similarity. A coefficient of -100% would indicate a linear relationship with opposite sign (i.e. Low value of one measurement map to high values of the second measurement).

The Pearson correlation was calculated for the mean, mean squared error, P95, P99, P99 mean squared error, P99.9, minimum, and smoothed minimum (0.5 seconds) normalized cross correlation coefficient. These various moments were correlated with the average score for each audio clip. The data used was for all 6 codecs and all 18 audio clips. This yielded 108 comparisons.

Adding filtering to the Method

First, I tried to incorporate some of the human hearing range by band limiting the clips before comparison with a low pass filter. SOX (http://sox.sourceforge.net/) was used to apply a lowpass filter to both the codec wav file and the original wav file before correlation tests. Here is the P99 vs. high cut frequency graph:

http://amturing.batcave.net/images/PvsF.gif

As you can see the correlation is maximum and pretty flat between 16 and 17 kHz. Compared to the original unfiltered algorithm this improves the correlation from 54.25% to 55.125%. Not quite a 1% increase over the previous method.

A 55% correlation is not great, but there is probably a lot of noise in the listening tests. Different listener's are involved which means that they will have different sensitivities and different ratings. Also a lot of the scores are composed of 5's which means that the listener could not ABX the difference between the original and the compressed clip.

A much better correlation is obtained between the average P99 from correlation and the average scores for all the clips (6 codecs). The correlation with a 17 kHz high cut is 97.18%. However, this correlation is not as impressive as it sounds. The correlation coefficient is dominated by the spread between Shine mp3 encoder (the anchor) and the other codes.

I chose to use a 17 kHz low pass filter for the following tuning. This is more stringent in requiring good fidelity up to the higher end of the good correlation zone (16-17 kHz).

Optimum Correlation Window Length

I varied the correlation window length from the original version (20 ms) and plotted the correlation of the P99 moment vs. correlation window length. The average of all 18 clips correlation results are shown here:

http://amturing.batcave.net/images/PvsT.gif

As you can see, 10 ms windows produce a better correlation than the 20 ms windows I originally used. However, the improvement is less than 0.2 %.

The correlation of the individual codec averages results in a P99 of 58.04%. Where I work that result would be OK, but not good. In other words, there is a relationship between the two measurements (correlation and listening tests), however it is not strong.

Moreover, as my previous post (see above) on Sebastian's test indicated, there is some significance the fact that Ogg Vorbis and iTunes consistently posted higher scores in the listening tests. In other words, the fact that Ogg Vorbis got the highest score on an individual clip is not significant (as indicated by the Tukey HSD). However, the fact that it was 1st or 2nd most of the time is significant.

This correlation technique consistently rated WMA and Nero in the top spots. There are two reason for that. One is the borrowing of bitrate that codecs can use. There is a bit buffer that most variable bit rate (VBR) codecs use to even out the quality of the encode. The intent was to allow simple sections of the audio to allow the codec to store up potential higher bitrates for later more difficult parts of the audio. This is sort of like a savings account.

As was discover during Sebastian's listening test, some codecs front load the beginning of an encode with high bitrates. This is like mortgaging your future. The results are good in the beginning, but the dept must be paid later on. Nero was disqualified from the final comparisons for this reason (download and read Nero (”http://www.maresweb.de/listening-tests/mf-128-1/miscellaneous/nero.txt”)).

Note that the discussion about Nero does not prevent me from using the results for calibration. The listening tests are still an accurate representation of what the testers heard. However, it does indicate that the results can not be used as an indication of the performance of Nero AAC in normal use. In my mind it also throws all tests involving short (< 30 second) clips into question. Perhaps a different protocal should be adobpted for listeing tests (i.e. encode longer clips and pull a section out of the middle or end instead of the beginning).

For example, in tests I will present later, I found that LAME –vbr-new is better than LAME –vbr-old when judged over a longer encoding. However, if I use the short clips normally used in listening tests, I get the opposite result (vbr-old is better than vbr-new).

The conclusion is that LAME vbr-new is better than vbr-old.

Finally, you may have noticed I used P99 in the above tunings. This is because the Pearson correlation is slightly better for P99 than for P95 (used in previous studies). Since this calibration is better controlled than the Amorim calibration, I believe the new result. This is shown in the following table:

http://amturing.batcave.net/images/Pearson.gif

Conclusion

Although there is a similarity between the running correlation measure of fidelity and the Sebastian listening test results, the result is still not satisfactory. I do not believe this technique can replace the human listener.

I have succeeded in reproducing the Sebastian results in detail using a different physiology based approach and modifying the test procedure (quit using short clips). I will present these results in another post. I am going out of town on a business trip and will not get to it until I get back. Also don't think I am ignoring you if I don't respond in the next week. I am just off the grid.

You can read more about the correlation method and get the software at my website (AMTuring (”http://amturing.batcave.net”)).