Public AAC Listening Test @ ~96 kbps [July 2011]. Nero, Apple, Fraunhofer and CT AAC

[IgorC]

Hi, Guys,

There is a new public test.

http://listening-tests.hydrogenaudio.org/igorc/aac-96-a/results.html
Summary: Apple won, FhG is the second, Coding Technologies is the third and Nero is the last

http://listening-tests.hydrogenaudio.org/igorc/aac-96-a/nonblocked_means.png


The following codecs are presented in this test:
Nero 1.5.4
Apple QuickTime 7.6.9 true VBR
Apple QuickTime 7.6.9 constrained VBR
Fraunhofer (Winamp 5.62)
Coding Technologies (Winamp 5.61)
ffmpeg's AAC (low anchor)

Everybody is welcome to participate.

Public AAC Listening Test @ ~96 kbps [July 2011] (http://listening-tests.hydrogenaudio.org/igorc/aac-96-a/)

[IgorC]

The test is extended until August 5.

We need more participants.

[IgorC]

The test is finished.

Summary: Apple won, FhG is the second, Coding Technologies is the third and Nero is the last

CVBR - Apple AAC (constrained VBR)
TVBR - Apple AAC (true VBR)
CT - Coding Technologies AAC (Winamp 5.61)
Fhg - Fraunhofer AAC (Winamp 5.62)

Bootstrap analysis:

bootstrap.py v1.0 2011-02-03
Copyright (C) 2011 Gian-Carlo Pascutto
License Affero GPL version 3 or later

Reading from: results_AAC_2011.txt
Read 6 treatments, 280 samples => 15 comparisons
Means:
Nero CVBR TVBR FhG CT low_anchor
3.698 4.391 4.342 4.253 4.039 1.545

Unadjusted p-values:
CVBR TVBR FhG CT low_anchor
Nero 0.000* 0.000* 0.000* 0.000* 0.000*
CVBR - 0.128 0.002* 0.000* 0.000*
TVBR - - 0.059 0.000* 0.000*
FhG - - - 0.000* 0.000*
CT - - - - 0.000*

CVBR is better than Nero (p=0.000)
TVBR is better than Nero (p=0.000)
FhG is better than Nero (p=0.000)
FhG is worse than CVBR (p=0.002)
CT is better than Nero (p=0.000)
CT is worse than CVBR (p=0.000)
CT is worse than TVBR (p=0.000)
CT is worse than FhG (p=0.000)
low_anchor is worse than Nero (p=0.000)
low_anchor is worse than CVBR (p=0.000)
low_anchor is worse than TVBR (p=0.000)
low_anchor is worse than FhG (p=0.000)
low_anchor is worse than CT (p=0.000)

p-values adjusted for multiple comparison:
CVBR TVBR FhG CT low_anchor
Nero 0.000* 0.000* 0.000* 0.000* 0.000*
CVBR - 0.130 0.005* 0.000* 0.000*
TVBR - - 0.107 0.000* 0.000*
FhG - - - 0.000* 0.000*
CT - - - - 0.000*

CVBR is better than Nero (p=0.000)
TVBR is better than Nero (p=0.000)
FhG is better than Nero (p=0.000)
FhG is worse than CVBR (p=0.005)
CT is better than Nero (p=0.000)
CT is worse than CVBR (p=0.000)
CT is worse than TVBR (p=0.000)
CT is worse than FhG (p=0.000)
low_anchor is worse than Nero (p=0.000)
low_anchor is worse than CVBR (p=0.000)
low_anchor is worse than TVBR (p=0.000)
low_anchor is worse than FhG (p=0.000)
low_anchor is worse than CT (p=0.000)

[nibus]

I'm surprised Nero placed last.

[tebasuna51]

Seems CVBR - Apple AAC (constrained VBR) is better than TVBR - Apple AAC (true VBR)
There are some explain about that?

Also, please, how we can test Apple AAC without install QuickTime 7.6.9?

There are Public Listening Test with 5.1 at high bitrates?
Maybe more interesting in this forum than stereo low bitrates.

[nurbs]

TVBR and CVBR are statistically tied so you can't really call one of them better. Looking at the bitrates (http://www.hydrogenaudio.org/forums/index.php?showtopic=90403&view=findpost&p=767123) though they are a few percent higher for the CVBR encodes on average. There is on case where CVBR gave the file a 40% higher bitrate than TVBT. Maybe that has something to do with it, but I didn't look for the ratings of single samples so it's just speculation.

[mindbomb]

how did you encode vbr with quicktime?
did you uses qtaacenc?

[IgorC]

There are Public Listening Test with 5.1 at high bitrates?

5.1 test will be more difficult. Higher bitrates will be also difficult.
5.1 + high bitrates will fail completely.

Reasons:
1. 5.1 will fail because there are not enough trained listeners with 5.1 systems.

2. 96 kbps stereo test was already hard to perform. The quality was actually high for some AAC encoders.


Maybe more interesting in this forum than stereo low bitrates.

Do you have any data what amount of people have 5.1 or 2.0?

how did you encode vbr with quicktime?
did you uses qtaacenc?
qtaacenc + foobar

[mindbomb]

I have a theory for why CVBR is better than TVBR.
Apple's algorithm must be choosing a bitrate that is too low to maintain transparency in the case of TVBR, but is saved by the constraints in CVBR.

Probably not an issue with higher bitrates.

[tebasuna51]

...
1. 5.1 will fail because there are not enough trained listeners with 5.1 systems.
Do you have any data what amount of people have 5.1 or 2.0?

Well, I don't know.
But here we write many times about movie audio tracks 5.1 (even 7.1)

2. 96 kbps stereo test was already hard to perform. The quality was actually high for some AAC encoders.

Then we can assume than for bitrates > 48 Kb/s per channel all encoders are good and is dificult listen differences.

[IgorC]

Then we can assume than for bitrates > 48 Kb/s per channel all encoders are good and is dificult listen differences.


2. 96 kbps stereo test was already hard to perform. The quality was actually high for some AAC encoders.


It's more correct to say:
Not all but some.

[shon3i]

1. 5.1 will fail because there are not enough trained listeners with 5.1 systems.I think testing 5.1 is more testing all six mono channels separately against original channels. I think is impossible to ABX all channels in same time, or downmixed. High bitrate AAC is not that high, AAC usually requires 320kbps for decent quality while HE-AAC can go to 160kbps, both bitrates are easily abxable. And it's very interest how encoder spread bits bettween chanels, so usually LFE can be very lowpassed.

I'm surprised Nero placed last. I am not because they optimised encoder for low bitrate, which blow everything under 64kbps.

[IgorC]

I think testing 5.1 is more testing all six mono channels separately against original channels.
Completely false.



High bitrate AAC is not that high, AAC usually requires 320kbps
for decent quality while HE-AAC can go to 160kbps, both bitrates are easily abxable.

Wrong.

LC-AAC 5.1 320 kbps is equivalent to stereo at 128 kbps.
High quality AAC encoders have performed very good already at 96-100 kbps for stereo (equivalent for 5.1 is 240-256 kbps). Also half of AAC encoders did excellent on male English speech (Sample 18). The same is valid for female English speech (Sample 06).
Then it's correct to say that modern high quality AAC encoders perform very well on speech at 96-100 kbps for stereo.
See the results for particular samples http://listening-tests.hydrogenaudio.org/igorc/aac-96-a/samples.html

HE-AAC 5.1 160 kbps is equivalent to stereo at 64 kbps. Previous test has shown that listeners with speakers have big troubles to listen the difference between HE-AAC 64 kbps and lossless because it's much harder to hear artifacts with speakers instead of headphones.
http://listening-tests.hydrogenaudio.org/igorc/Public%20Multiformat%20Listening%20Test%20@%2064kbps.htm

So at least for Apple Quick Time encoder it's not valid to say that HE-AAC 5.1 160 kbps or LC-AAC 5.1 320 kbps are easy to spot from original.

Placebo effect or excessively killer samples (not frequent in real scenario) is any story.

[IgorC]

There is some information about performance of HE-AAC 5.1 160 kbps

http://mpeg.chiariglione.org/working_documents.htm#MPEG-D
http://mpeg.chiariglione.org/working_documents/mpeg-d/sac/VT-report.zip

Also You can find some 5.1 AAC tests in google.

P.S. Some extra general information to have an idea what to expect from HE/LC-AAC at 160-320 kbps.
http://www.jeroenbreebaart.com/papers/aes/aes123.pdf
http://s2.ipicture.ru/uploads/20110828/722BizUS.png

[shon3i]

Previous test has shown that listeners with speakers have big troubles to listen the difference between HE-AAC 64 kbps and lossless because it's much harder to hear artifacts with speakers instead of headphones.That's why i think testing mono channels instead whole 5.1 is more optimal to find artifacts or to decide which encoder more respect surround channels and LFE. For example i had situation in 2007-2008 where CT CBR @ 160kbps HE-AAC give's me less artifact than Nero @ 320kbps LC-AAC, but this is impossible to find without listening all channels separately with headphones.

Btw thanks for explanation and graphs, it just confirm my early observations.

[datauser]

Really surprised too that Nero came last when in many subjective playback tests done with Nero format, it is always rated top! Interesting challenging stuff. Thanks.

[IgorC]

During last few years Nero has made its fame with high quality of HE-AAC encoder in past. This time we've tested LC-AAC encoders.

You can see all previous public tests here http://listeningtests.t35.me/

[nibus]

IgorC, for 5.1 Multichannel AAC, in general what is perceived as a transparent bitrate? I usually go for around 500-600 kbps but maybe that's overkill?

[IgorC]

Well, it depends of encoder, settings and source (lossless or lossy).

LC-AAC 5.1 320 kbps is equivalent to stereo at 128 kbps.

There is no 5.1 public listening test. Though if one particular AAC encoder is transparent at 128 kbps then it's a good hint that the same encoder will be transparent at 320 kbps for 5.1. LC-AAC uses classic joint and mid-side algorithms for both stereo and 5.1.

Quiktime AAC encoder has met the requirements of transparency at 128 kbps (stereo). http://listeningtests.t35.me/mf-128-1/results.htm

P.S. Also soundtracks are more easy to compress because mostly have a speech and some occasional nature sounds and music.
FhG, QuickTime and CT did excellent (score higher than 4.5!) on speech samples (Sample 06 and 18) at 96 kbps (stereo) http://listening-tests.hydrogenaudio.org/igorc/aac-96-a/samples.html

So if You are sure that some particular lossless soundtrack has mostly speech and nothing else then you can encode it with excellent quality at ~256 kbps multichannel.

Of course it's all valid if AAC encoder is high quality, high quality settings VBR and soundtrack is lossless.

[sneaker_ger]

IgorC, could you comment on this (http://forum.doom9.org/showthread.php?p=1516881#post1516881)? It seems you know a lot about AAC and the common encoders. Is there no coupling between front and back channels?

[IgorC]

It seems you know a lot about AAC and the common encoders.
Not more than experienced user should know. It's better always ask to developers.


Is there no coupling between front and back channels?

I'm not sure about Nero encoder. AFAIK LC-AAC's coupling is pairwise at those bitrate. You can visit http://www.hydrogenaudio.org/forums/index.php? to do search and/or make question.

[sneaker_ger]

I see. Guess it's time to recover my hydrogenaudio login data.

[Anakunda]

Greetings, would the test show diffeent order if performed on higher bitrate, ie. let's say at ~160kbps ? Especially if the CVBR mode is still better over TVBR at bitrates like this.

[poisondeathray]

I was also wondering if anybody was aware of any tests done at higher , and very low ranges?

[nibus]

I was also wondering if anybody was aware of any tests done at higher , and very low ranges?

There are 3 here that test a few AAC codecs but they're pretty old:
http://listening-tests.hydrogenaudio.org/sebastian/

Here is one at 64kbps from March:
http://listening-tests.hydrogenaudio.org/igorc/

Maybe some new listening tests could be organized that would include higher bitrates.