80kbps quality comparison of WMA9, Ogg Vorbis, AAC and MP3

[bond]

here the results from guruboolez's test at ~80kbps:

http://foobar2000.net/divers/tests/2005.07/80/80TEST_PLOTS_06.png

very interesting results imho:

- aotuv's Ogg Vorbis clearly stumbed everything to the ground (i am really thinking about dropping encoding to AAC and going with Vorbis)
- itunes's LC-AAC provided slightly better quality than nero's HE-AAC (what is nero doing???)
- microsoft's wma9 provided slightly less or the same quality than lame mp3 on classical samples and was slightly better on various music styles (anyone wanting to question the claim that wma9 provides the same quality as mp3 at 128kbps at 64kbps!?)

read the details about the whole comparison here (http://www.hydrogenaudio.org/forums/index.php?showtopic=35438)

[CiNcH]

Guess that at 80 kbps AAC can code most of the frequency spectrum so SBR does not come into play too much. Seems that Apple's basic AAC implementation is just better than Nero's.
And that Vorbis is pretty complex and efficient is no secret either.

[SeeMoreDigital]

I find it hard to believe AAC-LC sounds superior to AAC-HE at 80Kbps.... And "what" no MP3Pro ;)

I guess I better blow the dust off my headphones again!


Cheers

[bond]

it might be very well possible that a good lc-aac encoder sounds better than a worse lc-aac codec + sbr and guruboolez knows what he does when doing such a comparison

[stephanV]

It would probably be appropriate to add "YMMV" to the conclusions of the test. :)

[SeeMoreDigital]

Well.... I generated some 80Kbps tests of my own and even I was able to hear the differences between iTunes v4.9.0.17 AAC-LC and Nero v3.2.0.20 AAC-LC ;)

The differences between iTunes v4.9.0.17 AAC-LC and Nero v3.2.0.20 AAC-HE were less apparent but never-the-less I have to say I prefered Apples!

It kinda makes you wonder what would happen if Apple offered an AAC-HE encoder :D

[CiNcH]

It kinda makes you wonder what would happen if Apple offered an AAC-HE encoder

I believe not too much at 80 kbps.

[Rockaria]

but only for low bitrates around 20-64 kbps for AAC and 32-96 kbps for mp3PROin the beginning of this page. (http://www.audiocoding.com/modules/wiki/index.php?page=SBR)

[Slogra]

In my experience AAC-HE is really bad too. It sounds ok on speakers, but when i listen with my headphones it sounds really very bad. I guess it still need heavy tuning, or maybe it's bad no matter what you do.

BTW here are the aac, mp3, mpc, ogg @175kbps testresults:
http://www.hydrogenaudio.org/forums/index.php?showtopic=36465&hl=

Thanks to guruboolez

[SeeMoreDigital]

I believe not too much at 80 kbps.For encoding 2Ch content... maybe.

But then one would have to ask why Nero (within Recode2) offers 2Ch AAC-HE encoding up to 128Kbps and 6Ch AAC-HE encoding up to 448Kbps!


Cheers

[Rockaria]

The SBR best suiting in the range 20kbs~64kbs from the audiocoding.com is ambiguous whether it is on stereo or not.
But the projection of 64kbps based on the image attached(stereo, codingtechnologies.com (http://www.codingtechnologies.com/products/aacPlus.htm) ) implies they becomes similar(meet together).
So I guess the 20-64kbps range is for stereo.
http://www.codingtechnologies.com/products/assets/3GPP_mushra_18_48kbps_2.gif

Also I tested the neroAAC He profile, the upper limit for the HE profile was 96kbps.

[Doobie]

MP3 80 better than MP3 128 (genre:classical), AAC-LC better than AAC-HE, Vorbis way better than everything (even without SBR)... This may be correct and reasons have been given for it. But, it strongly reminds of of those Pepsi vs. Diet Pepsi blind taste tests that you can find on the internet that show that Diet Pepsi, as bodiless and funky as it may be with that bitter aftertaste, beat cool, refreshing Pepsi. Likewise, you can have a taste test with children, God's fresh Orange Juice vs. putrid reconstituted juice, and 4 of 5 kids will say the reconstituted junk tastes better.

Obviously, most people are brain damaged. Or, at least in the case of people taking the Pepsi and orange juice tests, their pallet have been damaged by abusive consumption of diet drinks and reconstitued juices. And, their idea of what is right and been turned upside down.

This applies to audio, too. Louder volumes, depressed midrange, LP hiss, etc. tend to make some people think the sound is better, when it's worse.

Why can't someone design a machine that can hear? There can even be software to model the perception of the human hear, and have this machine objectively measure codec quality?

[bond]

Doobie , you obviously didnt read the thread i pointed to

the test made was a double blind listening test, meaning the tester doesnt know what sample was encoded with what encoder and the tester also doesnt know what is the encoded sample and what is the source

the test is always done "encode vs. source" and not "encode A vs. encode B", as you suggested with your post

[KpeX]

Yea guruboolez's personal tests can be relied on quite well... if you search his history he's proven to have one of the best sets of ears out there and he executes his tests objectively and precisely. (i am really thinking about dropping encoding to AAC and going with Vorbis)Ditto, after this test and guru's high bitrate test, vorbis is looking good.

[Nil Einne]

Does anyone know if aoTuV Vorbis is optimised for 48k as well as 44.1k? As far as I can tell, these tests were done with 44.1k samples. So if aoTuV is optimised for 44.1k only/mostly then it's a bit irrelevant for those of use who are backuping DVDs or recompressing digital streams etc that are 48k. (Of course, multi-channel is a whole different kettle of fish again but that's a totally different issue).

[stephanV]

Since most audio encoders are tested on CD material, I wouldn't expect them to be heavily tuned for 48 kHz. Also not that music is tested here, not voice or explosions or cars or whatever, which would be more relevant for DVDs too.

On the other hand, somehow it would feel unnatural if results would be completely opposite on that kind of material.

[Nil Einne]

Actually as discussed here:

http://forum.doom9.org/showthread.php?t=99071

The HE-AACPlus2 is HEAVILY tuned for 48k. So much so that it just doesn't work with 44.1k. This is one of the reasons I raised such an issue. Clearly, it is easily possible Vorbis will perform poorly on 48k if it is not optomised/tuned, especially in comparison to HE-AACPlus2.

Also, voice is frequently tested, not normal speaking voice of course (well except for rap songs) but voice is still tested in music. However you're right that explosions, cars etc are not tested much but the music and talking in the movies are probably one of the difficult aspects so there is at least some applicability.

[KpeX]

Does anyone know if aoTuV Vorbis is optimised for 48k as well as 44.1k? As far as I can tell, these tests were done with 44.1k samples. So if aoTuV is optimised for 44.1k only/mostly then it's a bit irrelevant for those of use who are backuping DVDs or recompressing digital streams etc that are 48k. (Of course, multi-channel is a whole different kettle of fish again but that's a totally different issue).
I don't think anyone's ever objectively proven that whether an encoder is 'tuned for 44.1 khz' makes a significant difference with 48khz. Multichannel doesn't really need additional tuning either, just channel coupling, which vorbis lacks.

[Nil Einne]

So you're saying that dimzon is wrong in saying that HE-AACplusv2 won't work at 44.1k because it is tuned to 48k. Or are you saying that things optimised/tuned for 48k won't work at 44.1k but things optimised/tuned for 44.1k will work perfectly at 48k?

In any case, I haven't yet seen any evidence suggesting tuning doesn't make a significant difference. I admit, I have very limited understanding of how encoding works and I have always expected it would not matter but since some people seem to think it does matter therefore I still stick with my original point. Tests at 44.1k have little relevance to tests at 48k IMHO until someone, perhaps the coder of aoTuv can say with a degree of certainty that either aoTuV is well tuned for 48k or that it will not make any difference based on their understanding of how it works.

As for multichannel, my point was not whether it needed tuning or not but that tests comparing stereo have only limited relevance to tests comparing multichannel as additional features are needed for multichannel and therefore, it's a whole different issue which I don't want to get in to (since I don't care at the moment and it's perhaps a bit too offtopic)

[KpeX]

So you're saying that dimzon is wrong in saying that HE-AACplusv2 won't work at 44.1k because it is tuned to 48k. Or are you saying that things optimised/tuned for 48k won't work at 44.1k but things optimised/tuned for 44.1k will work perfectly at 48k? Nope, what was being referred to was an AAC encoder optimized for 48khz, as opposed to some encoders (lame for example) that have been tuned at 44.1 khz. Any encoder will 'work' at any sample rate it accepts, which is usually anything you can come up with. My point was just that I've never seen a test that proves it makes a difference. It would be tough to do because there aren't many samples available in both 44.1 and 48 khz (without resampling the original). One way to test would be (if you have access to high end recording equipment) to record a sample at a high sampling rate that is an even multiple of both 44.1 and 48 khz and then downsample to each frequency, encode, and ABX.

More simply, if your 48khz 5.1 channel encoding reaches your desired level of quality (transparency, etc.) which can be verified using ABX software, why worry about what sampling rate an encoder is 'tuned' for.

[JohnV]

here the results from guruboolez's test at ~80kbps:
- itunes's LC-AAC provided slightly better quality than nero's HE-AAC (what is nero doing???)[/URL]
Well, regardless that there's no statistically significant difference, you are right in one thing: Vorbis is imo pretty much at its best regarding effieciency at 80kbps bitrate level, so reaching that should be our goal.
To answer to your question, we are doing a lot of progress at the moment on all areas.
Just not implemented yet in release versions, but don't worry.. :cool:
It's a lots of work since pretty much all of our aac technology is under heavy updating at the moment: sbr, parametric stereo, lc-core..

[Rockaria]

Actually as discussed here:

http://forum.doom9.org/showthread.php?t=99071

The HE-AACPlus2 is HEAVILY tuned for 48k. So much so that it just doesn't work with 44.1k. If anybody gave you this STRONG impression, he has the responsibility to prove the NATURE of the codec. The simple fact that the encoder does not support the 44.1k never means it is tuned(optimized) for 48k.

The major factor for prefferring the 48k sampling rate to 44k in the PC Audio is that the system mixer is mostly(or usually) setup for 48k resampling other sampling rate input sources before feeding to the mixer, where the rounding/cutoff might occur.

[stephanV]

Actually as discussed here:

http://forum.doom9.org/showthread.php?t=99071

The HE-AACPlus2 is HEAVILY tuned for 48k. So much so that it just doesn't work with 44.1k. This is one of the reasons I raised such an issue. Clearly, it is easily possible Vorbis will perform poorly on 48k if it is not optomised/tuned, especially in comparison to HE-AACPlus2.

The reason why it doesn't support 44.1 kHz, is not because it is tuned for 48 kHz, but because no one has written support for it. At first glance I can't even seem to find information that that encoder is tuned on or for anything.

Again, I would be really surprised if a codec that would perform excellent on one sample rate, would perform horrible on another.

[bond]

Nil Einne and all, on hydrogenaudio there is the nice rule that quality statements have to be backed up with double blind listening results, what you are doing is only speculating that vorbis might perform worse at 48khz, still there is no proove of any kind for this. but i would be interested in seeing any

apart from that at 80kbps it might make sense to downsample to 32khz anyways

It's a lots of work since pretty much all of our aac technology is under heavy updating at the moment: sbr, parametric stereo, lc-core..great! keep it coming :)

[Caroliano]

apart from that at 80kbps it might make sense to downsample to 32khz anyways
80kbps is too soon to downsample. As you can see in the comparision, vorbis at 80kbps is almost equal to LAME at 128kbps in "various music" area. You would downsample a 128kbps encode?

I would rather downsample to 32KHz under vorbis q0. 32kbps at 32KHz sounds better than 32kbps (q-2) at 44.1KHz, but at 80kbps downsample is bad IMHO.

[PatchWorKs]

It's a lots of work since pretty much all of our aac technology is under heavy updating at the moment: sbr, parametric stereo, lc-core..
[abitOT] just curiosity: why don't you work on Vorbis instead ? You probably have to pay to release AAC with Nero, so why don't you switch to a patent-free codec ?

I would rather downsample to 32KHz under vorbis q0. 32kbps at 32KHz sounds better than 32kbps (q-2) at 44.1KHz, but at 80kbps downsample is bad IMHO.
Totally agree.

[Rockaria]

80kbps is too soon to downsample..
I would rather downsample to 32KHz under vorbis q0. 32kbps at 32KHz sounds better than 32kbps (q-2) at 44.1KHz, but at 80kbps downsample is bad IMHO.Totally agree.
Yeah, If you have to reencode because of the size, it's better doing it from (or closer to) the original.

It seems to be overlooked here the fact that what is playing eventually is the rebuilt(restored :: decompressed) wav signals in the buffer(queue), not the compressed format directly to the DAC. So I wonder why there comes '32khz sampling rate' which seems nothing to do with rebuilding the the good original wav..
The more you change from the original(through the transcode chain) wav spectrum( sample size(volume), sample rate(density), channels...), you are loosing the fidelity, making it harder to simulate the original spectrum.

[gURuBoOleZ]

It may be worth to note that HE-AAC would probably perform better than what my test is showing. I did a comparison of different HE-AAC encoders at 64 kbps this week-end, and Nero Digital is by far the weakest one - even worse than 48 kbps "aacPlus v.2" with Parametric Stereo.

http://foobar2000.net/divers/tests/2005.09/AACHE/01/plots.png


http://www.hydrogenaudio.org/forums/index.php?showtopic=36868

[SeeMoreDigital]

Thanks for the info gURuBoOleZ,

Would you be able to generate some 6Ch AAC-HE comp tests with these codecs?

Personally speaking, I'd like to think (hope) that AAC-HE will be used mainly in the multi-channel (not 2-channel) arena!

A while ago, bond, myself and Sagittaire discussed 6Ch AAC-HE against 6Ch WMA Pro at just 128Kbps... But we only had Nero's AAC-HE codec on offer at the time... It would be useful to see how well the other codecs AAC-HE codecs compare!


Cheers

[Kurtnoise]

Would you be able to generate some 6Ch AAC-HE comp tests with these codecs?
Actually, only Nero Digital can generate 6ch HE-AAC streams.


In addition, It's much more difficult to perform blind tests with multichannel files. And I assume that Guru doesn't have the raw materials for this.


btw, thanks Guru for these tests. :)

[gURuBoOleZ]

Kurtnoise13 guess is correct: I've only two speakers. By the way I'm used to test files on headphone.

[SeeMoreDigital]

Actually, only Nero Digital can generate 6ch HE-AAC streams. What a shame.... It's not very useful having a choice of one :eek:

With AACPlus v2 and Helix showing significant improvement over Nero Digital in the 2 channel arena.... We are left guessing if similar improvements could be achieved for 6 channel content....


And yes.... thanks Guru for these tests. :)

[bond]

guruboolez also made a comparison with 6 ac3 movie soundtracks which might be very interesting for us doom9 guys:

http://foobar2000.net/divers/tests/2005.09/AACHE/01/plotsx3.png

and his comments:
DVD transcoding : there are too few samples to make any strong conclusions. First comment: notations are higher (for all four encoders) with this group of sample. This could be partially explained by the presence of one mono-encoding which sounded transparent with all encoders (which obtained as consequence 5 points for this sample). Helix & Winamp quality at 64 kbps are really excellent. I’m not fond of DVD ripping but I think I will consider HE-AAC again (I was very disappointed by my previous tests, all made with Nero…). Nero Digital performs less badly than with music encoding, but is still far from all other competitors, including the 48 kbps “aacPlus” encoder. It confirms my previous experience with HE-AAC and DVD ripping: poor. The usual artefacts of Nero are also audible, altering voice as well as music. damn seems we need to get helix to support 5.1! :D

[Rockaria]

damn seems we need to get helix to support 5.1! Actually, Helix has multi channel and loseless codec also with the same file extention rmvb. The problem is it has been impossible so far to know what kind of internal codec it has.
The multichannel music couldn't be demuxed by dtdrive (2ch aac is known to be demuxed).

[Nil Einne]

Some of you seem to be missing the point. I don't know nor am I saying whether Vorbis or any codec will perform poorly at a sample rate (e.g. 48k) it is not tuned for then one it is tuned for (e.g. 44.1k). I do know that people do tuning of a codec to make it perform better with samples that they test and I assume that in some cases, most/all of these samples will be at a certain sample rate.

I also know that certain people (e.g. aoTuV) seem to think that optimisation/tuning does matter for sample rates and certain people seem to think it won't. Both sides seem to understand encoding theory. I personally do not understand encoding theory much at all. Originally I had thought it would make no difference but given how little I understand about encoding theory, my opinion is useless. As far as I can tell, both sides do not have stronger arguments either way nor has either side done much testing to show that a codec def performs worse or the same if it is not 'optimised/tuned' for a certain sample rate. Therefore, I have no reason to go with either side. I am an amateur and only do this kind of things for fun and not very often at that. Ideally perhaps, I should conduct a blind test and a software analysis of every thing I encode to find which is the best codec for my source. But this is not practical for a pro most of the time and def not practical for an amateur. Therefore, my best hope is to read up on the various tests and use this information to work out what is likely to be the best codec I have available for my use (at the time when I want to encode for my source). Since no one is likely to have tested my source and conditions, my best hope is to extrapolate the info I have to my particular source and conditions. When doing this I clearly have to be careful not to extrapolate too far. For example, I could say because Vorbis is the best at 80kbps when I plan to encode at 160kbps I should use Vorbis too. Clearly this is a stupid extrapolation and should not be done. Similarly, I have to query whether a certain test conducted on a a certain source e.g. stereo classical music is applicable to another source e.g. 2 channel movie audio. In doing so I have to consider numerous factors. One of the important ones that arises is how applicable are tests conducted at 44.1k to 48k. I still do not know. Therefore, I have to be careful when making such an extrapolation.

P.S. In terms of sources, let's not forget most of our 44.1k CD music is originally 48k or higher anyway! In fact, I believe very little music was ever recorded at 44.1k

[Nil Einne]

BTW, in terms of a test, I would argue you don't really have to test with the same source at different samples rates. All we really want to see whether there is any codec that performs well at 44.1k but poorly at 48k (or vice versa or whatever). Therefore, if you were to include in codec tests some 48k samples of the same type you are testing (classical music or whatever) you could get a good idea. If there was one codec that consistently performs poorly (in relation to the others) at 48k but much better (in relation to the others) at 44.1k then we have a good enough answer IMHO. Whether this is due to tuning or some bug at 48k or maybe the codec is very very bitrate sensitive and the effective bitrate at 48k is just to much for it doesn't matter IMHO. What does matter is we have to be careful when applying info from 44.1k to 48k and vice versa!

[Rockaria]

I think myself still at the stage of applying my ELECTRICAL knowledge to this SOUND technology, but.
There are three terms never to be confused :
. sample size (bit): the dynamic volume of each sample.
. sampling rate (khz): number of sample per second
. bit rate (kbps): amount of data (transfer, record, encode) per second

I think somebody initially confused the (20~)48k(bps) proper bit rate range of the Enhanced AAC plus V2 as the 48k(hz) sampling rate NATURE and never fixed it causing the huge confusions thereafter among some people.
In my view, it is theorectically possible that to achieve the extremely low bit rate, the sample size and/or sampling rate must be compromized than optimized(tuned). So there can appear 32khz or even lower sampling rate in these bit rate range, heavily reorganizing the original spectral image.