Public Multiformat Listening Test @~64 kbps[2011]. Nero/Apple HE-AAC, CELT, Vorbis

[IgorC]

Greetings!!!

There has been significant advancement since the last 64kbps test. The Nero HE-AAC and Aoyumi Vorbis encoders improved, Apple added HE-AAC capability, and we have a new low delay codec with high quality: CELT.

The following codecs are presented in this test:
Nero 1.5.4 HE-AAC
Apple QuickTime 7.6.9 HE-AAC
Ogg Vorbis AoTuV 6.02 Beta
CELT 0.11.2


Public Multiformat Listening Test @ ~64 kbps [March 2011] (http://listening-tests.hydrogenaud.io/igorc/Public%20Multiformat%20Listening%20Test%20@%2064kbps.htm)


The test is finished.
Here are the results:
http://listening-tests.hydrogenaud.io/igorc/results.html

* CELT/Opus --------- 4.00
* Aple HE-AAC ------- 3.82
* Nero HE-AAC ------- 3.55
* Vorbis --------------3.51


http://listening-tests.hydrogenaud.io/igorc/nonblocked_means_all.png





Note: Important workaround was made. It is not uncommon to see that some soundcards have isues with not 48 kHz sample rates. Aslo until moment CELT is optimized for 48 kHz while other codecs are well tested at 44.1 kHz. To avoid these limitations all decoded files (Vorbis and Nero, Apple HE-AAC) were resampled with very highly transparent resampler SOX VHQ while CELT files were directly encoded from the resampled source. The resampling hadn't impact on final results.

[jmac698]

Interesting. Most were terrible. Two were warbly, one was choppy, own seemed to be low pass filtered, and the best one had a slight pre-echo.
This was sample 1.
I wouldn't use any of them for music, however. 64bps FAIL.

[jmac698]

voices samples hard to tell. lol@sample 4, see http://www.youtube.com/watch?v=PTGkf21UpJ8
lol@sample 5! This is the remaster version, right? I think it sounds horrible! Much prefer the original. Kinda bad to test with a sample I rate 3 to begin with :)
sample 10: there's an obvious bug in two of them, sounds like fake stereo (it was mono). If this is randomized, then there's a bug in one of them I guess :)
Ok, after quickly browsing 12 samples, I can say we have one and only one clear winner here.
I think you can guess who it is.

[IgorC]

Ok, after quickly browsing 12 samples, I can say we have one and only one clear winner here.
I think you can guess who it is.
Until the moment it's very hard to say who is winner. Personal equipments and preferments make the results very different.


I wouldn't use any of them for music, however. 64bps FAIL.
Well, modern codecs do quite nice job at 64 kbps.
It is very suitable for streaming.



lol@sample 5! This is the remaster version, right? I think it sounds horrible! Much prefer the original. Kinda bad to test with a sample I rate 3 to begin with :)
sample 10: there's an obvious bug in two of them, sounds like fake stereo (it was mono). If this is randomized, then there's a bug in one of them I guess :)
Ok, after quickly browsing 12 samples, I can say we have one and only one clear winner here.
I think you can guess who it is.

Interesting findings.
Have you already send some results or it's your personal experience?
Your results could be helpful.


It will be good if other people participate in test as well it's very easy. All you nead is follow the steps http://listening-tests.hydrogenaudio.org/igorc/readme.txt

[SeeMoreDigital]

Well, modern codecs do quite nice job at 64 kbps.
It is very suitable for streaming.

I agree... At 64Kbps, AAC-HE with Parametric Stereo sounds even better ;)

[IgorC]

Nero, CT/Dolby and Apple. None of them suggest use Parametric Stereo (PS) at 64 kbps. That's why there are only HE-AAC v1 codecs in the test.
The intensity stereo (IS) is used instead of it. It performs much better than PS at 40-64 kbps.
All HE-AAC encoders, CELT use IS while Vorbis has an equivalent of it.

PS is mono file + side information.
Stereo image is narrower for PS.
In case of pure stereo (two instruments. Each one in single channel) the opening is 180º. But PS will make it like only 40-60º.

So, PS is 1 channel + 20-30% what has left from stereo information.

[IgorC]

BTW, If you don't want to download separate packages there is All-In-One package (ABC-HR and all samples)
It should be more easy.

ABC-HR_bin_and_samples (http://listening-tests.hydrogenaudio.org/igorc/ABC-HR_bin_and_samples.zip) (All-In-One)

[shon3i]

I almost finished all samples, btw why there is no CT/Dolby? i wish to hear is there some improvement over years?

isn't HE-AAC use classic JS/MS by default, IS is much destructive, IIRC CT has option to switch between MS and IS and PS, but MS is used by default.

[IgorC]

hi, shon3i
I almost finished all samples,
Good


btw why there is no CT/Dolby? i wish to hear is there some improvement over years?

The test is multiformat. Two HE-AAC codecs is already much.
And, yes, it will be interesting to see CT codec in future tests.


isn't HE-AAC use classic JS/MS by default, IS is much destructive, IIRC CT has option to switch between MS and IS and PS, but MS is used by default.
IS is automatically enabled at low bitrates (at least <80 kbps).
Hope this old post will clear the situation http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=40022&view=findpost&p=352425

[IgorC]

Also It will be great if people look into guide of how to perform blind test. http://ff123.net/64test/practice.html

[IgorC]

Some people wait for the last day to send their results. That's ok.
But it might be the case that someone can forget to send the results until 30th of March.
So if You have already even a few results send them soon.

Of course, the last moment results will be accepted during 30th and 31st (~01.00 of GMT -3.00) of March.

As far as I can see all participants have received the answer. Thank You to All who participate.

[menno]

IS is automatically enabled at low bitrates (at least <80 kbps).
Hope this old post will clear the situation http://www.hydrogenaudio.org/forums/index.php?s=&showtopic=40022&view=findpost&p=352425

A post from 2005, and it doesn't say anything about a released encoder actually using IS. In fact, the Nero AAC encoder has never used IS in any released version.

[IgorC]

Thank you for clarification, Menno.

I've made a lot of assumptions basing over old post

[shon3i]

IIRC, i remember Dimkovic's words that IS gonna be usefull only on low bitrate and LC encoding, and is not usefull for HE-AAC because of SBR. But CT can switch bettwen modes, so more testing can be show how much is good since IS on MP3 is useless.

btw IgorC, and preliminary result ?

[IgorC]

btw IgorC, and preliminary result ?

The test is extended until the 10th of April.
We need more results for samples 11 and 14-30 (14,15....29,30) and results for other samples are welcome as well.

[kieranrk]

As I have said by email, the understanding of VBR seems to be wrong.

[IgorC]

As I have said by email, the understanding of VBR seems to be wrong.

From email
Do you understand actually what VBR is?
1. Do you understand that your email is rude?
2. Do you understand that only Apple LC-AAC (and not HE-AAC) has pure VBR mode?
3. Do you understand that some developers and ex-developers were involded in checking all conditions during preparation of this test?
4. Do you understand that your message is in spam folder now?

[IgorC]

I almost finished all samples
Fine, where are they?

[kieranrk]

From email

1. Do you understand that your email is rude?
2. Do you understand that only Apple LC-AAC (and not HE-AAC) has pure VBR mode?
3. Do you understand that some developers and ex-developers were involded in checking all conditions during preparation of this test?
4. Do you understand that your message is in spam folder now?

1. No, it was just asking a question.
2. So then compare all codecs correctly.
3. OK, but that doesn't mean the test is perfect. I hope you understand the process of peer review. Do you perhaps know that you are speaking to a codec developer?
4. Well then that's your problem if you want to perform a biased test.

You haven't answered the point as to why you think you understand what VBR is. You can't compare constrained VBR against pure VBR.

[kieranrk]

I ignore you.

That is a very mature response.

[IgorC]

2. So then compare all codecs correctly.

Define correctly


4. Well then that's your problem if you want to perform a biased test.
I won't receive even single suggestion after that you were rude. Forget it. You can talk about this comparison on hydrogenaudio that is where it had its origin.

You haven't answered the point as to why you think you understand what VBR is.
Do you think I don't know such simple thing as what it is quality based VBR (constant quality).

You can't compare constrained VBR against pure VBR.
You can't do such statements if you don't know the real situation around tested audio codecs.
You didn't even know that constrained VBR of Apple encoder is actually more unconstrained than Nero VBR.

So, please, if you don't know what you are talking about first of all consult it.

[IgorC]

That is a very mature response.
your email was even more mature

[Skuto]

isn't HE-AAC use classic JS/MS by default, IS is much destructive, IIRC CT has option to switch between MS and IS and PS, but MS is used by default.


But CT can switch bettwen modes, so more testing can be show how much is good since IS on MP3 is useless.


Do you have a link to the CT encoder that can do this? None of the ones I have seem capable of it.

Switching between (JS/MS/IS) and (PS) would be very complicated to do well and might break some software, as it requires a change of channels in the LC-AAC part of the stream.

[shon3i]

Switching between (JS/MS/IS) and (PS) would be very complicated to do well and might break some software, as it requires a change of channels in the LC-AAC part of the stream.
I am not mean switching in bitstream, just before encoding, to set encoding mode, switching in encoding process will be ridiculous of course. And only encoder that support all channel modes (JS/MS, JS/IS, SS, DC, PS, M) is CT, but that not mean is better :P

[Skuto]

I am not mean switching in bitstream, just before encoding, to set encoding mode, switching in encoding process will be ridiculous of course. And only encoder that support all channel modes (JS/MS, JS/IS, SS, DC, PS, M) is CT, but that not mean is better :P

Ah, that's what I thought. As IS does not combine well with SBR, such a codec has no chance to be competitive at 64kbps. Maybe it could win on a single sample with very few HF, but if the codec cannot do the switch automatically, that is useless.

And even at 48kbps, it seems PS is worse or at the very least no better than JS/MS, so at 64kbps it should be even worse. This is true for both Nero and CT encoders:

http://www.mp3-tech.org/tests/aac_48/results.html

So Nero HE-AACv1 should be the best representative of AAC at this bitrate.

The Apple HE-AAC encoder is new and we don't know anything about it yet, so it's interesting to test for the first time. Their LC-AAC encoder was pretty good. We'll see how their HE-AAC does...

[shon3i]

So Nero HE-AACv1 should be the best representative of AAC at this bitrate.Well this test is very old from this point of view, because CT from that stage is much improved. In this test CT is been only CBR because that is only RC that supported at this time, while nero uses VBR and win by hair. That's why i would not use that test to say that Nero is No1, and Apple is here now, but without CTAAC here on test we can't decide what is optimal.

About PS, all authours are recommend it for ultra low < 48kbps, but again CT seem improved their encoder, and leave PS up to 56kpbs. And if we look that 48kps test we see that PS on 48kbps have mostly higher scores.

[IgorC]

I almost finished all samples
Fine, where are they?

You have answered me via PM. It's ok but possible drawback is that tommorow someone possibly may claim that his/her results weren't accepted.

You say that you have already sent your results but I haven't received anything from your address. I'm checking spam folder as well. Until now every mail has been received from other listeners.

You also say that you have deleted all results from your computer as well as from your yahoo account. So you haven't it anymore.
But there is still trash bin for deleted messages in yahoo mail. But I've understood that you have deleted it from this place too.

If you still have these results then they will be accepted or you can do listening test again.

If there won't be results then this same post is a evidence of what has happened.

[IgorC]

Skuto,

The main issue is that there is no equivalent of HA's TOS (rule) #8.

[Skuto]

Well this test is very old from this point of view, because CT from that stage is much improved.

Do you know where one can buy the latest encoder?

[Dark Shikari]

your email was even more matureKierank is correct. Flamewars like this are stupid, and both kierank and you should take this to private messages.

To be fair, however, the purpose of this seems to be to compare CELT against other codecs, not the other codecs against each other, in which case the fact that "unconstrained VBR" is being compared against "constrained VBR" is not a serious problem unless someone tries to make conclusions about anything other than CELT from this test.

Of course, you've already noted this down in the test info, so I think things are fine.

[IgorC]

Dark Shikari,
first of all read this

You didn't even know that constrained VBR of Apple encoder is actually more unconstrained than Nero VBR.
Apple HE-AAC has no unconstrained mode.
And we compare all codecs against each other not only CELT vs others.

[IgorC]

Your flamewar is stupid, and kierank is 100% correct.

Thank you for kind word. You make it even better.

[IgorC]

so what do you prupose to do? Exclued Apple encoder because it has no unconstrained VBR mode?

[IgorC]

We do enable unconstrained VBR when it's available. If there is no unconstrained VBR then we go to constrained VBR. Otherwise we should exclude codec completely.
And actually CELT's VBR is sort of ABR. Because there is no real VBR mode yet.

This way none of codecs can be compared because:
CELT's VBR is actually ABR
Apple VBR is actually constrained
Nero VBR is more constrained than Apple constrained

Jason, look here. http://listening-tests.hydrogenaudio.org/sebastian/

http://listening-tests.hydrogenaudio.org/sebastian/mf-128-1/results.htm . iTunes constrained VBR vs Nero VBR

[shon3i]

Do you know where one can buy the latest encoder?
IIRC you can find most recent version in Winamp but i am not sure is complete with all features. Otherwise i think can be buy it from dolby.

[Dark Shikari]

so what do you prupose to do? Exclued Apple encoder because it has no unconstrained VBR mode?No, just carefully note the limitation in the results of the test with an appropriate disclaimer. (You've already done this, so things are fine).

Some video encoders don't have 2-pass: that doesn't mean we can't compare them to those that do, it just means we have to note said limitation when we do.

[Skuto]

About PS, all authours are recommend it for ultra low < 48kbps, but again CT seem improved their encoder, and leave PS up to 56kpbs.


I don't really agree here: CT allows it up to 56kbps, but doesn't seem to default it. And allowing it doesn't necessarily mean it's a good idea. You can allow it as high as you want, that doesn't make it good.

And if we look that 48kps test we see that PS on 48kbps have mostly higher scores.

Are you looking at the same test as I am??


IIRC you can find most recent version in Winamp but i am not sure is complete with all features. Otherwise i think can be buy it from dolby.

The encoder in Winamp is CT version 8.2.0 from 2009-8-10.

BTW. Are you sure this supports intensity stereo? I saw a frontend with an option named that way, but Independent Stereo has nothing to do with Intensity Stereo.

The official CT encoders I have seen (though I do not have exactly this version) had no IS support at all.

[shon3i]

I don't really agree here: CT allows it up to 56kbps, but doesn't seem to default it. And allowing it doesn't necessarily mean it's a good idea. You can allow it as high as you want, that doesn't make it good.Eariler (dated around 2008) versions had limitation up to 48kpbs but they are increase for some reason. Anyway they somehow force HE-AAC over 128kbps, i don't know is classical marketing or it's real potential of this, but no one do a real test, and everything is based on test's from beginning of AAC era, and fact that CT encoder can use DSBR for high bitrate and that extra cost. Maybe sound stupid, but i still there is possibility that HE-AAC can preform better than LC-AAC, since there is different implementations, and different efficiency.

Are you looking at the same test as I am??Yep, but when i mean "mostly" i mean, 48kpbs is transition point and really depend when to use PS and when not. Some samples have really higher score than other, can't be clearly say that PS is completely useless for 48kbps and near 56kbps.

The encoder in Winamp is CT version 8.2.0 from 2009-8-10.I thought they update it. Btw iirc i last time installed winamp on mine pc, i can bet that version is been something like 8.2.5 and higher, what winamp do you have ?

BTW. Are you sure this supports intensity stereo? I saw a frontend with an option named that way, but Independent Stereo has nothing to do with Intensity Stereo.

The official CT encoders I have seen (though I do not have exactly this version) had no IS support at all. Of course, Independent = simple or real stereo, i know difference, but last time i had access to their cli (dated around 2010) it had switches JS/IS, JS/MS, for channel mode, but is marked as beta version.

[IgorC]

Latest Winamp 5.61 and Easy CD-DA extactor 2011.2 have the same Coding Technologies 8.2.0 encoder (Aug 10 2009)

[Skuto]

I thought they update it. Btw iirc i last time installed winamp on mine pc, i can bet that version is been something like 8.2.5 and higher, what winamp do you have ?


5.61, I downloaded just downloaded it from their website after you posted that.


Of course, Independent = simple or real stereo, i know difference, but last time i had access to their cli (dated around 2010) it had switches JS/IS, JS/MS, for channel mode, but is marked as beta version.

Ok, thanks for the info.

[IgorC]

http://www.mp3-tech.org/tests/aac_48/results.html

I downloaded a few samples from this test and compared them to latest Winamp 5.61 CT encoder.( ABC-HR blind test)
Exactly the same quality. Exactly the same artifacts in the same places.

[IgorC]

The test is finished.
The results will be soon.

[IgorC]

All right,

Here are the results: http://listening-tests.hydrogenaudio.org/igorc/results.html
Enjoy

http://listening-tests.hydrogenaudio.org/igorc/nonblocked_means_all.png

[jmac698]

Can you tell me some of the results by sample instead? I find them quite distinct; some couldn't be encoded while others were transparent.

ps I knew the answer in the first 10 minutes already :)

[Skuto]

Somebody did per-sample averages here:

http://people.xiph.org/~greg/opus/ha2011/

[LigH]

The SILK/CELT hybrid codec is now officially named "Opus", and was just approved by the IETF as possible coming internet standard. A RFC can be expected soon. And hopefully a dedicated branch on RareWares too ...