48kbps AAC public listening test

[Gabriel_Bouvigne]

A 48kbps AAC public listening test is coming:
http://www.mp3-tech.org/content/?48kbps%20AAC%20public%20test

[Gabriel_Bouvigne]

Test is now open

[Niddhogg]

Listened to a few samples, but honestly I think my audio-sensory organs need a tuning... :D
Sometimes 1 or 2 were noticably worse than others, yet most of time it was hard for me to distinguish between them.

As for the 48kbps itself, what is the target for the codecs?
I have listened to online radio a lot, and that was on dial-up. One thing was for sure, listening to anything over 32kbps was sure to result in interrupted broadcasts.
As this test is intended to see progress in AAC HEv2 compared to others wouldn't it make sense to test ER (Error Resilience) as well?
The AAC paper features many techiniques to limit audio degradation during limited transfer speeds. This goes for mobile transmission as well, of course.

So... if broadcast or mobile use is (also) scope of the test it might be more usefull to ask / set up an online broadcast (degredation here can be equalled to that of mobile networks) and have 1 station use the 4-5 codecs you want tested for a week. Listeners can then just casually listen and switch between station no. 1, 2, 3 etc. without having to pay full attention to comparing.
After a week there'd sure be preferences over 1 station over another, you can allways simulate transmission failure during the test to test resillience of the codes.

Naturally, all this is moot when you're interested in locally stored samples, but I for one don't see the use in cbr 48 for that. Also, the AAC amendment doesn't seem to focus on that and sort of states that AAC LC is intended for those purposes.

hope i'm making sense here.

Timetable :

The public 48kbps aac test will start on the 20th of February, 2006. It will end on the 12th of March, 2006.

[starts listening again]

[IgorC]

The test was extended untill 19 of the March. There are not enough results for test. Please, participate in this test.
We all need to know wich HE-AAC is good, right?
PS (HE-AAC2) is not always better (but often even noticebly worse) than simply SBR (HE-AAC1) at 48 kbits.
That's another good reason to do this public test.
Thank you.

For each sample as minimum 15 results are needed. http://www.hydrogenaudio.org/forums/index.php?showtopic=41598
Single result is accepted.

[IgorC]

The test is finished.
http://www.hydrogenaudio.org/forums/index.php?showtopic=42679

[Gabriel_Bouvigne]

Test results are now available:
http://www.mp3-tech.org/tests/aac_48/results.html
http://www.mp3-tech.org/tests/aac_48/plot_overall_zoomed.png

*Nero and CT are both quite competitive
*3gpp code is quite good for a reference code
*It seems that parametric stereo does not overall bring benefits at 48kbps.

[Rockaria]

One thing which cannot be overlooked here is that the NERO encoder was using ABR(new implementation) while others were encoded with CBR.
If the segment of the ABR becomes larger, I guess it becomes closer to VBR(same overall bit rate) which is considered more than 10% more efficient than CBR.

We are not sure, ATM, what kind of ABR algorithm is used in the encoder, as well as how differently it is affecting the PS encoding : note the difference 317 - 308(9 for CT CBR) < 329 - 310(19 for Nero ABR) .
It can also mean the CBR is more efficient than ABR or VBR for the PS encoding.

With that small sample size, I agree the results are statistically tied, with an exception of the (itune's) LC encoding. But still it can also mean the lower bit rate LC encoding is not efficient with CBR at all.

[shon3i]

One thing which cannot be overlooked here is that the NERO encoder was using ABR(new implementation) while others were encoded with CBR.
Rockaria i agree with you. I just now debate with Ivan Dimkovic about that.

[Ivan Dimkovic]

There was a listening test on Hydrogenaudio where various Nero encoding modes were tested:

http://www.hydrogenaudio.org/forums/index.php?showtopic=41191

http://static.morbo.org/test1_results.png

So, it could be seen that the difference between Nero ABR and Nero CBR is not that big:

Nero VBR1 - 3.56
Nero ABR - 3.53 (note, this is exactly the same encoder as used in Gabriel's test)
Nero CBR - 3.48

Difference Nero CBR/Nero ABR - 1.4% (absolute)

And, in the current test we have:

Nero ABR - 3.29
CT CBR - 3.17

Difference CT CBR/Nero ABR - 3.78% (absolute)

It is very hard to compare two different tests, but since Nero codecs are the same as used in Gabriel's test I am quite convinced that Nero, if forced to pure CBR mode, would still be a bit better than CT CBR.

Also, this is the same old argument we are seeing on many tests - but long time ago the decision was made to test the best what a codec could offer at a bit-rate range - if CT does not have VBR / ABR - it is not the fault of the other encoders with more advanced options.

Besides, Nero ABR was well within the bit rate deviation limits on this test (in fact it was less than average bit rate of the test) - so I cannot see any reason to complain.

Just my 2 eurocents ;)

[Rockaria]

I appreciate the hydrocents! Grains of salt.. I will drop by more often to read the news, even if I am not planning on posting on the forum yet.

Btw, a recent CT based-Winamp5.1 encoder DLL frontend CLI (enc_aacPlus.exe) now seems to support the bitrate-based(better result size estimation than quality based one) vbr(or abr) encoding.

TEST 1 - HE TEST
...
CODEC B - VBR1 - quality factor 0.15
CODEC C - VBR2 - quality factor 0.37
..
TEST 2 - HE v2 TEST
..
CODEC B - VBR1 - quality factor 0.32
CODEC C - VBR2 - quality factor 0.54
..

I am not sure if Nero's encoder is also providing the bitrate-based VBR and still being puzzled by the quality factor mentioned in the original quote.
If it is related to naac.exe's quality grades(3~10) and actually having the varying bitrate to keep the given even quality grade, I believe the algorithm would be very different than CT's one.
And of course, it would be very hard to get the similar overall bit rate(yeah, it didn't mentioned the result bit rates for the vbr keys), which makes the comparison vary rough on the very bitrate-sensitive low-bitrate area. The ABR with entire segment would be similar to CT's one, if I am correct.

Another thing to mention is the algorithm optimization for the target bitrate zone : I guess the AAC HE is optimized for the bit rate around 64k(ST) , overlapped then surpassed by PS(optimized @ around 32k?) below 48k.
If the VBR(or ABR) does not yield the noticeably(far more than 2.2% I guess) better quality than CBR, I believe it's the algorithm optimization problem on the zone.

Lastly, my common sense has been VBR > ABR > CBR theoretically and practically by some reasonable margin. I seem to need some more precise data to be understood by such as :
. performing some comparison between CT CBR, VBR(or ABR I am not sure), Nero CBR, ABR and VBR.
. quantitative analysis with more accurate/relevant data & strictly controlled variables.
...

Thanks.

[Ivan Dimkovic]

I appreciate the hydrocents! Grains of salt.. I will drop by more often to read the news, even if I am not planning on posting on the forum yet.

Btw, a recent CT based-Winamp5.1 encoder DLL frontend CLI (enc_aacPlus.exe) now seems to support the bitrate-based(better result size estimation than quality based one) vbr(or abr) encoding.

I am not sure if Nero's encoder is also providing the bitrate-based VBR and still being puzzled by the quality factor mentioned in the original quote.

Yes, new upcoming Nero encoder will have CBR, ABR, quality based VBR and 2-Pass VBR which gives specified bitrate/size.


If it is related to naac.exe's quality grades(3~10) and actually having the varying bitrate to keep the given even quality grade,

New VBR won't be similar in any way to old one used in current N7 encoders.


I believe the algorithm would be very different than CT's one.
And of course, it is very hard to get the similar overall bit rate(yeah, it didn't mentioned the result bit rates for the vbr keys), which makes the comparison vary rough on the very bitrate-sensitive low-bitrate area. The ABR with entire segment would be similar to CT's one, if I am correct.


VBR can be quite different yes, but for SBR 48 kbps bit rate deviation is much smaller than people would expect - and hence the small quality difference as noted in last Nero-internal test.


Another thing to mention is the algorithm optimization for the target bitrate zone : I guess the AAC HE is optimized for the bit rate around 64k(ST) , overlapped then sundued by PS(optimized @ around 32k?) below 48k.
If the VBR(or ABR) does not yield the noticeably(far more than 2.2% I guess) better quality than CBR, I believe it's the algorithm optimization problem on the zone.


Indeed, each codec/profile has a "sweet spot" - problem of quality expectations is quite complex, as there is not too much to "tweak/optimize" in SBR (leave alone PS) , as more than 50% of the spectrum is the parametric reconstruction of the low part coded with LC-AAC. You have not too much choices what to improve in SBR (maybe filtering / noise floor estimation and up to some point pre-echo detection) - and LC-part of the SBR has not so much impact to total quality.

With that in mind - I think even 2.2% quality improvement is a success :)

But I do expect 2-pass to yield even better results, so there is definitely more space for improving.

[Sagittaire]

So, it could be seen that the difference between Nero ABR and Nero CBR is not that big:

Nero VBR1 - 3.56
Nero ABR - 3.53 (note, this is exactly the same encoder as used in Gabriel's test)
Nero CBR - 3.48

Difference Nero CBR/Nero ABR - 1.4% (absolute)

And, in the current test we have:

Nero ABR - 3.29
CT CBR - 3.17

Difference CT CBR/Nero ABR - 3.78% (absolute)

yes but for bitrate:

CT HE-AAC V1 : 46.22 Kbps
Nero HE-AAC V1 : 47.94 Kbps

Difference CT CBR/Nero ABR - 3.72% (absolute)

always these bitrate problem ...

[IgorC]

@Sagi.
Audio bitrate issue is different from video.

Yes there is different on short samples.
Try to do :
1. Encode 1 CD 10-20 songs with CT CBR 48 kbps and Nero VBR v3 (48 kbprs). You will see the same size.
2. But when you will cut some part of the song and will evualuate that bitrate has very big variance.

VBR buffer is repected so final size of a big files (like 10 songs of CD or soundtrack 1-2 hours) will have 48 kbit/s.

You already discused it with guys from HA so do I. :)
You know how hard for me to agree with facts sometimes but I did it ;)
If you will read some information about it you will find it's abolutly normal if bitrate is higher on short testing

[shon3i]

You already discused it with guys from HA so do I.Can you give link about this discusion, Thanks.

But my personaly notice is that this test is unfair beacouse CBR and ABR can't compare, ABR is much better in all cases. This test is very good and i not agins't Nero but name of the test is wrong. For me the test will be right if in test be used old N7 codec. Bye

[bond]

isnt ct's encoder available for free?

[dimzon]

isnt ct's encoder available for free?
Yes, it's free. Just download/install WinAmp 5.2x (free edition) and get enc_aacplus.dll from plugins folder!

[IgorC]

NO!!!!! (God gave me capslock for a little while) It's not free http://www.winamp.com/player/
To encode to AAC+ pro version $19.90 . It's some kind of demo version like Nero7 package.

It's not meaning if you are free to download pro version you can use it forever for free.

Another time politic games what is freeware what's shareware etc.

"Use FAAC if you open source beleiver. Nero,Apple and CT are commercial products." Good point, isn't it?

Open source and commercial projects are always here *together* . Without open source com. prods would not exist and vice versa without com. projects nobody will working on reference code.

[dimzon]

NO!!!!! (God gave me capslock for a little while) It's not free http://www.winamp.com/player/
To encode to AAC+ pro version $19.90 . It's some kind of demo version like Nero7 package.
As I said before You does not need PRO version! You need FREE version and use enc_aacplus.dll from it (from FREE version) - everything work fine!
http://www.winamp.com/player/free.php

[Ivan Dimkovic]

As I said before You does not need PRO version! You need FREE version and use enc_aacplus.dll from it (from FREE version) - everything work fine!
http://www.winamp.com/player/free.php

Please forgive my display of ignorance - but what is the difference between that approach, and using command line frontends that directly use aacenc32.dll from Nero package? :)

[dimzon]

Please forgive my display of ignorance - but what is the difference between that approach, and using command line frontends that directly use aacenc32.dll from Nero package? :)
AFAIK Winamp is FREE so I can use this encoder for free
Nero is not free so using aacenc32.dll from Nero (without having Nero licence) is illegal

[shon3i]

As I said before You does not need PRO version! You need FREE version and use enc_aacplus.dll from it (from FREE version) - everything work fine!
http://www.winamp.com/player/free.php
Absolutley right, only you must paid to encode via Winamp. Practically we paid sonic runtime.

[Ivan Dimkovic]

AFAIK Winamp is FREE so I can use this encoder for free
Nero is not free so using aacenc32.dll from Nero (without having Nero licence) is illegal

I don't know... if the DLL functionality you are using also supposed to be FREE than it is correct - but if the DLL is not licensed in the free version, I am not so sure.

[dimzon]

I don't know... if the DLL functionality you are using also supposed to be FREE than it is correct - but if the DLL is not licensed in the free version, I am not so sure.
according this page http://www.winamp.com/player/free.php this DLL IS LICENSED in the free version too...

[Ivan Dimkovic]

Then forgive my display of ignorance - I was not so informed about Winamp's licensing.

[IgorC]

Dimzon
please read what it says here http://www.winamp.com/player/

: AAC+ costs 20$. If you donwloading free Winamp 5.21 and get .dll files and use it for your cli is totally illegal beacuse you did not use restriciton of speed 6.x . Your cli doesn't use any restriction. It's illegal. You also support higher than 128 kbit/s and it's available only in pro version.

Nero7 is updating each month so theoricly speaking you have a 30 days trial each month for personal use.

Yes. That's is. When one commercial encoder is better than open source or freeware there is nothing to say but "na na na. I like open source"
Ok Nero isn't open source but it's best HE-AAC encoder. And that's I'm caring about. Quality. If it's open source or payware it's another question.

Don't mix economics and politics factors. That never was usefull.

[shon3i]

You also support higher than 128 kbit/s and it's available only in pro version.There is no any limitation in enc_aacplus.dll, and dimzon's cli, of course,

Ok Nero isn't open source but it's best HE-AAC encoder. LOL, Newer!

[IgorC]

Free version only up to 128 kbit/s http://www.winamp.com/player/free.php

Nero is the best HE-AAC Stereo encoder
You can't be disagree with the results of professional public test.
http://www.mp3-tech.org/tests/aac_48/plot_overall_zoomed.png

[sasam]

I think that codecs should be judged by maximum quality they can reach with fixed filesize no matter what modes are used.
Codecs with more features should not be punished.
There was similar discussion where MSU limited AVC video codecs to main profile and 1 pass in their testing and thats not good.

I cant wait to test 2pass nero audio encoder. It should do wonders with movies with lots of silence compared to CBR and ABR...

just my 2 pare :cool:

[shon3i]

@IgorC i know that diagram and whole day i debate with Ivan Dimkovic about that on some local forum. Newermind. I agree with result but this result is unfair beacouse Nero use ABR instead CBR. Someone can tell me that only 2% difference is ABR than CBR, but i think that is not true because ABR is more better. That's my notice. The right result will be if on test be used old Nero7 codec. Bye

[IgorC]

Did you read what I've writen in this forum respect to ABR,CBR and VBR?

Please, read it again my messages.
I'm not going to explain all already eplxained stuff by people who know about audio compression.

[GodofaGap]

Uhm... there seems to be no winner in this test. All error bars overlap, so declaring any codec as best as a result from this test is wrong.

[rjamorim]

Nero is the best HE-AAC Stereo encoder

You need to learn how to interpret test results

You can't be disagree with the results of professional public test.

You need to learn proper grammar. And Gabriel's test wasn't professional, or even formal at that

Uhm... there seems to be no winner in this test. All error bars overlap, so declaring any codec as best as a result from this test is wrong.

Correct. At most, you can claim Nero HE AAC v1 is better than the 3GPP reference

[IgorC]

I know that estadisticly it's not sure to say what codec is better. They are all tied, except 3GP. I've learned probability and estadistic. Yes, it's not correct to interpret like that.

I also understand that it wasn't ideal test and maybe it wasn't correct from my part to say "professional public". But in the same time this test is enough fair (based on abxing of 18 samples by approx. 12-16 persons) .

P.S. My personal respect to people from neigbour Brazil. Igor Cavalera and Lula da Silva well knowed people. Igor Cxxx is my real name. It's coincidence. Tudo bem! :)

[Gabriel_Bouvigne]

I know that estadisticly it's not sure to say what codec is better.

It's a little more than "not sure". Except perhaps 3gpp vs NeroV1, there is a high degree of uncertainty regarding relative rankings of contenders.

However, we are more than 95% sure that all contenders are lower quality than Lame -V5 (high anchor) and higher quality than iTunes (low anchor).