Multiformat@128kbps listening test - FINISHED

[rjamorim]

Hello.

I'd like to announce the results of the Multiformat at 128kbps listening test

Vorbis aoTuV is tied to Musepack at first place, Lame MP3 is tied to iTunes AAC at second place, WMA Standard is in third place and Atrac3 gets last place.

The results page is here:
http://www.rjamorim.com/test/multiformat128/results.html

For those in a hurry, here are the zoomed overall results:
http://www.noveo.net/rjamorim/plot18z.png

Big thanks to everyone that helped and participated.

Best regards;
Roberto.

[DSPguru]

interesting results !
now we need an aoTuV libvorbis.dll.. :cool:

[Kurtnoise]

Hi !!!

These results sounds great. Thank you Roberto for your work. ;)

@DSPguru : we can have the aoTuV dll here : http://www.rarewares.org/quantumknot/aoTuV-dlls.zip .This is not exactly libvorbis.dll, but ogg.dll + vorbis.dll + vorbisenc.dll + vorbisfile.dll.

[rjamorim]

You can also get aoTuV beta 2 source codes at Aoyumi's page
http://www.geocities.jp/aoyoume/aotuv/

[DSPguru]

Originally posted by Kurtnoise13
@DSPguru : we can have the aoTuV dll here...thanks, i'm already familier with this package, but still thinks that a libvorbis.dll is somewhat a more elegant solution.. :)

[bond]

woow, now thats what i not expected

- vorbis aotuv: vorbis is back! and i am proud to have been 1 of only 3 people who tried finding out what vorbis encoder should be used, it was worth it :)
- mpc vs aac: funny that mpc was that better than itunes (with a only 0.15 higher setting than in the last test)
- wma9: worse than mp3 (and i even wonder that it got rated that high, even at 128 it had this metallic sound sometimes) -> go away m$
- atrac3: even worse than wma9 -> go away sony

and if you take this test as a comparison between some online music stores (itunes vs. wma9 based ones vs. sonys new store) itunes clearly comes out as the winner, leaving wma9 behind by far!

[SeeMoreDigital]

I can't believe I missed the bloody test again... The last thing I heard it was postponed!

"Mummy I don't want to play anymore" :mad:


Okay I'll let it go this time... Cheers

[rjamorim]

Originally posted by SeeMoreDigital
I can't believe I missed the bloody test again...

Subscribe to the newsletter (http://www.rjamorim.com/test/newsletter.html) ;)

[SeeMoreDigital]

Originally posted by rjamorim
Subscribe to the newsletter (http://www.rjamorim.com/test/newsletter.html) ;) Job done!

Thanks rjamorim

[JohnV]

Or visit HA more often. ;)

[DSPguru]

Originally posted by bond
- mpc vs aac: funny that mpc was that better than itunesindeed! maybe apple should hire frank.. :rolleyes:

[KpeX]

Interesting results, thanks to rjamorim and all who worked on it..

I am also quite impressed with aoTuV's performance, very solid :) this is the beauty of open source. Hopefully these tunings will work their way into the official vorbis releases.

Originally posted by DSPguru
indeed! maybe apple should hire frank.. :rolleyes: Or frank should work on FAAC :D

[Blue_MiSfit]

alright, so I'm impressed with the test results, and I would like to do some encoding. I'm far too lazy to encode with the provided sample encoder (this would require me converting some AC3s to 2ch WAVs which takes far too much time )... As such I am wondering if these new libraries can be integrated into BeSweet...

DSPguru? What do you say? ;)

Thanks all for more great open source work
~misfit

[hellfred]

DSPguru:
Can you please edit the links to the faqs in your signature? Every time i press one of the links i get:

You do not have permission to access this page. This could be due to one of several reasons:

1. You do not have permission to access the page that you were trying to. Are you trying to edit someone else's post or trying to access administrative features? Check that you are allowed to perform this action in the Forum Rules.
2. If you are trying to post, the administrator may have disabled your account, or it may be awaiting activation.

[Stux]

Originally posted by Blue_MiSfit
alright, so I'm impressed with the test results, and I would like to do some encoding. I'm far too lazy to encode with the provided sample encoder (this would require me converting some AC3s to 2ch WAVs which takes far too much time )... As such I am wondering if these new libraries can be integrated into BeSweet...

DSPguru? What do you say? ;)

Thanks all for more great open source work
~misfit

These results are not for multi-channel encoders.

[Tuning]

Thanks rjamorim,

Now moving to MPC ...:D

[IvS]

"MPG" should be corrected to "Musepack" or at LEAST "MPC."
It seems as if doom9 had no prior knowledge about Musepack and therefore was able to confuse "C" with "G."
So please correct that to not cause major confusion. I can imagine people reading that and thinking "MPG?! but, huh, wha?"
Musepack's site is here (http://www.musepack.net).

[M7S]

I might be repeating myself, but is it safe to assume that Vorbis AoTuv should be the best choise for 2 ch, 128 kbps audiotracks for DVDbackups? Or maybe I should ask the other way around: Is there any reason why this testreslult shouldn't apply for movietracks?

[spiff]

Originally posted by M7S
I might be repeating myself, but is it safe to assume that Vorbis AoTuv should be the best choise for 2 ch, 128 kbps audiotracks for DVDbackups? Or maybe I should ask the other way around: Is there any reason why this testreslult shouldn't apply for movietracks?

For quality/size reasons there is none..
The only problems I can think of are the harware support for AviMpeg4Mp3 , that .ogm as a container isn't as well known and Vorbis _AoTuv_ is a fork (small chance of errors not known yet, some tools hasn't got support)

[m0rbidini]

Originally posted by spiff
For quality/size reasons there is none..
The only problems I can think of are the harware support for AviMpeg4Mp3 , that .ogm as a container isn't as well known and Vorbis _AoTuv_ is a fork (small chance of errors not known yet, some tools hasn't got support)

Well, even Monty (http://slashdot.org/comments.pl?sid=108647&cid=9241587) doesn't like the fact people use term fork to describe AoTuV's Vorbis encoder.
It's odd to keep hearing this code referred to as a 'fork'. Yes, it's based on our reference code while doing further tuning just like all the free MP3 encoders are based off of the original dist8 or dist10.

Fork seems to imply that they're trying to make something incompatible or doing it without our blessing. Neither is true! We never wanted to have *the* only encoder. Nor did we want to be the only people trying to improve Vorbis's encoding.

AoTuV is a 100% real Vorbis encoder and the results of the test speak for themselves. Aoyumi and crew deserve kudos, and I'm glad to see them working on improving Vorbis encoding.

Monty

I also would like to see 'MPG' corrected to the correct term which is Musepack (http://mpc.corecodec.org/), in the homepage.

cya

[SeeMoreDigital]

Originally posted by IvS
"MPG" should be corrected to "Musepack" or at LEAST "MPC."
It seems as if doom9 had no prior knowledge about Musepack and therefore was able to confuse "C" with "G."
So please correct that to not cause major confusion. I can imagine people reading that and thinking "MPG?! but, huh, wha?"
Musepack's site is here (http://www.musepack.net). Probably even better to call it either MPK or Muse!


Cheers

[KpeX]

Originally posted by M7S
I might be repeating myself, but is it safe to assume that Vorbis AoTuv should be the best choise for 2 ch, 128 kbps audiotracks for DVDbackups? Or maybe I should ask the other way around: Is there any reason why this testreslult shouldn't apply for movietracks? IMO, These test results can be safely applied to most stereo encoding scenarios. Although results might be different at extremely high and extremely low bitrates, for most normal bitrates when encoding in stereo these results can most likely be extrapolated. As previously mentioned, encoding multichannel is another story, and AFAIK no vorbis encoder has implemented channel coupling for 5.1 as of yet.

As to whether Vorbis AoTuv is the best choice to you, depends on your ears, your desired destination format, and your desired playback solution.

[Lefungus]

Originally posted by SeeMoreDigital
Probably even better to call it either MPK or Muse!

Cheers

Official extension for musepack files is ".mpc"
It won't change soon, because it will only add to the confusion

[PhuzzyDay]

This is my first post here, but I am a 'veteran' of audio compression experimentation, if there is such a thing. No.. I do not know everything, but I have been at it for a while, and I do know what to listen for.

This test, combined with a couple other factors, caused me to do a few tests of my own.

For one thing, I had always been mildly curious about the ogg format, but I had never really seen compelling reason to give it some investigation. Perhaps I haven't been reading in the right forums over the years... Perhaps the compatibility of MP3 in so many places has caused me to ignore things a bit.

Another factor was that a few weeks ago I ‘joined’ the ‘Minidisc-er’ community, which I probably had assumed was in existence, but I hadn’t sought out. I bought a portable MD recorder for recording long seminars, actually, and the ‘new toy’ factor caused me to try out the ATRAC3 technology. Perhaps one of the reasons for my NOT finding a Minidisc ‘community’ was the fact that investing cash in such an inflexible device can so easily cause insecurities in people which cause them to close their minds to reason. Inflexible? I say this since DRM technology limits me from copying recorded material over digitally, and, for this reason, I haven’t yet figured out how I would convert it OUT of the file format Sony is using now. (OMG)

My first reaction to the listening test results was a bit of skepticism. I still believed that 128 kbps was pretty weak for sound quality, no matter what codec was used, although my experience with the MD was suprising to me, since I couldn’t hear the artifacts that I thought the low bitrate would bring. (133 kbps.) I NEVER thought MP3 at a bitrate this low could be as good sounding as what I thought I was hearing. Perhaps these comments, combined with my calling myself a ‘veteran’, indicate that I haven’t been paying attention for a while. Perhaps this is true, and this is my way of catching up a bit!

I also wondered about the biased opinions of people affecting the results of the tests. I have investigated the method used, and I am quite impressed, and convinced that this is not the case. I have been using the same test software to do some tests of my own.

Specifically, again, I thought MP3 at 128 would be bad, especially compared to ATRAC3.

Well, to make a long story short, my findings agree completely with the posted test results. I am surprised at how far MP3 has come with LAME, and a bit disappointed with ATRAC3, after doing the side by side tests myself.

Yes – I am a MINIDISC owner, and I admit, and agree that ATRAC3 IS INFERIOR to OGG, AND LAME for SURE at 128kbps or thereabouts.

Some have been saying that VBR gives MP3 and OGG an advantage, and I actually have to agree with this reasoning to a point. After all, if ATRAC3 could save some bandwidth during easy to compress sections, and utilize the extra for a good cymbal or snare drum hit, IT would be better then it is. BUT.. it DOESN’T, so in the real world, you get more bang for your storage buck with the other formats. (Would that be enough for ATRAC3 to catch up? NO idea.)

I don’t know if I I will start using OGG for everything now.. Probably not actually. Why? I have portables that don’t use it, not the least of which is Sony’s software for the MD…. (Yes – I know that converting any other lossy format to ATRAC3 only makes it worse.) Anyway, I am SERIOUSLY impressed with OGG Vorbis. It BLEW my mind when I put it against MP3 Pro at 64 Kbps, and I though MP3 Pro was pretty dang good. ATRAC3 at 64.. well, let’s not talk about that.. too painful. I have for a while been using Lame with the ‘extreme’ preset, which gives me about 224 kbps VBR. I find that Ogg Vorbis at quality 5, which runs at about 160 kbps, to be BETTER, even when the audio spectrum is viewed in Cool Edit.

In short… (Yes… too late…) The OGG Vorbis people have gained some SERIOUS respect from me, and so have the people managing the listening test. I hope to be involved in the next one.

Thanks everyone.. And keep talking through the Forums.. You are helping a LOT of people like me who very rarely say anything.. (And then, apparently, make up for it with a big text flood.)

PD
----------------------------
'An empty barrel DOES make the most noise, though a full one tends to make quite a mess when it finally cracks.'

[KpeX]

Welcome to the forums PhuzzyDay :)

If you continue to observe the AV world in such an open-minded and objective manner you'll be a valuable member here. You're right, that the Hydrogenaudio folks have some impressive software for objectively testing audio compression. It's also good to hear that you ultimately did some objective tests of your own. IMHO it is much better to trust such personal tests over anyone else's ears and over any spectral graph.

Cheers,

[yup]

Question abot ATRAC3. Which codec use for this test? From http://www.minidisc.org or from Sonic Stage Software. Recently I buy ATRAC CD player DNE-9 and try compare free ATRAC3 codec from minidisc.org and codec supplied with Sonic Stage 1.0, second was better for IMO. Now availible 2.0 version, but I not tested.
yup.

[rjamorim]

I used SonicStage 2.0

[yup]

rjamorim
How You convert ogm files created SonicStage 2.0 to wav files?
yup.

[rjamorim]

Originally posted by yup
How You convert ogm files created SonicStage 2.0 to wav files?

I play them in SonicStage and capture the output audio with Total Recorder.

[Bogalvator]

Originally posted by yup
How You convert ogm files created SonicStage 2.0 to wav files?

He captured the wav using Total Recorder.

Edit: Woah, apparently took me 6 mins to write the above! Apologies for my slowness