Multiformat@128kbps public listening test - OPEN

[rjamorim]

Hello.

I'd like to announce the opening of my Multiformat at 128kbps public listening test.

The test consists of comparing how 6 encoders: iTunes AAC, Ogg Vorbis, Lame MP3, Musepack, WMA Std. and Atrac3 behave compressing 18 different samples representing a wide spectrum of musical styles.

People interested in participating are invited to visit this page:
http://www.rjamorim.com/test/multiformat128/presentation.html

The test will end on May 16th and results will be posted soon after.

Best regards;
Roberto.

[aketon]

Why itunes 4.2! Apple announced itunes 4.5 a few days ago!

[echo]

Originally posted by aketon
Why itunes 4.2! Apple announced itunes 4.5 a few days ago!

version 4.5 has some problems. Take a look here (http://www.hydrogenaudio.org/index.php?showtopic=21148&hl=) .

regards
echo

[aketon]

Damn, I should have read it earlier! I already installed the new version and there is nowhere to find the old version again! That's bad!!! I hope apple is going to announce a new version soon!

[rjamorim]

You can still download v4.2 here:
http://a1408.g.akamai.net/7/1408/9955/20031218/akamai.info.apple.com/iTunes4/WW/win/019-0123.20031218.zes4d/iTunesSetup.exe

Get it fast before Apple pulls it off!

[bond]

i will personally strike every doom9 user who doesnt join the test!

[Doom9]

i will personally strike every doom9 user who doesnt join the test!I dare you to strike me :devil: I rather stick to my guns.. my audiovisual system is better in the video department than in the audi o one.

[bond]

ok than i will only strike 53,342 and not 53,343 doom9 members (6th May 2004 11:38) :D

[avih]

i don't get it, why the discrimination? ;)

[SeeMoreDigital]

Originally posted by bond
i will personally strike every doom9 user who doesnt join the test! Shit-a-brick... where do I sign :D

I wonder if there's going to be any 128kbps 6Ch codec tests in the near future. We could do with some 'impartioal' testing. As we sure don't don't want or need another WMA9-vs-AAC-HE slanging match :scared:


Cheers


EDIT: Originally posted by bond
ok than i will only strike 53,342 and not 53,343 doom9 members (6th May 2004 11:38) :D You forgot to un-strike yourself... Doh!

[rjamorim]

I'm very sorry to inform that the test has been cancelled due to problems with some samples.

It should start again next wedsneday, if no issues arise.

My sincere apologies to anyone that already invested his time on participating.

Best regards;

Roberto.

Obs: You can keep informed of what is going on with my tests subscribing to my low-traffic newsletter:
http://www.rjamorim.com/test/newsletter.html

[SeeMoreDigital]

Sorry to hear about that rjamorim.

I had only gone so far as to blowing the dust off my AKG K240 headphones and having my ears syringed :p

Looking forward to the tests...

I have to ask. Will there be any 6Ch codec tests?


Cheers

[rjamorim]

Originally posted by SeeMoreDigital
I have to ask. Will there be any 6Ch codec tests?

I have considered that. A test comparing WMA Pro, Vorbis, HE AAC and Real Audio at, say, 192kbps would be very interesting, IMO. But there are some problems.

First, very few people have surround systems that could be used for such test. Second, there is no comparator program a la ABC/HR that supports playing 5.1 files.

Some people suggested downmixing to solve both issues, but IMO that is a bad compromise. The point of encoding to multichannel is preserving surround information, and that is lost on downmixing.

Still, I'm open to suggestions and while I probably won't conduce such test, I would be very happy to help anyone willing to conduce it.

Regards;

Roberto.

[E-Male]

i was thinking about just that today
it could be done like this (another bad compromise, i know):
take 3 stereo sources, encode them to 5.1 (3*2=5+1), then reencode the result to 5ch-wave, demux it and remux the stereo chans, so they can be used with the comparison program
the whole thing would have to be planed much more than i just described, but it's 4:00 here
i gues you got the idea

another alternative would be to leave out the comparison program, and just give everyone the 6ch waves that whare encoded with the competing codecs and back to wave and just let them compare them (using cryptic filenames ofcourse)

[SeeMoreDigital]

Originally posted by rjamorim
I have considered that. A test comparing WMA Pro, Vorbis, HE AAC and Real Audio at, say, 192kbps would be very interesting, IMO. But there are some problems.

First, very few people have surround systems that could be used for such test. Second, there is no comparator program a la ABC/HR that supports playing 5.1 files.

Some people suggested downmixing to solve both issues, but IMO that is a bad compromise. The point of encoding to multichannel is preserving surround information, and that is lost on downmixing. I must admit an interest in encoding at even lower bitrates than 192kbps, in fact there was quite an heated discussion regarding encoding at 128kbps VBR.

I guess when it comes to a comparator program I can see that this would be a little difficult. I also think it's fair to say that some of the codec manufacturers are even slightly biased toward their own players (ie: RA10 = RealPlayer, WMA = WinMedia Player9 etc). In the end I resorted to using Media Player Classic for listening to all the codecs I tried.

So yes, what do we use for an source(s). In the first instance I think it may be prudent to provide some speaker mapping test files first (which I just happen to have on my web site) as I can quite imagine some speaker systems being incorrectly set up. But when it comes to musical/dialogue material, no way do I think down-mixing to two channels or up-mixing two channel material to six channels is a way forward. I would be far happier using extracts from DTS or a high bitrate DolbyDigital DVD's!

Cheers

[bond]

Originally posted by rjamorim
Some people suggested downmixing to solve both issues, but IMO that is a bad compromise. The point of encoding to multichannel is preserving surround information, and that is lost on downmixingyeah but the important point imho in a listening test is the quality the codecs provide, and the quality will stay untouched when downmixing
still we already discussed all this :(

[Doom9]

is there a truly lossless multichannel format that does not pose any major problems for playback? I think there are enough people with 5.1 capabilities, and you don't really need a program to conduct those tests.. just decode all encoded 5.1 files to that common source format and offer them for download. Use nondescript filenames, and have people enter their results in a http form. Alternatively, to save space, an application to decode the original format clips would have to be written. You download the original encoded files, then you run an application that does the decoding to a common format, using nondescript filenames..
Obviously, this isn't quite as advanced, but I think for the average user it's more than enough.. I had a look at ABC/HR and I have my doubts that everybody participating in such a test actually uses more than a fraction of the tool's capabilities (yeah no go ahead and flame me.. I'm a clueless n00b after all ;)) Also, since for a 5.1 test movie soundtracks seem like the most likely source candidates, and when watching a movie you are not entirely focused on the audio part, you're probably less likely to spot the differences, and breaking differences down to a sample by sample level is imho overkill... just as is comparing codecs frame by frame.

I agree that downmixing wouldn't be a good idea for a multichannel listening test... who uses multichannel audio if his/her setup can only handle 2 channels?

[M7S]

How much sense does it make to look at rjamorim tests when one chooses which codec/app to use for encoding the audio for a movie? How big is the difference between music and movie tracks?

Eg. ITunes was the winner in the AAC-test, but Nero wasn't far behind and it uses VBR which could (a wild guess) be more important to movie tracks than to music.


I might join the test if I'm able to find some headphones. Does the soundcard make any difference? I’m using a not-so-great, built-in soundcard. :(

[SeeMoreDigital]

Well M7S, the answer to most of your questions is yes!

However, I do fell that you would benifit by doing a little reading up in the audio FAQ's, where I'm sure your questions could be answered more fully.

Good luck and cheers for now.

[bond]

Originally posted by Doom9
Obviously, this isn't quite as advanced, but I think for the average user it's more than enough..hrhr doom9 doesnt realise the importance of abc/hr ;)

one important point is that its a double blind listening test. your suggestion would allow to make sure that people dont know which sample is encoded with which codec, but it excludes (or better said doesnt ensure) another important thing:
the test is not in the first place about comparing the encodes between each other, its about comparing each encode to the uncompressed source, thats a small but important difference
in abc/hr you dont know which one is the encode, which one is the source aso...


also abc/hr is very powerfull for doing the test: it allows switching during playback between the streams aso.. its not like you listen to the whole 20secs and than to the 20secs of the other sample, or that you always have to search in the player to be able to find the same place in the stream again you listened to before, to be able to really compare two parts

also abc/hr allows you to efficiently test if you are really capable to hear a difference, often your brain tricks you: abc/hr displays the propability of if you really hear a difference

the last two points are imho very important cause otherwise i think people will often simply only guess the quality "over the thumb"

[Doom9]

@bond: I've looked at the app before posting ;) I know what it does.

I doubt anybody could get their hands on an uncompressed movie soundtrack, which only leaves an AC3 or DTS stream as a source.. but that comparison could be done easily. And if you get a bunch of files with nondescript names, all using the same lossless codec (the 5.1 material is being encoded in a lossless format), nobody would know which file corresponds to which codec ;)

also abc/hr is very powerfull for doing the test: it allows switching during playback between the streams aso.. its not like you listen to the whole 20secs and than to the 20secs of the other sample, or that you always have to search in the player to be able to find the same place in the stream again you listened to before, to be able to really compare two partsThat's the part where I said it's overkill for a 5.1 soundtrack ;) You still have video to focus on, plus with 5.1 your audio setup plays a big role.. not everybody has the speakers ideally aligned, not everybody has their sitting position where it should be based on the speaker position.

[bond]

Originally posted by Doom9
I doubt anybody could get their hands on an uncompressed movie soundtrack, which only leaves an AC3 or DTS stream as a source.. but that comparison could be done easily. And if you get a bunch of files with nondescript names, all using the same lossless codec (the 5.1 material is being encoded in a lossless format), nobody would know which file corresponds to which codec ;)yeah but giving the people 5 samples ala
sample1.wav, sample2.wav ... sample5.wav and maybe the source too and than asking "plz rank them" or so is not enough
you have to ensure that everyone tests in similar circumstances (-> abc/hr with the different levels 1-5), that everyone tests against the source (and not the encodes against each other) aso...

and yes dts sources should be a must for such a test

That's the part where I said it's overkill for a 5.1 soundtrack ;) You still have video to focus on, plus with 5.1 your audio setup plays a big role.. not everybody has the speakers ideally aligned, not everybody has their sitting position where it should be based on the speaker position.well imo the surround hardware, the speakers, the video stream aso... dont belong at all to this test
this test should be about audio codec quality only
hardware or your eyes shouldnt kick in when the goal is measuring audio quality!

and if the goal is to test only the quality of the codecs, downmixing and than listening with headphones is a good solution imo as it eliminates problems like different hardware/speaker systems, other noise around when listening in the living room, not enough people which have such a 5.1 system aso...

the only argument against the downmixing is that the so called "surround effect" gets lost...
wel still for me thats no argument when it comes to comparing codecs quality, as the quality doesnt get hurt during downmixing, its exactly the same quality coming out of 2 speakers now instead of 5/6

also it shouldnt be forgotten that the 5.1 codecs do no magic...
all of them have the same info in the same channels! so to say the left front channel carries the same info no matter if encoded with vorbis, wma9 pro, aac or whatever other 5.1 codec!
because of this i wonder how does this "surround effect" should influence the quality? -> basically i wonder why the "surround effect" should vary between different codecs? -> the same info is carried in the same channels with every codec anyways (they dont mix channels, or leave channels out to be more efficient or so) -> only the quality differs between the codecs, and the quality can also be measured when downmixing to stereo
maybe lower quality hurts the "surround effect feeling", but again quality can also be measured with stereo


so to sum it up i like the downmixing way as
- hardware differences should never be allowed to influence tests about codecs quality, or at least the influence hardware has should be minimized afap
- every codec has the same info in the same channel anyways. only the quality in each channel is different, not the channels themselves
- living room listening is not a good enviroment for testing - headphones should be recommended!
- not enough people with 5.1 systems
- the quality doesnt change with downmixing

[Mug Funky]

I doubt anybody could get their hands on an uncompressed movie soundtrack

it gets worse than that... it's not unheard of for sound labs to use 192CBR mp3 for spotFX samples...

and there's the issue that most PAL movies have transcoded soundtracks. it's true i don't notice most of the time, but pre-echo is a pain for 5.1 applause and such (in fact, i laugh bitterly when i see the "dolby digital, OMG how amazing is surround" intro with the helicopters that comes with a lot of movies. there is huge pre-echo in that, but it's meant to be showcasing how good it sounds :()

[SeeMoreDigital]

Originally posted by Mug Funky
it gets worse than that... it's not unheard of for sound labs to use 192CBR mp3 for spotFX samples...

and there's the issue that most PAL movies have transcoded soundtracks. it's true i don't notice most of the time, but pre-echo is a pain for 5.1 applause and such (in fact, i laugh bitterly when i see the "dolby digital, Yes, this audio halo effect reminds me of what often used happened with analogue compact cassette tapes. Whereby the tape was so tightly wound around the spool you could often hear an faint imprint slightly before and after the 'main sound'! This was of course generated because of magnetic tape transference...

That said, most 'digitised' movie sound tracks are stored on magnetic tape, so even though the sound is stored digitally, there might well be transference.

Anyway, back to the subject at hand. Personally I think a DTS source will be fine. I have a few DTS audio only CD's and they sound very good indeed.

Cheers

[rjamorim]

Originally posted by bond
yeah but the important point imho in a listening test is the quality the codecs provide, and the quality will stay untouched when downmixing
still we already discussed all this :(

Well, if you only want to know about quality, you can conduce a stereo listening test and easily extrapolate.

And we discussed this at HA which isn't the appropriate place to discuss multichannel, IMO. Most people there care only about CD sources. :)

[rjamorim]

Originally posted by SeeMoreDigital
In the first instance I think it may be prudent to provide some speaker mapping test files first (which I just happen to have on my web site) as I can quite imagine some speaker systems being incorrectly set up.

Right, this test would surely introduce more challenges than a stereo test. But, if it is doable, the results would surely be VERY interesting.

But when it comes to musical/dialogue material, no way do I think down-mixing to two channels or up-mixing two channel material to six channels is a way forward. I would be far happier using extracts from DTS or a high bitrate DolbyDigital DVD's!

I agree. Test sources should reflect what people use on average, so DVD excerpts in AC3 and DTS would do an appropriate source.

Also, AC3 could be used as test anchor.

[rjamorim]

Originally posted by Doom9
is there a truly lossless multichannel format that does not pose any major problems for playback?

FLAC.

I think there are enough people with 5.1 capabilities, and you don't really need a program to conduct those tests.. just decode all encoded 5.1 files to that common source format and offer them for download. Use nondescript filenames, and have people enter their results in a http form. Alternatively, to save space, an application to decode the original format clips would have to be written. You download the original encoded files, then you run an application that does the decoding to a common format, using nondescript filenames..
Obviously, this isn't quite as advanced, but I think for the average user it's more than enough..

Well, there are several point in preferring ABC/HR, but the most important one is the HR part (hidden reference). It serves two important purposes:
- Check if the listener is really able to detect a difference in the encoded sample. If he can't, there's 50% chance that he'll move the wrong slider.
- Avoid bias. If the listener knows what is encoded and what is the reference, he might start hearing differences that are not really there, just because his brain knows it is the encoded file, so it must sound worse.

I agree that downmixing wouldn't be a good idea for a multichannel listening test... who uses multichannel audio if his/her setup can only handle 2 channels?

Right. I believe it definitely wouldn't have the same significance.

and yes dts sources should be a must for such a test

Not really. No everyone encodes from DTS. We should try to emulate what people use on a daily basis.

the only argument against the downmixing is that the so called "surround effect" gets lost...
wel still for me thats no argument when it comes to comparing codecs quality, as the quality doesnt get hurt during downmixing, its exactly the same quality coming out of 2 speakers now instead of 5/6

In that case, if you only want to know sound quality, check the tests I already conduced. The only difference from a 5.1 test to a stereo test is the surround info. You get rid of the surround info, and you don't need to worry anymore - test audio from CD.

[Doom9]

Well, there are several point in preferring ABC/HR, but the most important one is the HR part (hidden reference). It serves two important purposes:
- Check if the listener is really able to detect a difference in the encoded sample. If he can't, there's 50% chance that he'll move the wrong slider.
- Avoid bias. If the listener knows what is encoded and what is the reference, he might start hearing differences that are not really there, just because his brain knows it is the encoded file, so it must sound worse
I've been thinking how I could do this for video, but I just end up clicking at my source files randomly.. I've also been wondering if there were a way to mark a certain range for later review (I've done that manually so far.. but most sources get watched over and over.. but without deciding if one is really better than the other.. I just try to be as specific w.r.t. the differences as I can be).

If you remove the reference, and randomize the other sources, you have solved the bias problem.. but then the test becomes codec X against codec Y, not codec X/Y/Z against the reference. At least in the video area I consider the former method to be a valid testing method... given a certain filesize, which output do you like most seems to be a valid question to me (and I know if I compare them to the reference, I'll never be happy since every so often I'm not even happy with the reference itself).

[M7S]

This could be an n00bish solution, but what if one muxes all the samples into an mka-file and plays it with MPC?

Then you could change from one sample to the next on the fly. If the reference sample is randomly muxed in before or after every encoded sample you could even get that 50% change to choose the wrong sample.

To make sure nobody misunderstood me, this is what you'll see when you open the mka file in MPC and select the menu play -> audio ->

---------------------------------
Options...
---------------------------------
file.mka / English, 1a (audio1)
file.mka / English, 1b (audio1)
file.mka / English, 2a (audio1)
file.mka / English, 2b (audio1)
file.mka / English, 3a (audio1)
file.mka / English, 3b (audio1)
file.mka / English, 4a (audio1)
file.mka / English, 4b (audio1)
file.mka / English, 5a (audio1)
file.mka / English, 5b (audio1)
file.mka / English, Ref (audio1)
--------------------------------

Either 1a or 1b is the reference...

Is this a possible method?

[SeeMoreDigital]

Hi M7S,

That's not such a bad idea (putting all the different streams in the same container). Not only could it be used for audio codec testing but for video codec testing too!

All we need is to get someone like Gabest onside to create a special 'blind test' player. And we'll be sorted!

At last... an even better use for the Matroska container :D


Cheers

[M7S]

Originally posted by SeeMoreDigital
Not only could it be used for audio codec testing but for video codec testing too!
---
At last... an even better use for the Matroska container :D

Matroska can't deal with more than one video stream iirc. Maybe OGM or MP4 could be used for video codec testing.

[SeeMoreDigital]

Originally posted by M7S
Matroska can't deal with more than one video stream iirc. Maybe OGM or MP4 could be used for video codec testing. Yes ofcourse... in my enthusiasm I forgot this was not one of MKV's strong points!

MP4 is out of the question as it would require extreme hacks to input anything other than Mpeg4 video. I don't know about OGM as I've never used it!

Maybe it's time for the guys at Matroska to up the support for multiple video streams...

Cheers

[bond]

mkv, mp4, ogm can handle multiple video streams, the problem is the only player which handles video stream switching is quicktime afaik

still a tool like abc/hr for video testing would create a better testing environment than simply using any player which handles video stream switching