Repeated lossy-encoding test: AAC, Opus, MP3, and MP2

[GeoffreyA]

In reference to a recent discussion (https://forum.doom9.org/showthread.php?p=2021163#post2021163), I gave the repeated lossy-encoding test a go. The results are interesting, not that it matters for encoding in practice.

Except for QAAC, all encoding was done with FFmpeg, using the relevant libraries. 101 iterations were performed. QAAC and Opus seem to be the winners, though neither escapes unscathed. Warning for your ears in all cases except QAAC, FDK, and Opus!

https://workupload.com/archive/ARup4FAjnQ

ffmpeg -i REF.flac -c:a aac -b:a 128k a.m4a

for /l %%x in (1, 1, 100) do (

ffmpeg -y -i a.m4a -c:a copy tmp.m4a
ffmpeg -y -i tmp.m4a -c:a aac -b:a 128k a.m4a

)


/////////////////////////////////////////////////////////////////////////

zzffmpeg -i REF.flac -c:a libfdk_aac -b:a 128k a.m4a

for /l %%x in (1, 1, 100) do (

zzffmpeg -y -i a.m4a -c:a copy tmp.m4a
zzffmpeg -y -i tmp.m4a -c:a libfdk_aac -b:a 128k a.m4a

)


/////////////////////////////////////////////////////////////////////////

d:\tmp\av\qaac\qaac64 --cbr 128 REF.flac -o a.m4a

for /l %%x in (1, 1, 100) do (

ffmpeg -y -i a.m4a -c:a copy tmp.m4a
d:\tmp\av\qaac\qaac64 --cbr 128 tmp.m4a -o a.m4a

)


/////////////////////////////////////////////////////////////////////////

d:\tmp\av\qaac\qaac64 --tvbr 64 REF.flac -o a.m4a

for /l %%x in (1, 1, 100) do (

ffmpeg -y -i a.m4a -c:a copy tmp.m4a
d:\tmp\av\qaac\qaac64 --tvbr 64 tmp.m4a -o a.m4a

)


/////////////////////////////////////////////////////////////////////////

ffmpeg -i REF.flac -c:a libopus -b:a 128k a.opus

for /l %%x in (1, 1, 100) do (

ffmpeg -y -i a.opus -c:a copy tmp.opus
ffmpeg -y -i tmp.opus -c:a libopus -b:a 128k a.opus

)


/////////////////////////////////////////////////////////////////////////

ffmpeg -i REF.flac -c:a libopus -b:a 128k -vbr constrained a.opus

for /l %%x in (1, 1, 100) do (

ffmpeg -y -i a.opus -c:a copy tmp.opus
ffmpeg -y -i tmp.opus -c:a libopus -b:a 128k -vbr constrained a.opus

)


/////////////////////////////////////////////////////////////////////////

ffmpeg -i REF.flac -c:a libmp3lame -b:a 128k a.mp3

for /l %%x in (1, 1, 100) do (

ffmpeg -y -i a.mp3 -c:a copy tmp.mp3
ffmpeg -y -i tmp.mp3 -c:a libmp3lame -b:a 128k a.mp3

)


/////////////////////////////////////////////////////////////////////////

ffmpeg -i REF.flac -c:a libtwolame -b:a 128k a.mp2

for /l %%x in (1, 1, 100) do (

ffmpeg -y -i a.mp2 -c:a copy tmp.mp2
ffmpeg -y -i tmp.mp2 -c:a libtwolame -b:a 128k a.mp2

)


The test is not rigorous, and these were separate scripts that I pasted together above.

[tebasuna51]

It has always been said that it is a bad idea to recode what has already been recoded, but it is comforting to know that the free software opus and the recommended qaac acceptably withstand the challenge.

[GeoffreyA]

Today, I found that using FDK in VBR mode brings the quality up to QAAC. But when both are CBR, QAAC does have the edge.

FFmpeg's encoder is concerning: even after 10 iterations, it degrades considerably.

[Z2697]

FFmpeg's native AAC is even worse than FAAC, in terms of generational loss. Maybe the overall quality as well.
It's extra concerning that it's probably very often used by, um, "average joe" (amonst FFmpeg users)... which are exactly the ones that are more likely to re-encode lossy audio over and over...
Yes, commercial softwares (which probably have more user base) probably gonna have their own or licensed encoder, but the lack of high quality FOSS AAC encoders is still disappointing.

[GeoffreyA]

I think many videos "out there" have used FFmpeg's AAC,* and this test shows there are considerable issues when it comes to generational loss. With a single encoding, it sounds all right these days, but distortion can often be detected in challenging parts. Anecdotally, I've found that older encodings, likely FFmpeg AAC when it was worse, tend to give me a headache because of the strange stereo distortions.

* Doesn't YouTube use FFmpeg for their AAC tracks?

[SeeMoreDigital]

Out of interest...

Where does Nero's v1.5.4.0 AAC-LC encoder (from 2010) stand in the rankings when compared the other AAC encoders?

[GeoffreyA]

Out of interest...

Where does Nero's v1.5.4.0 AAC-LC encoder (from 2010) stand in the rankings when compared the other AAC encoders?

Using a binary from Free Codecs (https://www.free-codecs.com/nero_aac_codec_download.htm) and the following script:


neroAacEnc -q 0.4 -if REF.wav -of a.m4a

for /l %%x in (1, 1, 100) do (

ffmpeg -y -i a.m4a tmp.wav
neroAacEnc -q 0.4 -if tmp.wav -of a.m4a

)

It doesn't fare too well, falling behind FDK but ahead of FFmpeg, displaying pervasive "tinkling."

QAAC > FDK > Opus > Nero > MP3 > MP2 > FFmpeg AAC

Here are the updated files (https://workupload.com/archive/z9J4UT4Yct), where all are using VBR except for FFmpeg AAC and MP2.

This is perhaps consistent with HydrogenAudio's ranking of AAC encoders (https://wiki.hydrogenaudio.org/index.php?title=Advanced_Audio_Coding) for ordinary encoding (unlike here):

QAAC > FhG > FDK > Nero > FFmpeg > FAAC > Libav

[j7n]

I am curious how things would change without latency compensation or random delay where the audio blocks in input and output no longer precisely overlap, which would happen if you cut the stream per some requirements and edited it in any way.

[GeoffreyA]

I am curious how things would change without latency compensation or random delay where the audio blocks in input and output no longer precisely overlap, which would happen if you cut the stream per some requirements and edited it in any way.

Here is QAAC with and without delay: https://workupload.com/archive/nf9VmJMkKg.

I used the --no-delay setting. Not sure if that would be enough.

[VoodooFX]

QAAC > FDK > Opus > Nero > MP3 > MP2 > FFmpeg AAC

Is this from ABC/H blind ranking?

[GeoffreyA]

Is this from ABC/H blind ranking?

No. The differences were audible enough that it was reasonable, if not ideal, to rank them without blind testing, which I'm bad at.

[VoodooFX]

No. The differences were audible enough that it was reasonable, if not ideal, to rank them without blind testing, which I'm bad at.

Mind can play tricks on you, always do blind testing.
Here is ABC/H tool if you don't have it already:https://www.swisstransfer.com/d/892c20b7-998a-4ff3-96e4-d64102e7a747

[GeoffreyA]

Mind can play tricks on you, always do blind testing.
Here is ABC/H tool if you don't have it already:https://www.swisstransfer.com/d/892c20b7-998a-4ff3-96e4-d64102e7a747

That's right. I've used fb2k for blind testing in the past, but my brain tends to get lost. I'll give the linked tool a try when I get a chance. Thanks.

[VoodooFX]

When you rank them do ABX test [you'll see the button] 8-10 times for 1st vs 2nd places.

[FranceBB]

Thanks for the comparison.
I listened to the samples and I confirm what everyone else said:

1) Apple AAC AudioToolbox (qaac)
2) Frauhnofer AAC (fdk aac)
3) Nero AAC
4) FFMpeg AAC (lavf aac)



Here's the spectrum analysis using Spek via the Discrete Fourier Transform.

https://i.imgur.com/AdA3ZWi.png
https://i.imgur.com/XNHKh95.png
https://i.imgur.com/DjmcKUu.png
https://i.imgur.com/12tezgS.png
https://i.imgur.com/JoZ4Jvp.png

Apple's Audio Toolbox encode looks surprisingly close to the original despite the 101 encoding iterations it has gone through. It really managed to get a very good result and wasn't affected much by the generational loss. Even the peaks, despite the soft frequency rollback, are still the same which means that it hasn't been hard-clipped during the encode. We're still losing frequencies compared to the original lossless FLAC in the 18-20 kHz region, but those were subject to a rollback to keep the original feeling of the song alive, while other were compressed and removed. Overall it's a very nice result. Frauhnofer AAC via FDK_AAC is second, but it's miles away. We don't have anything in the 15000Hz region and while there are a few peaks at 13500Hz most of the frequencies have been compressed and there's a very substantial clipping. Nero AAC is a mess. Don't get fooled by the frequencies in the 22000Hz region, those are actually artifacts that got preserved and led to other artifacts which - as iterations went through - led to non existing frequencies with very noticeable crackling and distortion. Last we have FFMpeg AAC (lavf) whose spectrum is basically unrecognizable. There's almost no discernable pattern any longer and the artifacts make up pretty much the entire spectrum because - as iterations piled up - artifacts are the ones that got preserved leading to averaging of the entire signal.


It really pains me to say it because I'm a great supporter of free and open source encoders, but once again Apple's closed source encoder in Audio Toolbox is the winner... :(

[hellgauss]

libfdk has a default cutoff at 14khz, which is confirmed by your graph. Also it has lower bitrate, which is a small % for a standard encoding, but it can leads to very bad results if reiterated.

https://trac.ffmpeg.org/wiki/Encode/AAC

PS: I'm a fan of libfdk :)

[GeoffreyA]

Thanks for the comparison.
I listened to the samples and I confirm what everyone else said:

1) Apple AAC AudioToolbox (qaac)
2) Frauhnofer AAC (fdk aac)
3) Nero AAC
4) FFMpeg AAC (lavf aac)



Here's the spectrum analysis using Spek via the Discrete Fourier Transform.

https://i.imgur.com/AdA3ZWi.png
https://i.imgur.com/XNHKh95.png
https://i.imgur.com/DjmcKUu.png
https://i.imgur.com/12tezgS.png
https://i.imgur.com/JoZ4Jvp.png

Apple's Audio Toolbox encode looks surprisingly close to the original despite the 101 encoding iterations it has gone through. It really managed to get a very good result and wasn't affected much by the generational loss. Even the peaks, despite the soft frequency rollback, are still the same which means that it hasn't been hard-clipped during the encode. We're still losing frequencies compared to the original lossless FLAC in the 18-20 kHz region, but those were subject to a rollback to keep the original feeling of the song alive, while other were compressed and removed. Overall it's a very nice result. Frauhnofer AAC via FDK_AAC is second, but it's miles away. We don't have anything in the 15000Hz region and while there are a few peaks at 13500Hz most of the frequencies have been compressed and there's a very substantial clipping. Nero AAC is a mess. Don't get fooled by the frequencies in the 22000Hz region, those are actually artifacts that got preserved and led to other artifacts which - as iterations went through - led to non existing frequencies with very noticeable crackling and distortion. Last we have FFMpeg AAC (lavf) whose spectrum is basically unrecognizable. There's almost no discernable pattern any longer and the artifacts make up pretty much the entire spectrum because - as iterations piled up - artifacts are the ones that got preserved leading to averaging of the entire signal.


It really pains me to say it because I'm a great supporter of free and open source encoders, but once again Apple's closed source encoder in Audio Toolbox is the winner... :(

Thanks for the excellent analysis, FranceBB.

Yes, as much as I am "anti Apple," I've got to admit they are good at what they do. Perhaps the most alarming finding is FFmpeg because of its widespread use. Even after 10 iterations, it degrades badly. Something is certainly faulty in the encoder.

Did you manage to try Opus? I hear a strange "whooshing" sound.

libfdk has a default cutoff at 14khz, which is confirmed by your graph. Also it has lower bitrate, which is a small % for a standard encoding, but it can leads to very bad results if reiterated.

https://trac.ffmpeg.org/wiki/Encode/AAC

PS: I'm a fan of libfdk :)

I used VBR 2 but should have used 3, which would have been closer to the other files in terms of bitrate.

Here are encodings with VBR 3 and 4. (https://workupload.com/archive/t6xDRuF2R7)

Carrying on with FranceBB's idea, Spek shows that higher VBR settings raise the lowpass cut-off to over 15 kHz. A quick listening, not blind, suggests that it is now on par with Apple. My apologies for making a mess of this part of the test! I think it may be best to redo all the files. Also, I was using a build of FFmpeg from April for libfdk, but a new binary for the others, though I doubt that makes any difference.

https://i.imgur.com/qSoH4wE.png
https://i.imgur.com/H6oiEbl.png

When you rank them do ABX test [you'll see the button] 8-10 times for 1st vs 2nd places.

Apologies for not tackling this yet. Much of my weekend was spent calibrating a new monitor, breaking head and eyes :)

[FranceBB]

Perhaps the most alarming finding is FFmpeg because of its widespread use. Even after 10 iterations, it degrades badly. Something is certainly faulty in the encoder.


Yes, it's bad and it used to be worse in the past, but by the time FFMpeg 6.x came out, it got a bit better, so if you think the current version is bad, believe me, I've seen worse unfortunately. There are still commits going on (https://github.com/FFmpeg/FFmpeg/commits/c2d17c4db88c4ca11d264d15efde66233733ed51/libavcodec/aacenc.c), though, so we can always hope that it will eventually get better. Back when I was re-encoding 192 kbit/s AC3 48000Hz from .ts I ended up needing 320 kbit/s in AAC 48000Hz to get the same quality, which totally defeated the purpose of re-encoding from a space perspective and only really made it worth it for compatibility reason. AC3 384 kbit/s in 5.1? That could easily lead to 520 kbit/s in AAC. Obviously FDK_AAC did much better and was able to reduce those bitrates. Not much on the Stereo as I never liked squeezing 192 into 128 kbit/s, but I was able to squeeze 384 kbit/s 5.1 in 320 kbit/s.


Did you manage to try Opus?


Nope, but only because I wanted this to be a comparison between AAC encoders, a bit like the ones Hydrogen Audio does. Still, thank you for doing this, I always wanted Doom9 to be active on this front as well. :)

I guess next on the list would be the open source FFMpeg AC3 Encoder (lavf) vs the Dolby AC3 Encoder (Dolby Media Encoder). I could use the old Dolby DP600 to do this, but I don't think using Dolby's software from 2007 would be fair, so if you have access to it let me know, otherwise we can organize something else.

A quick listening, not blind, suggests that it is now on par with Apple.

As much as we both want that, that is unfortunately not the case. There's still substantial clipping. :(

[hellgauss]

FFMpeg AC3 Encoder (lavf) vs the Dolby AC3 Encoder (Dolby Media Encoder)

AFAIK Dolby Encoder, at least for DD+, has mandatory DRC, so 100x encoding would not be possible. I've read about a DD certified ffmpeg plugin by Mainconcept, but I've not read about any test with it.

[GeoffreyA]

Nope, but only because I wanted this to be a comparison between AAC encoders, a bit like the ones Hydrogen Audio does. Still, thank you for doing this, I always wanted Doom9 to be active on this front as well. :)

I guess next on the list would be the open source FFMpeg AC3 Encoder (lavf) vs the Dolby AC3 Encoder (Dolby Media Encoder). I could use the old Dolby DP600 to do this, but I don't think using Dolby's software from 2007 would be fair, so if you have access to it let me know, otherwise we can organize something else.

Well, ever since I downloaded LAME 3.96.1 as a high-school student in 2004, I've enjoyed, occasionally, seeing what lossy codecs can do. Back then, using the demanding WAV (https://workupload.com/file/KhcNBa5wbFq) from the Diablo II installer, I found that 192 kbps non-joint-stereo was transparent for me. It would be nice to do more tests here, but we've got to do some blind ones; otherwise, the HydrogenAudio members will laugh at us :)

An AC3 test would be interesting. Unfortunately, I do not have access to any Dolby encoder.

As much as we both want that, that is unfortunately not the case. There's still substantial clipping. :(

That's quite all right: I only use QAAC, nothing else, not even Opus :)

[j7n]

I believe that older Dolby encoders are better. The new one in the Encoding Engine is made with some VBR-style methods and sometimes leaves unused space in the bitstream. But encoding 100 times to produce garbage is too much work.

[j7n]

Here are AC-3 and E-AC-3 encodings made with Dolby Enconding Engine. Decoding was made with ffmpeg. REF.flac was upsampled to 48000 Hz because the new encoder doesn't accept 44,100 Hz. I couldn't find a command-line decoder that was more accurate. I manually removed whole frames of delay and padding. As incomplete frames are dropped by DEE, I initially padded the stream with a second of silence. Two bitrates 160 and 256 kbit/s were selected. Available result after 10 and 100 iterations.

http://j7n.sytes.net/misc/d9_transcoding_test/

Other tools are GUI based and it is not practical to encode 100 times with them.

[GeoffreyA]

Here are AC-3 and E-AC-3 encodings made with Dolby Enconding Engine. Decoding was made with ffmpeg. REF.flac was upsampled to 48000 Hz because the new encoder doesn't accept 44,100 Hz. I couldn't find a command-line decoder that was more accurate. I manually removed whole frames of delay and padding. As incomplete frames are dropped by DEE, I initially padded the stream with a second of silence. Two bitrates 160 and 256 kbit/s were selected. Available result after 10 and 100 iterations.

http://j7n.sytes.net/misc/d9_transcoding_test/

Other tools are GUI based and it is not practical to encode 100 times with them.

It holds up respectably. The artefacts are a "rumbling in the background." Similar to Opus but better.

[GeoffreyA]

When you rank them do ABX test [you'll see the button] 8-10 times for 1st vs 2nd places.

Not sure if I did this right, but I just took the ABX test and got the following results:

Original vs ffmpeg-aac.wav
8 out of 8, pval = 0.004

Original vs libfdk.wav
4 out of 8, pval = 0.637

Original vs libmp3lame.wav
9 out of 9, pval = 0.002

Original vs nero.wav
9 out of 9, pval = 0.002

Original vs qaac.wav
2 out of 8, pval = 0.965

Original vs dee_ac3.wav
8 out of 8, pval = 0.004

Original vs libopus.wav
8 out of 8, pval = 0.004

[VoodooFX]

Not sure if I did this right, but I just took the ABX test and got the following results:

But you didn't ranked anything, you need to blind rank them by the perceived quality:

5 - Imperceptible
4 - Perceptible but no annoying
3 - Slighty annoying
2 - Annoying
1 - Very annoying

There are sliders to do that:

https://i.imgur.com/yiIwm3n.png

There are play button at the bottom. Easiest approach would be to rank the edge cases first, first rank the worst quality and highest quality (leave at 5 if you can't hear a difference), then rank the middle.
No point in doing QAAC/libFDK ranking as those are transparent for you (from your ABX tests).


Your ABX tests show that you don't hear a difference between Original/QAAC/libFDK (actually it shows that you prefer QAAC over Original :) ).


PS:
BTW, when you rank all, then you can do ABX [a place vs original] starting from top rank (lower than 5), when you successfully ABXed some then you can skip ABX of lower ranks.
If you could ABX the first two places then it would be interesting to ABX 1st vs 2nd place. (you could do ABX all places, to make sure that you ranked them correctly, I personally not interested how crap encoders rank between each other)

PS2:
Your test is just another example that you can't trust non-blind tests, your " Original > QAAC > FDK" is not correct because you can't ABX those vs Original.

[GeoffreyA]

Thanks for the helpful explanation. As I said, I'm not good at this, but I see what you're saying; it makes sense. There's little use ABXing all, where the majority can be told apart easily.

I agree that one has to use blind testing before making statements about quality. If this were HA, it would have been the infamous TOS8 violation! Having said that, the picture was on the right track in that QAAC and FDK leave the others far behind, and with those laggards, the differences were so severe that it was reasonable to make an informal quality assessment.

Also, I did the ABX test with FDK VBR3, which raised the quality, whereas for the earlier ranking, QAAC > FDK, I had used VBR2, a mistake on my part that led to its being audibly inferior to QAAC. I'd like to redo all the encoding and testing.

[j7n]

Notice that I uploaded two formats. My observation is that AC-3 holds up better than E-AC-3.

But see here what happens if I misalign the frames, which is what I asked you to try. The default delay is 256 or one small audio block. So it is always the same exact interval that is encoded. But I trimmed off 3 samples on every step. Now an audio block get shifted. If you were to cut, you'd have to go out of your way to maintain alignment. The quality is poor even at 256 kbit/s.

Most codecs should be affected by this. An odd delay is likely worse than a number like 64. A block in MP3 is 576 and in AAC it is 2048. In E-AC-3 they do some kind of combining with DCT, but maybe less at high bitrate.

sox.exe tmp0.wav -t wavpcm tmp.wav trim 3s -0

http://j7n.sytes.net/misc/d9_transcoding_test/3s/

[GeoffreyA]

Notice that I uploaded two formats. My observation is that AC-3 holds up better than E-AC-3.

But see here what happens if I misalign the frames, which is what I asked you to try. The default delay is 256 or one small audio block. So it is always the same exact interval that is encoded. But I trimmed off 3 samples on every step. Now an audio block get shifted. If you were to cut, you'd have to go out of your way to maintain alignment. The quality is poor even at 256 kbit/s.

Most codecs should be affected by this. An odd delay is likely worse than a number like 64. A block in MP3 is 576 and in AAC it is 2048. In E-AC-3 they do some kind of combining with DCT, but maybe less at high bitrate.

sox.exe tmp0.wav -t wavpcm tmp.wav trim 3s -0

http://j7n.sytes.net/misc/d9_transcoding_test/3s/

Yes, non-blind, I also find that AC-3 seems to hold up better than E-AC-3 for the 160/100 iteration files.

I tried your system, borrowing the SOX line. It destroys the encoders. Even the mighty QAAC is humbled!

https://workupload.com/archive/T2uFZDAXtR

[filler56789]

.......................

QAAC > FDK > Opus > Nero > MP3 > MP2 > FFmpeg AAC

Well, at least you did admit that your test was not rigorous at all :D
It was less about repeated lossy-reencoding and more about what codec sucks less at 128 kbps ;)

MP2 should be encoded at 384 kbps; MP3, at 320 kbps; AAC and Opus, at 256 kbps per channel.

[lvqcl]

MP2 should be encoded at 384 kbps; MP3, at 320 kbps; AAC and Opus, at 256 kbps per channel.
Nonsense.

[filler56789]

Nonsense.

That's just your usual groundless opinion, of course.
MP3 is worse than MP2 at 256 kbps. But the pseudotest run by Geoffrey says the MP3 encoder performed better than libtwolame.

[Z2697]

That's just your usual groundless opinion, of course.
MP3 is worse than MP2 at 256 kbps. But the pseudotest run by Geoffrey says the MP3 encoder performed better than libtwolame.

You mean the generational loss, or just the general "sound quality" from non-cycled encoding?

[Z2697]

Thanks for the excellent analysis, FranceBB.

Yes, as much as I am "anti Apple," I've got to admit they are good at what they do. Perhaps the most alarming finding is FFmpeg because of its widespread use. Even after 10 iterations, it degrades badly. Something is certainly faulty in the encoder.

Did you manage to try Opus? I hear a strange "whooshing" sound.



I used VBR 2 but should have used 3, which would have been closer to the other files in terms of bitrate.

Here are encodings with VBR 3 and 4. (https://workupload.com/archive/t6xDRuF2R7)

Carrying on with FranceBB's idea, Spek shows that higher VBR settings raise the lowpass cut-off to over 15 kHz. A quick listening, not blind, suggests that it is now on par with Apple. My apologies for making a mess of this part of the test! I think it may be best to redo all the files. Also, I was using a build of FFmpeg from April for libfdk, but a new binary for the others, though I doubt that makes any difference.

https://i.imgur.com/qSoH4wE.png
https://i.imgur.com/H6oiEbl.png



Apologies for not tackling this yet. Much of my weekend was spent calibrating a new monitor, breaking head and eyes :)

There's a bandwidth option in fdkaac that controls the cutoff. The somewhat generic audio encoder option "-cutoff" in FFmpeg also controls this when the encoder is libfdk_aac (other encoder "interfaces" in FFmpeg may implement this differently).
FDK AAC should be as good as Apple's (just lacks some flexibility in rate control IMO), but sadly it's still nonfree.

In the previous discussion I said libopus makes low frequency sound like wind... that's what I mean.

[GeoffreyA]

Well, at least you did admit that your test was not rigorous at all :D
It was less about repeated lossy-reencoding and more about what codec sucks less at 128 kbps ;)

I should like to redo the test, using better settings and omitting obsolete encoders, except LAME as a legacy anchor. For practical purposes nowadays, the battle is between AAC, xHE-AAC, Opus, and perhaps AC-3. A blind listening test would also help, though in blatant cases, such as FFmpeg AAC and QAAC, the difference is obvious.

That's just your usual groundless opinion, of course.
MP3 is worse than MP2 at 256 kbps. But the pseudotest run by Geoffrey says the MP3 encoder performed better than libtwolame.

It's similar to new video codecs performing better in the low-bitrate regime, but the older codec doing better once given enough bits. Here's a quick 256-kbps CBR comparison (https://workupload.com/archive/zpLSGQtU9p) of LAME and TwoLAME.

FDK AAC should be as good as Apple's (just lacks some flexibility in rate control IMO), but sadly it's still nonfree.

In the previous discussion I said libopus makes low frequency sound like wind... that's what I mean.

FDK's being non-free has been a curse to the world, leading to many AAC encodings being inferior to LAME, thanks to FFmpeg's encoder.

Regarding Opus, yes, I heard the "wind" sound.

Apart from redoing this test, don't you think it would be interesting to perform a video one? x264, vvenc, etc.

[Z2697]

Apart from redoing this test, don't you think it would be interesting to perform a video one? x264, vvenc, etc.

Sure it will be interesting, privately I have done that ;) even something like x264 -> x265 -> SVT-AV1 cycle ;) (maybe do this for audio as well) but I deleted the results :o

[Z2697]

MP3 and MP2 are both MPEG-1 Audio though, just different "Layer". Is there "newer" and "older" relation between them?

[GeoffreyA]

Sure it will be interesting, privately I have done that ;) even something like x264 -> x265 -> SVT-AV1 cycle ;) (maybe do this for audio as well) but I deleted the results :o

I'd like to give it a go, but to put it in a presentable fashion on the forum, what settings to use? CRF, preset, tune.

MP3 and MP2 are both MPEG-1 Audio though, just different "Layer". Is there "newer" and "older" relation between them?

I think it's fair to treat them as newer and older because there is a bit of difference between the two.

[john33]

There's a good and fairly brief description of the mp2 and mp3 codecs, similarities and differences here: https://en.wikipedia.org/wiki/MPEG-1_Audio_Layer_II

[j7n]

With Layer-2 you can have the delay not compensated, which would make the quality much worse. LAME was designed to enable gapless playback, but that wasn't important ealier. The encoder and decoder both have a delay. In a video file you would trim that much silence from the beginnning to ensure lip sync. There could be a difference between a simple encoder like QDesign i-Media in Fast mode, which encodes all bands, and one that discards high frequencies with a psymodel. But they are GUI tools. I'd say that a good quality is about 96 kbit/s per channel. A bit less with recent codecs and a bit more with old codecs.

[GeoffreyA]

LAME's heavy tuning, especially of the psychoaucoustic model, gave it an advantage over competing encoders. Originally, it was worse. I think by around version 3.8x, from ~2001, LAME had hit critical mass of quality, though improvements would still come in 3.9x. It was fun to open MP3s in a hex editor and see the tag; LAME wrote the version number.

[Z2697]

Oops, looks like the old CELT has better (lower) generational loss than libopus.
https://workupload.com/archive/HEpHj2AnfA

Which might have a slight chance to explain why FFmpeg's experimental native Opus encoder also does pretty well, because it only implements CELT?
BUUUUUT... When I tried testing libopus with frame length less than 10ms (I used 5ms), which makes SILK impossible, there's still that wooshing sound.
So, who knows...:rolleyes:

[GeoffreyA]

Sounds pretty good.

Who knows where the culprit lies with libopus. Perhaps it's a bug or oversight, masked in ordinary usage.

At high bitrates, does Opus use SILK at all for certain components?

[Z2697]

I think it should use CELT-only mode for our "normal" bitrates...
But the structure of the libopus and opus-tools is, uhh, a bit too much for me, so I can only be 99% sure.
(Look: you need (lib)opus, libopusenc, opusfile and opus-tools, 4 repos! to build this opusenc.exe tool)


/* Threshold bit-rates for switching between mono and stereo */
static const opus_int32 stereo_voice_threshold = 19000;
static const opus_int32 stereo_music_threshold = 17000;

/* Threshold bit-rate for switching between SILK/hybrid and CELT-only */
static const opus_int32 mode_thresholds[2][2] = {
/* voice */ /* music */
{ 64000, 10000}, /* mono */
{ 44000, 10000}, /* stereo */
};

[GeoffreyA]

I wonder if it's a hard switch, or if it can decompose the audio into music and voice parts, applying CELT or SILK as needed. But of course, at the bitrates we're using, it should be pure CELT.

[Z2697]

IIRC, in automatic mode (which is the default) the voice / music type detection is run on each frame, also there's no "lowest bitrate constraint" unless you use hard-cbr, so there's no guarantee even if the content type detection is overridden.

Even if I modified the encoder to always use the CELT mode, the artifact is still there.
Well... maybe it's time to git bisect. Let's go. Oh no, need to sleep first.

[GeoffreyA]

It's interesting how the audio would be unfolded into voice and music, but I suppose that's a solved problem.

If I remember correctly, there were changes when it became Opus and wasn't a straight port of CELT. In other words, it might be like finding the needle in a haystack. Well, I hope you slept well and are ready to vanquish the code :)

[Z2697]

*Vanquished by the code*
Luckily there's a "primitive" opus_demo.exe encoding/decoding CLI tool available just from the main libopus repo so I don't have to go through all other repos each step.

Look what have I found! I said there's some filter stuff going on didn't I?
(Need to complete the rest of the steps to confirm, of course)

Bisecting: 2 revisions left to test after this (roughly 2 steps)
[0869829f343f85935fac22462d228a065d0ba320] Adds a 3 Hz high-pass filter and boost allocation on leakage

[Z2697]

0869829f343f85935fac22462d228a065d0ba320 is the first bad commit
Yeah... I'm not surprised.


git bisect start
# status: waiting for both good and bad commits
# bad: [bf52802a47dafd6408aa972dd593599d0d431847] abyss thresh (this is the local commit that I sent the mode threshold to the abyss so that it will always use CELT only mode... there's probably a better way to do that)
git bisect bad bf52802a47dafd6408aa972dd593599d0d431847
# status: waiting for good commit(s), bad commit known
# good: [63590897db35326cd1ce7784806f9b89a98631ea] Initial commit with the autotools stuff and files taken from Speex and Vorbis.
git bisect good 63590897db35326cd1ce7784806f9b89a98631ea
# bad: [a6d663c6ae089b0a9682c848f3ff7701b316526f] Disables tf_analysis() for hybrid mode
git bisect bad a6d663c6ae089b0a9682c848f3ff7701b316526f
# good: [5c80391b3529bcb3fc8afd3b83d7562bae4991ae] Comments, low bit-rate busting avoidance
git bisect good 5c80391b3529bcb3fc8afd3b83d7562bae4991ae
# skip: [07f884042eccd913c1e96c1503cc15dfe6af9d2b] Wrapping all allocation within opus_alloc() and opus_free()
git bisect skip 07f884042eccd913c1e96c1503cc15dfe6af9d2b
# good: [fc8b605c1712a9b64b587835dde75920e0df71c6] Eliminate signed overflow in constant, minor makefile.draft updates.
git bisect good fc8b605c1712a9b64b587835dde75920e0df71c6
# skip: [dac1b4fc92b0581874af28f84f013b8618ec315c] Make vararray and restrict checks fail GCC 2.95.3's broken implementation.
git bisect skip dac1b4fc92b0581874af28f84f013b8618ec315c
# bad: [0918365b5141780969e2a09608a11d91228b502e] Fixes a VBR bug with 2.5 ms frames
git bisect bad 0918365b5141780969e2a09608a11d91228b502e
# skip: [5609cec9a5e1ea8fcb056f2306a115cb3b61c4c9] Fixes two minor issues found in random testing at ridiculously low rate.
git bisect skip 5609cec9a5e1ea8fcb056f2306a115cb3b61c4c9
# skip: [217cdae98e44699648a215ebb527c4c3f4f171ac] Make it possible for run_vectors.sh to fail on the mono tests.
git bisect skip 217cdae98e44699648a215ebb527c4c3f4f171ac
# good: [6619a736376221f2782cecff55d051c3ecfc2ff7] Move nbits_total initialize before renormalization.
git bisect good 6619a736376221f2782cecff55d051c3ecfc2ff7
# good: [7143b2d0ff61690b698cc8a8b0e61852ba74d984] Merge branch 'tmp_draft'
git bisect good 7143b2d0ff61690b698cc8a8b0e61852ba74d984
# good: [9881484dbde25707b93d988cff6316d2f375727a] test_opus_api: Fix valgrind expectations broken by last commit.
git bisect good 9881484dbde25707b93d988cff6316d2f375727a
# skip: [747c817d96482883527c592b48367975c6f9a1a2] Adds MFCC standard deviation features
git bisect skip 747c817d96482883527c592b48367975c6f9a1a2
# good: [9cf62baafc02e53ffb86e498ac3dc0e6b1f8e03e] Implements a better transient metric for VBR
git bisect good 9cf62baafc02e53ffb86e498ac3dc0e6b1f8e03e
# good: [70d90d115d01cb47426295efca7a5a3ffcafd130] VBR tuning
git bisect good 70d90d115d01cb47426295efca7a5a3ffcafd130
# bad: [ac2e623d251bc336ca1d401b95ee47e6ffef0c51] Converting most of the new code to fixed-point (not complete yet)
git bisect bad ac2e623d251bc336ca1d401b95ee47e6ffef0c51
# bad: [0869829f343f85935fac22462d228a065d0ba320] Adds a 3 Hz high-pass filter and boost allocation on leakage
git bisect bad 0869829f343f85935fac22462d228a065d0ba320
# good: [2a9fdbc93dd048bb8fe5d992c1d4502fe3510666] Transient/VBR tuning, give more bits to frames where pitch changes
git bisect good 2a9fdbc93dd048bb8fe5d992c1d4502fe3510666
# good: [96d7a079425d39712f0214a868beb415c46b8061] Dynalloc based on a bands that stand out of the "noise floor"
git bisect good 96d7a079425d39712f0214a868beb415c46b8061
# first bad commit: [0869829f343f85935fac22462d228a065d0ba320] Adds a 3 Hz high-pass filter and boost allocation on leakage

The skipped steps are either fail to build or have other glitches (this thing is old, so probably fixed)

[Z2697]

I changed the hp filter from 3 to 0, trying to not break anything... (this is done on current main branch)
The problem is mostly gone.
https://workupload.com/archive/b2dMuedNrn
(Somehow I still think the old CELT is better, is this just something in my head?)

diff --git a/src/opus_encoder.c b/src/opus_encoder.c
index 276dc58d..e87a5379 100644
--- a/src/opus_encoder.c
+++ b/src/opus_encoder.c
@@ -1897,7 +1897,7 @@ static opus_int32 opus_encode_frame_native(OpusEncoder *st, const opus_res *pcm,
}
#endif
} else {
- dc_reject(pcm, 3, &pcm_buf[total_buffer*st->channels], st->hp_mem, frame_size, st->channels, st->Fs);
+ dc_reject(pcm, 0, &pcm_buf[total_buffer*st->channels], st->hp_mem, frame_size, st->channels, st->Fs);
}
#ifndef FIXED_POINT
if (float_api)

[j7n]

The highpass filter also impacts gapless playback in case DC or rumble is present. I think it should cause bass content to be delayed, maybe enough to be noticeable after 100 times. The encoder doesn't allow any such quality switches, and only people who can edit source code can toggle them.