AAC encoding and comparison method. So can you compare?

[dns]

Hi people.

I decided to compare several audio codecs. I want to know what the community thinks about this method of comparison?
I used:

faac-1.30
fdkaac-0.6.3 (libfdk-aac_3.4.22p3) ("p" version stands for modified by Poikosoft audio laboratories.)
fdkaac-1.0.4
ffmpeg-N-109149-g459527108a-20221121
qaac-2.76
sox-14.4.2
ImageMagick-7.1.0-48

I took 15 tracks and encoded with different settings.

[faac*.m4a] faac ( -b 128, -b 256, -b 384, -b 512, -q 300, -q 500 )
[fdk1*.m4a] fdkaac-0.6.3 ( -b 128k, -b 256k, -b 384k, -b 512k, -q 3, -q 5, -q 8 )
[fdk2*.m4a] fdkaac-1.0.4 ( -b 128k, -b 256k, -b 384k, -b 512k, -q 3, -q 5 )
[fdk3*.m4a] fdkaac-1.0.4 ( -b 128k, -b 256k, -b 384k, -b 512k, -q 3, -q 5 ) --bandwidth 20000
[fhg*.m4a] fhgaacenc ( --cbr 128, --cbr 256, --cbr 384, --cbr 512, --vbr 3, --vbr 6)
[qaac*.m4a] qaac ( --cbr 128, --cbr 256,--cbr 320, --tvbr 90, --tvbr 127 )
[nero*.m4a] nero ( -cbr 128000, -cbr 256000, -cbr 384000, -cbr 512000, -q 0.5, -q 1 )
[ff*.mp4] ffmpeg -c:a aac ( -b:a 128k, -b:a 256k, -b:a 384k, -q:a 3, -q:a 5 )


Spectrograms of all received m4a files.

sox INPUT_FILE.wav -n spectrogram -x 1024 -z 120 -w Hann -t -h -o *.m4a.png

Then I compared the spectrogram of each file with the original (changing fuzz several times)

magick.exe compare -metric AE -fuzz ( 0%, 1%, 2%, 3% ) SPECTROGRAM.png *.m4a.png CMP_FILE.png

Where:
-metric type - measure differences between images with this metric
* AE - absolute error count, number of different pixels (-fuzz affected)
-fuzz distance - colors within this distance are considered equal
For one track I made a collage, for visual comparison (more red - worse). Full resolution here. (https://i.imgur.com/W2I5Wuo.jpg)
https://i.imgur.com/792Pmur.jpg
Here are (https://disk.yandex.ru/d/ILvlWgScxRuoUg) all the tracks in m4a, as well as their spectrograms
| *.png - original spectrograms
+-- [TRACK NAME\m4a]
| *.m4a - recoded tracks
| ...
+-- [TRACK NAME\m4a]
| *.png - spectrograms of all .m4a files
| n-cmp_*.png - comparative spectrograms
| ...
L---
And here are (https://docs.google.com/spreadsheets/d/1vTu7mG7IlY0otsadateDg400rjYhA_b7ewwDOmQk1g4/edit?usp=sharing) the comparative tables of all tracks (in cell D1 you can enter the name of the encoder)
It is difficult to draw conclusions, but for myself I drew conclusions

that ffmeg isn't that bad
that the old fdkaac-0.6.3 is much better than the new fdkaac-1.0.4
that ...

Or is it all nonsense and I just wasted my time and yours?

Thanks to all.

[chainring]

I want to know what the community thinks about this method of comparison?

Do you listen to a spectrogram analysis or do you listen to music? Let your ears, not your eyes, be the guide.

Hate to tell you this, but people before you have done similar and it's almost always shot down in a fiery ball of wasted time.

[dns]

Do you listen to a spectrogram analysis or do you listen to music? Let your ears, not your eyes, be the guide.

Hate to tell you this, but people before you have done similar and it's almost always shot down in a fiery ball of wasted time.

Actually, I rarely hear a difference. Only on certain compositions, but that doesn't mean I wouldn't want to find a better encoding option.

Let me tell you why I came up with this idea.
Since we live in the digital age and there are many programs: Google Assistant, Siri, Shazam, etc...
We are not talking about an analog signal here, but about a digital one, from which I concluded that it is possible to find an audio comparison algorithm without human ears.
And what I suggested, I could not find on here on the forum.

And thank you very much for your answer.

[john33]

It has been attempted a number of times without any great success. The problem is partly due to the fact that all lossy codecs generate artefacts, to a greater of lesser degree, and our individual sensitivity to those artefacts often colours our opinion as to which is a good encoder and which isn't. You can't program to deal with that. Blind listening tests are the only reliable test. Most public listening tests suggest that 'qaac', which uses the Apple encoder, is the best of the aac codecs.

[richardpl]

Using sox and imagetragick in these ages is story for itself.

[chainring]

Actually, I rarely hear a difference. Only on certain compositions, but that doesn't mean I wouldn't want to find a better encoding option.

Let me tell you why I came up with this idea.
Since we live in the digital age and there are many programs: Google Assistant, Siri, Shazam, etc...
We are not talking about an analog signal here, but about a digital one, from which I concluded that it is possible to find an audio comparison algorithm without human ears.
And what I suggested, I could not find on here on the forum.

And thank you very much for your answer.

I don't want to squash your enthusiasm, but I'd hate to see you waste time on something that's not going to do much other than satisfy your curiosity. Peruse the forums at Hydrogen Audio (http://hydrogenaud.io) for the testing procedures and lots of tests over the years.

[pandy]

For spectral comparison you may try to use https://junkerhq.net/MDFourier/
Different purpose but should work also for this task.

[VoodooFX]

I prefer this lil program to quickly look at spectrograms: http://spek.cc

To compare audio codecs you need to use your ears, spectrograms are useless for that.