HALAC (High Availability Lossless Audio Compression)

[Hakan Abbas]

I found out about this forum a little late, but I'm happy to make my first post.

HALAC 0.5.7 is ready. Linux and Windows executables are in the link below.
https://github.com/Hakan-Abbas/HALAC-High-Availability-Lossless-Audio-Compression/releases/tag/0.5.7

* New parameter Usage : Console parameters are now entered using spaces. And for practical use, only the input name is sufficient for compression and decompression operations.
"encoder input_file.wav -o output_file.halac -m 1 -b 2048 -mt 8 -ow -s"

* New "-o" parameter for output

* New Silent Mode (-s)
By default, encoder and decoder provide more detailed information. This introduces a light processing overhead. It can be disabled with the "-s" flag.
https://private-user-images.githubusercontent.com/158841237/604638799-7e834647-88cf-4e21-a0b6-225132f80f4d.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3ODEwMjI5NTIsIm5iZiI6MTc4MTAyMjY1MiwicGF0aCI6Ii8xNTg4NDEyMzcvNjA0NjM4Nzk5LTdlODM0NjQ3LTg4Y2YtNGUyMS1hMGI2LTIyNTEzMmY4MGY0ZC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjYwNjA5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI2MDYwOVQxNjMwNTJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1mYzQ1MWZlYjZkMDVjNWEwMTM0NTk0ZDI2MTUzYTAzYjYwNzFhMjE0ZDc5ZDQyNDY4MDUxODhlZDJlMDU2YTIzJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZyZXNwb25zZS1jb250ZW50LXR5cGU9aW1hZ2UlMkZwbmcifQ.ullw2_vRVRTAHhHJrRmLhbJ95kzxFTsq8P06zIQGZdg

* New Compression Levels (0...5)
The "-m 0" mode has been added, which is a faster mode than the old ultrafast mode. The old ultrafast mode is "-m 1". The old normal mode is now "-m 3". Also added "-m 4" which is slightly faster than the old plus mode. The old plus mode is now "-m 5". A few more higher levels may be added in the future.

* New lossyWAV parameter (-lw)
It treats the input as lossyWAV and makes the necessary adjustments. In the new case, WAV header information is no longer taken into account. And the decoder always knows that the input is lossyWAV. In its current draft, without "-lw" the results are not very good. However, the structure I prepared for the next version gives good results even without "-lw".

Structural changes required for correlated support for up to 64 channels are ongoing. The number of channels could be increased further, but I don't know if this is necessary.

AMD Ryzen 9600X.
Input is 2,154,029,144 bytes 6 wave files. (2 ch, 24 bit, 48khz to 384khz)

// Single thread results. Disc read&write times are included.
-m 0 : 1.881s 2.508s 1,231,112,058 bytes
-m 1 : 2.085s 2.734s 1,028,551,027 bytes
-m 2 : 2.237s 3.347s 862,622,884 bytes
-m 3 : 2.532s 3.642s 784,674,531 bytes
-m 4 : 4.244s 4.734s 769,782,785 bytes
-m 5 : 4.741s 4.974s 766,978,010 bytes

// Multithread results. Disc read&write times are included.
-m 0 -mt 12 : 0.872s 1.446s 1,231,112,058 bytes
-m 1 -mt 12 : 0.913s 1.550s 1,028,551,027 bytes
-m 2 -mt 12 : 0.864s 1.679s 862,622,884 bytes
-m 3 -mt 12 : 0.906s 1.734s 784,674,531 bytes
-m 4 -mt 12 : 1.091s 1.596s 769,782,785 bytes
-m 5 -mt 12 : 1.153s 1.676s 766,978,010 bytes

[DJ Bobo]

Wow! 5x faster than FLAC and 7x faster than ALAC, that's amazing.
I hope you reach v1.0 soon and find wide support.
As you know, broad support from hardware manufacturers and software editors is key to popular adoption!

Wasn't there a way to include your speed optimizations within FLAC itself though?!
I recall here the Gogo-no-coda MP3 encoder, a fork of LAME that was at least 2x faster than standard LAME, while outputting standard MP3 files playable everywhere.

[Hakan Abbas]

Wow! 5x faster than FLAC and 7x faster than ALAC, that's amazing.Thank you very much. Yes HALAC is very fast. And it offers this speed difference with a pretty good compression ratio. At this point it is on average equivalent to FLAC.

I hope you reach v1.0 soon and find wide support.
As you know, broad support from hardware manufacturers and software editors is key to popular adoption!I hope so. Unfortunately, there is not much I can do other than making HALAC better. Since the same thing happened to HALIC (https://github.com/Hakan-Abbas/HALIC-High-Availability-Lossless-Image-Compression), I cannot say that I am very lucky in this regard.

Wasn't there a way to include your speed optimizations within FLAC itself though?!FLAC has a completely different structure. When I started my studies, I had no knowledge about the structure of FLAC. Most things won't technically be compatible with FLAC anyway. Because FLAC's backward support needs to continue. Most structural changes cannot be made in FLAC. (No one specifically asked for such support) However, since HALAC is in the development stage, all kinds of innovations can be added.

[DJ Bobo]

Since the same thing happened to HALIC
May be it's the version number that is off-putting?
I wouldn't want to include a 0.x version in my product either. It gives the impression that it is work in progress.
If there is still work to be done, finish it and bump it to v1.0, and then try to integrate in open source products.

[Hakan Abbas]

May be it's the version number that is off-putting?
I wouldn't want to include a 0.x version in my product either. It gives the impression that it is work in progress.
If there is still work to be done, finish it and bump it to v1.0, and then try to integrate in open source products.

What needs to be done with HALAC is not quite finished. But there isn't much left. I'll evaluate it once some of the main features are completed.

[GeoffreyA]

Hi Hakan,

I haven't tested the encoder yet, but taking a look at the code, I think it would benefit from splitting out the processing functions into a library separate from the CLI.

[Hakan Abbas]

Hi Hakan,

I haven't tested the encoder yet, but taking a look at the code, I think it would benefit from splitting out the processing functions into a library separate from the CLI.

Current codes belong to the first version of HALAC (0.1.9). Of course the current code base is much more specific.

[GeoffreyA]

All right. I didn't realise the version difference.

[Z2697]

Wow this is super fast, and with nice compression!
I don't think lossyWAV is that important, it's like niche in a niche, or I'm just not knowing much.

[Hakan Abbas]

Wow this is super fast, and with nice compression!
I don't think lossyWAV is that important, it's like niche in a niche, or I'm just not knowing much.

Thank you.
There are those who especially prefer and use lossyWAV. Yes, it is a niche study. It is known to be very good in terms of quality. That's why I think it's important to support.

[Z2697]

Thank you.
There are those who especially prefer and use lossyWAV. Yes, it is a niche study. It is known to be very good in terms of quality. That's why I think it's important to support.

I mean it's good to let it just happen "naturally", because that means there's a good wasted bits support.

[Hakan Abbas]

I mean it's good to let it just happen "naturally", because that means there's a good wasted bits support.
Yes, HALAC makes good use of this feature of lossyWAV. Besides, FLAC, WMAL and TAK can also be made good use of. However, unfortunately other lossless codecs are not very good in this regard.

[DJ Bobo]

I don't think lossyWAV is that important, it's like niche in a niche, or I'm just not knowing much.
Totally agree.
I don't think there is anybody sane that would take a lossy wave file and try to compress it with a lossless encoder.
Does not make sense to me whatsoever.

[john33]

Totally agree.
I don't think there is anybody sane that would take a lossy wave file and try to compress it with a lossless encoder.
Does not make sense to me whatsoever.

LossyWAV is designed to be used with FLAC and other suitable lossless codecs, it's the whole point of it.

[Hakan Abbas]

For lossyWAV I just added a mechanism to check the meaningless bits. And then adaptive block size. So I guess I didn't do much. I couldn't refuse requests from a certain user base on this very non-specific subject.
However, I have an idea for a high quality lossy solution that is completely different and also works very fast. I might be interested if I can find the time and motivation.

[hello_hello]

Hakan Abbas,

I added HALAC as an encoder to foobar2000 and it's working fine, but decoding, not so much.
Foobar2000 has a command line decoder wrapper component that can be configured to decode with just about any command line decoder, but I must be doing something silly because I haven't had any luck getting it to decode. Does the decoder require a specific command line and can it pipe the output?

Cheers.

[Hakan Abbas]

Hakan Abbas,

I added HALAC as an encoder to foobar2000 and it's working fine, but decoding, not so much.
Foobar2000 has a command line decoder wrapper component that can be configured to decode with just about any command line decoder, but I must be doing something silly because I haven't had any luck getting it to decode. Does the decoder require a specific command line and can it pipe the output?

Cheers.

Encoding operations can be performed by defining an encoder in foobar2000. But I couldn't do the opposite either. However, before, I had produced .dll and .so files for foobar2000 and other players. They worked flawlessly.
A valuable developer (@Case) even added a special component (https://foobar.hyv.fi/?view=foo_input_halac). I even developed a special audio player for this. (https://github.com/Hakan-Abbas/HALAC-Audio-Player) Since there were so many structural changes afterwards, I didn't bother producing the .dll/.so each time. This will probably become clear in the next version.

[DJ Bobo]

@ john33
Never mind. I've got confused about the concept for a moment (It was thinking about Wavpack). It's been a while since I touched that subject. I never touched it again, as my FLAC use is usually only for temporary files, and for anything compressed, I have an allergy to bitrates over 256kbps for stereo :D

[hello_hello]

I thought I'd have another try, and for whatever reason I have decoding working today even though I'm certain I would have tried the same configuration yesterday, more than once.

I thought I'd push my luck and also try saving tags. I assume there's no way to specify the tags being saved as foobar2000 wouldn't be familiar with HALAC, so I also assume it defaults to ape tags for unknown formats, and while I don't actually know how they're being read, they seem to be okay.

Commandline Decoder Wrapper Configuration
https://i.imgur.com/J58eb5U.png

I can't remember... is the audio MD5 in the first screenshot below something the FLAC encoder calculates, or Foobar2000?

FLAC
https://i.imgur.com/6DHdnyy.png

HALAC
https://i.imgur.com/JnaY7xC.png

[Hakan Abbas]

I thought I'd have another try, and for whatever reason I have decoding working today even though I'm certain I would have tried the same configuration yesterday, more than once.

I thought I'd push my luck and also try saving tags. I assume there's no way to specify the tags being saved as foobar2000 wouldn't be familiar with HALAC, so I also assume it defaults to ape tags for unknown formats, and while I don't actually know how they're being read, they seem to be okay.

It's really nice that you can do something like this with metadata.


I can't remember... is the audio MD5 in the first screenshot below something the FLAC encoder calculates, or Foobar2000?
It is much more likely to be the result of the hash produced by FLAC for general control purposes.

[hello_hello]

It is much more likely to be the result of the hash produced by FLAC for general control purposes.

It's the hash calculated for the audio itself, so you should be able to convert between lossless formats without the MD5 changing, assuming the audio remains unmolested, of course. An MD5 is also calculated for wavepack.
I thought maybe it'd be calculated for other types of lossless audio too, but apparently not.

By the way... based on the info I read for the command line decoder wrapper, when you select a tag type while configuring it to use a decoder, you're actually specifying the type of tag you want FB2k to write to the specified file/audio type. I mention this as writing and reading ape tags has so far worked as expected, but not so much ID3v2 tags. Writing them corrupts the files in some way and FB2K won't play them, so I thought I'd let you know in case you care. I haven't tried ID3v1.

flac CLI.
https://i.ibb.co/WvNKSgDj/flac.png (https://imgbb.com/)

ffmpeg outputting flac.
https://i.ibb.co/pjbJr302/ffmpeg-flac.png (https://imgbb.com/)

wavepack.
https://i.ibb.co/0Rb486f1/wavepack.png (https://imgbb.com/)

[Hakan Abbas]

It's the hash calculated for the audio itself, so you should be able to convert between lossless formats without the MD5 changing, assuming the audio remains unmolested, of course. An MD5 is also calculated for wavepack.
I thought maybe it'd be calculated for other types of lossless audio too, but apparently not.

By the way... based on the info I read for the command line encoder wrapper, when you select a tag type while configuring it to use a decoder, you're actually specifying the type of tag you want FB2k write to the specified file/audio type. I mention this as writing and reading ape tags has so far worked as expected, but not so much ID3v2 tags. Writing them corrupts the files in some way and FB2K won't play them, so I thought I'd let you know in case you care. I haven't tried ID3v1.


Thanks for all this valuable information.