eac3to - audio conversion tool

[madshi]

Thanks!

[nevcairiel]

And as I also said, I understand the reason it is made, but this is not a reason for changing reality of mathematics, and relying on only this check is not perfect. If you can rely on something else (e.g. metadata) please priotize this method over zeroes checking (zeroes checking should be only a fallback in the case there is not such metadata)

Well it really comes down to one simple argument:

Zeroes in the LSBs are useless ("empty" information), and therefor if metadata says 24-bit, but the signal consistently has 8 bits of zero at the end, can encode as 16-bit, save space, lose no signal information at all.
That seems really the information that madshi/eac3to is after.

[Zenitram]

Zeroes in the LSBs are useless

They are not. as 0 does not say the same thing as 0.0000 (and yes, quantization bits are similar to float).
I understand this is same for your ears, but it may be different for some compression algorithm and/or for people looking for the count of bits used for quantization (again, zeroes with 16-bit is not same as zeroes with 24-bit, zeroes with 24-bit quantization say that this is more sure that it is silence for real)

It is all about precision information, I understand that you don't care, but that does not mean it is useless for everybody. It is very important for a couple of people I work for.

[73ChargerFan]

I'm glad to see MediaInfo will recognize HDCD; now I can script a test for it and mark such albums in my collection. Thanks.

https://upload.wikimedia.org/wikipedia/en/thumb/4/4f/HDCD_logo.svg/148px-HDCD_logo.svg.png

[ndjamena]

I'm glad to see MediaInfo will recognize HDCD; now I can script a test for it and mark such albums in my collection. Thanks.

https://upload.wikimedia.org/wikipedia/en/thumb/4/4f/HDCD_logo.svg/148px-HDCD_logo.svg.png

It doesn't literally "detect" HDCD, it just reads a tag EAC3To has been adding to the FLAC files it produces.

If you have a HDCD FLAC/ALAC/WAVPACK file that DOESN'T have the tag MediaInfo will not detect it.

In that case, you could run your files through EAC3To or Foobar and get the tags added/add them yourself.

(MediaInfo has always displayed the HDCP tag, it just attached it to the file rather than the stream which means the tag was discarded if it was muxed into a Matroska file.)

[Music Fan]

You don't seem to be interested by the information I mentioned yesterday, your HDCD files are maybe not really HDCD, look at the link I gave in this post ;
http://forum.doom9.org/showthread.php?p=1746126#post1746126

[Zenitram]

You don't seem to be interested by the information I mentioned yesterday, your HDCD files are maybe not really HDCD, look at the link I gave in this post ;
http://forum.doom9.org/showthread.php?p=1746126#post1746126

And there is a link to a post saying that this is not correct (http://www.head-fi.org/t/151329/hdcd-technology-in-detales#post_11243554).
And now?

[Music Fan]

Interesting, this guy is not 100 % certain about it but he has good arguments.
If he is right, I wonder why MvB heard differences between the 2 ripping methods (with and without subchannels), admitting he was honest and not trying to discourage people to copy HDCDs.

[madshi]

FWIW, eac3to contains a user written HDCD decoder which seems to work just fine on WAV source files. I've no idea how the HDCD decoder works internally, though, nor can I guarantee whether it's 100% complete and accurate.

[Music Fan]

Ok, I never tried to decode HDCD files with eac3to, I guess the goal is to produce 24bit wav to keep HDCD quality without needing HDCD player ; in this case, one shouldn't specify the command -down16, right ?

[madshi]

If your target is lossless, eac3to by default doesn't decode HDCD, but leaves it untouched, so input and output bitdepth stays at 16bit. You can force HDCD decoding by using the "-decodeHdcd" switch. Or if you transcode to a lossy format, eac3to automatically decodes HDCD, so that the input to the lossy decoder has the highest possible quality. If you want to decode HDCD, and preserve its full quality, "-down16" is obviously not a good idea.

[Music Fan]

If your target is lossless, eac3to by default doesn't decode HDCD, but leaves it untouched, so input and output bitdepth stays at 16bit.
In this case, why to use eac3to with HDCD wavs ?
If one needs Flac, I guess the HDCD information will be lost (except if -decodeHdcd is used and -down16 is not, therefore you get 24 bit Flac with HDCD quality).

If you want to decode HDCD, and preserve its full quality, "-down16" is obviously not a good idea.
Ok, does its full quality correspond to 20 or 24 bit ?

[madshi]

Why would HDCD get lost when using FLAC? It doesn't. FLAC is lossless.

[Music Fan]

Do you mean that if -decodeHdcd is NOT used, and if the target format is Flac 16 bit, the Flac will still include the HDCD information that can be decoded as HDCD Flac ?

[madshi]

That's what I just said.

[Music Fan]

I'm surprised because I wonder how the HDCD information -which is done for LPCM- can be kept in Flac.
IIRC, HDCD encoding use random bits and I don't see how a FLac encoder can recognize this kind of pattern and keep it. It is lossless, ok, but its structure is different than LPCM.

[madshi]

What part of "lossless" do you not understand? FLAC is like a zipped WAV/LPCM file.

[ndjamena]

http://www.audiomisc.co.uk/HFN/HDCD/Enigma.html
https://en.wikipedia.org/wiki/Compact_Disc_Digital_Audio

It's possible he's right. CD's aren't strictly LPCM, they have a variable gain control.

Some CDs are mastered with pre-emphasis, an artificial boost of high audio frequencies. The pre-emphasis improves the apparent signal-to-noise ratio by making better use of the channel's dynamic range. On playback, the player applies a de-emphasis filter to restore the frequency response curve to an overall flat one. Pre-emphasis time constants are 50µs and 15µs (9.49 dB boost at 20kHz), and a binary flag in the disc subcode instructs the player to apply de-emphasis filtering if appropriate. Playback of such discs in a computer or 'ripping' to wave files typically does not take into account the pre-emphasis, so such files play back with a distorted frequency response.[

[Music Fan]

Interesting. But I believe HDCD and CD with De-emphasis are 2 different things and thus need different ripping method.
Anyway, I kept searching informations and it seems that when ripping HDCDs in a simple way (copying waves instead of making ISO keeping cd's sub channels), a part of the needed information to decode properly HDCD is not retained.
I found a post that summarizes clearly everything I read on this subject ;
http://forums.stevehoffman.tv/threads/standalone-hdcd-decoder.224268/#post-5695519
Foobar2000 and dBpoweramp HDCD detection & decoding are based on HDCD.exe, which was created by emulating Windows Media Player. What WMP does not do (and consequently, neither does HDCD.exe) is decode the transient filter function. Although, it can detect it.
And the transient filter information is apparently in the sub channels I evoked earlier.

It means that the HDCD conversion is done but not exactly how it should be.

[madshi]

Regardless of whether that's true or not, none of this has anything to do with eac3to, because eac3to does not rip CDs or ISOs. eac3to does the best it can with the data it gets. If something is lost when ripping a CD to WAV then this loss has happened before eac3to was involved. There is no (further?) loss when using eac3to, as long as the audio data is kept lossless. So this is my last post about this topic.

[Music Fan]

I don't really understand your reaction because as your program is supposed to decode HDCD, and if you are curious about all that stuff (and I guess you are, otherwise you wouldn't have created this tool), you should be happy to learn these things and maybe thank people who take time to understand how your tool handle a particular format and make research about it.
People who use your program deserve to know its limits which should be clearly mentioned on this topic or in the help of eac3to.

[Thunderbolt8]

eac3tos was created to be able to decode/deal with HD movies and their audio tracks. HDCD is not really part of that.

[Yoshi]

eac3to v3.30 released

http://madshi.net/eac3to.zip

* libDcaDec is now default for all DTS tracks except XSA / low bitrate

At least for two Blu-ray sources I already encountered after converting stuff for not even a day, this turns out to be a bad mistake:

In the case of the US release of "Ex Machina", the libDca decoder produces bad clipping and with the German release of "Still Alice", it produces white noise on all channels instead of the movie soundtrack. :(

Here you can see three times the left channel of the "Ex Machina" 7.1 DTS-HD MA source - 1. processed with eac3to and decoded by the ArcSoft decoder (1.1.0.7), 2. processed with eac3to and decoded by the libDca decoder and 3. processed with MakeMKV and decoded by the libDca decoder.

http://img5.fotos-hochladen.net/uploads/arcsoftvslibx4z5lo69rf.jpg

The first waveform is clipped as well, but this is most likely already contained in the source - the usual nowadays mastering stupidity. :( The second and third are identical, so it's obvious whose fault this is.

Maybe I missed something but did anyone actually test that libdca crap before deciding to use it? :angry:

Same question to the MakeMKV developers which seem to have fallen into the same trap. At least, eac3to can be forced to still use the ArcSoft-decoder.

[nevcairiel]

You should first stay calm and not result to insulting peoples hard work in providing a free and open decoder for a format thats been hard to handle completely for a long a time.
Of course its tested, but not every edge case can be thoroughly covered without access to every single disc on the planet, and if the source has baked in clipping, it could result in all sorts of problems - like this. Clearly clipping behavior can be improved by simply flat-lining the clipping instead of overflowing it, but I even saw a change related to that in newer libdcadec versions, so maybe its already resolved, just eac3to not updated yet.

On top of all that, if you can provide a short sample that illustrates such a problem, the developers will likely be happy to investigate and resolve these issues.

[ndjamena]

EAC3To should probably detect that and then run a second pass to normalise it, something that's probably not possible with Arcsoft.

libDcaDec is open source so can be modified, maybe to output 32 bit.

But the problem in this case is obviously in the audio stream itself.

[Yoshi]

You should first stay calm and not result to insulting peoples hard work in providing a free and open decoder for a format thats been hard to handle completely for a long a time.

My excuses, you're right. This was written right out of frustration about the screwed up rips which resulted in using the decoder which is supposed to replace the commercial add-ins required before. And to be fair, the ArcSoft decoder - depending on the version - isn't "perfect" either, at least with eac3to (the MakeMKV creators claim that by addressing the library directly, there are no issues at all with it, including 'strange 7.1 setups').


Of course its tested, but not every edge case can be thoroughly covered without access to every single disc on the planet, and if the source has baked in clipping, it could result in all sorts of problems - like this.

Sorry, but with all due respect to all the developers and their hard work - we are talking about a lossless codec system here. DTS-HD MA might me more complex due to its core and extension architecture, but for a proper decoder dealing with a lossless format, it must not matter one bit how screwed up the source is content-wise, meaning the raw PCM source, not damaged frames or headers which might lead to problems. In the latter case, one decoder might indeed be more tolerant than the other. Clipping in the source is not an excuse for a lossless decoder to mess up the resulting PCM completely.

If a decoder doesn't reproduce the original data of a losslessly encoded source, then it's faulty. There is absolutely no tolerance to that.


Clearly clipping behavior can be improved by simply flat-lining the clipping instead of overflowing it, but I even saw a change related to that in newer libdcadec versions, so maybe its already resolved, just eac3to not updated yet.

As I stated above, I don't see the tolerance of a lossless decoder regarding the reproduction of the PCM source.


On top of all that, if you can provide a short sample that illustrates such a problem, the developers will likely be happy to investigate and resolve these issues.

I'm certainly willing to contribute here, otherwise I wouldn't take the effort to compare and make screenshots. :cool:

[nevcairiel]

Sorry, but with all due respect to all the developers and their hard work - we are talking about a lossless codec system here. DTS-HD MA might me more complex due to its core and extension architecture, but for a proper decoder dealing with a lossless format, it must not matter one bit how screwed up the source is content-wise, meaning the raw PCM source, not damaged frames or headers which might lead to problems. In the latter case, one decoder might indeed be more tolerant than the other. Clipping in the source is not an excuse for a lossless decoder to mess up the resulting PCM completely.

Unfortunately, its not quite as simple as that. Without having the PCM source, we don't know what is "lossless", and when there is baked in clipping like this, its possible that no decoder is going to produce an exact copy of the source, since before encoding, it may not have had this clipping (maybe because it was higher bitdepth, somehow).

The problem in your screenshots is a simple one really. The decoder decodes the signal perfectly, its just that the decoded signal does not fit into the defined integer range and instead "wraps" into negative (just how integers work, too high positive numbers suddenly turn negative).
You can actually see the waveform "continue" in the negative - which clearly sounds terrible, but it gives us strong hints to whats going on.

So, the answer is instead of decoding it "perfectly", to actually clip the signal after decoding, so you get flatlines instead. Unfortunately a lot of information about DTS-HD MA has to be reverse engineered or "guessed" based on the behavior of other decoders, like ArcSoft.

So in short, the best we can do is "lossless to ArcSoft", or any other reference decoder, since thats the only data points we have (unless someone can encode something with such a problem and provide the original PCM)
Clearly this is a bug, and should be fixed, but I find it important to clarify a bit on how it came to be.

Anyway, like I mentioned earlier, a change was already performed in libdcadec to perform more aggressive clipping, which may just handle this particular sample already.
If you can cut a small segment with the problematic areas from the two Blu-rays you have been having issues with, we can make sure, and/or get them fixed - and test losslessness to ArcSoft after any potential fixes.

Its everyones goal here to provide proper "lossless" decoding on all sorts of broken samples, of course.

[Yoshi]

Unfortunately, its not quite as simple as that. Without having the PCM source, we don't know what is "lossless", and when there is baked in clipping like this, its possible that no decoder is going to produce an exact copy of the source, since before encoding, it may not have had this clipping (maybe because it was higher bitdepth, somehow).

Very interesting remarks. I have to admit that I'm not familiar with the internal structure of the encoding and decoding process of DTS-HD(MA) in detail except for the core and extension concept. If I understand you correctly, there might be inputs, DTS-HD(MA) won't be able to handle properly. To my best understanding, this goes against all what makes up a lossless codec in the first place. Maybe a sacrifice of making DTS backward-compatible this way.

But let's leave source-clipping aside and have a look what libDca does to the calm and innocent intro of "Still Alice" instead. ;)

http://img5.fotos-hochladen.net/uploads/arcsoftvslib5knl6o21uv.jpg

Regarding the test files, I'll send you a PM shortly.

[nevcairiel]

That one sure is weird. Will be interesting to see. ;)

[torturesauce]

Ex Machina has a new codec called DTS:X (not to be confused with DTS Headphone:X, which is also included on the disc). It is the rival format to Dolby Atmos. Therefore, the DTS-HD MA 7.1 is actually the core track. More info here: http://www.blu-ray.com/movies/Ex-Machina-Blu-ray/128113/#Review

Maybe this is the reason libdcadec produces inaccurate results?

[ndjamena]

Turbo:

MKV, 1 video track, 1 audio track, 1:35:41, 24p /1.001
1: h264/AVC, English, 1080p24 /1.001 (16:9)
2: DTS Master Audio, English, 7.1 (strange setup) channels, 24 bits, 48kHz
(core: DTS-ES, 5.1 channels, 1509kbps, 48kHz)
With the core extracted:

MKV, 1 video track, 1 audio track, 1:35:41, 24p /1.001
1: h264/AVC, English, 1080p24 /1.001 (16:9)
2: DTS-ES, English, 5.1 channels, 1509kbps, 48kHz

So... when is DTS-ES still regular 5.1...

...or is this likely a bug in the DTS encoder suite?

[ndjamena]

Ex Machina has a new codec called DTS:X (not to be confused with DTS Headphone:X, which is also included on the disc). It is the rival format to Dolby Atmos. Therefore, the DTS-HD MA 7.1 is actually the core track. More info here: http://www.blu-ray.com/movies/Ex-Machina-Blu-ray/128113/#Review

Maybe this is the reason libdcadec produces inaccurate results?

Yeah, the audio is being mixed on the fly, so there isn't any lossless source.

[ndjamena]

Very interesting remarks. I have to admit that I'm not familiar with the internal structure of the encoding and decoding process of DTS-HD(MA) in detail except for the core and extension concept. If I understand you correctly, there might be inputs, DTS-HD(MA) won't be able to handle properly. To my best understanding, this goes against all what makes up a lossless codec in the first place. Maybe a sacrifice of making DTS backward-compatible this way.

But let's leave source-clipping aside and have a look what libDca does to the calm and innocent intro of "Still Alice" instead. ;)

http://img5.fotos-hochladen.net/uploads/arcsoftvslib5knl6o21uv.jpg

Regarding the test files, I'll send you a PM shortly.

This second one looks like it's been amplified. Far too much gain has been applied.

[nevcairiel]

But let's leave source-clipping aside and have a look what libDca does to the calm and innocent intro of "Still Alice" instead. ;)


It seems to switch from 16-bit to 24-bit decoding here along the way, probably due to a change in the bitstream. It almost seems like eac3to doesn't handle this case properly, and not dcadec.
I have let madshi know so he can chime in here.

Other software using libdcadec like my own LAV Filters or ffmpeg can decode this file fine.

[NikosD]

Puzzled about this TrueHD sample:
https://www.sendspace.com/file/6cn8ub

I tried to convert it to AC3 using eac3to v3.30 source.thd dest.ac3 and the app crashed

TrueHD, 5.1 channels, 48kHz, dialnorm: -27dB
Removing TrueHD dialog normalization...
Decoding with libav/ffmpeg...
Remapping channels...
Encoding AC3 <640kbps> with libAften...
Initialization of the AC3 encoder failed.
Aborted at file position 262144.

Using eac3to v3.29 source.thd dest.ac3 works fine.

TrueHD, 5.1 channels, 48kHz, dialnorm: -27dB
thd, 48000, 5.1
Removing TrueHD dialog normalization...
Decoding with libav/ffmpeg...
Remapping channels...
Encoding AC3 <640kbps> with libAften...
Creating file "test329.ac3"...
The original audio track has a constant bit depth of 16 bits.
eac3to processing took 9 seconds.
Done.

[nevcairiel]

In the case of the US release of "Ex Machina", the libDca decoder produces bad clipping

I checked a sample provided by Yoshi of this track, and the new libdcadec version produces a bit-identical result to ArcSoft, so as I suspected the clipping issue was already fixed. Just needs an updated version of eac3to with a new dcadec.

[SeeMoreDigital]

Puzzled about this TrueHD sample:
https://www.sendspace.com/file/6cn8ub

I tried to convert it to AC3 using eac3to v3.30 source.thd dest.ac3 and the app crashed

Using eac3to v3.29 source.thd dest.ac3 works fine.

I'm able to confirm this observation too, using your TrueHD sample...

[Boulder]

I checked a sample provided by Yoshi of this track, and the new libdcadec version produces a bit-identical result to ArcSoft, so as I suspected the clipping issue was already fixed. Just needs an updated version of eac3to with a new dcadec.Is it possible to build libdcadec.dll outside of eac3to?

[nevcairiel]

Is it possible to build libdcadec.dll outside of eac3to?

I suppose it is, if its an unmodified version, which it probably is.

[madshi]

Yes, it is, but the dcadec interface changed a bit, allowing warnings now instead of errors. Anyway, I'll release a new build with an updated dcadec soon.

[Yoshi]

I checked a sample provided by Yoshi of this track, and the new libdcadec version produces a bit-identical result to ArcSoft, so as I suspected the clipping issue was already fixed. Just needs an updated version of eac3to with a new dcadec.

Many thanks for checking on this! After you successfully irritated me regarding how lossless lossless can be when dealing with essentially unknown sources (mixes) - what's the consensus, is the nominally correct (identical) PCM-result of libDca and ArcSoft considered to be the true reproduction or not?

The clipping which occurs in the "correct" result is probably already contained in the mix or mastering, isn't it?


@all: since the most recent version of MakeMKV produces the same faulty results, I tried to contact "mike" which seems to be the developer or at least part of the team. Stupidly, I can't register at that forum since my IP range is blocked for some reason which is beyond me.

Just in case, anyone knows if and how to enforce MakeMKV to enforce to still use the dtsdecoder.dll instead of the integrated libDca.

[nevcairiel]

Many thanks for checking on this! After you successfully irritated me regarding how lossless lossless can be when dealing with essentially unknown sources (mixes) - what's the consensus, is the nominally correct (identical) PCM-result of libDca and ArcSoft considered to be the true reproduction or not?

The clipping which occurs in the "correct" result is probably already contained in the mix or mastering, isn't it?

It seems to be possible to retrieve the audio without clipping by reducing the overall volume in this particular sample, if thats the desired goal is another question.
From what I understand, madshi wants to offer a two-pass option in eac3to to reduce the volume and re-process the file if such clipping issues are detected. It wouldn't be bitexact to ArcSoft or the source, but probably be better than clipping!

[Yoshi]

I don't understand this.

This is with that required volume reduction in general.

I'm aware of eac3to's two-pass-feature and understand the requirement for volume reduction depending on the source when downmixing a number of channels into fewer channels than the source since all channels could carry 0dBFS and have to fit into the lower range.

What I never understood is why this is sometimes required when dealing with lossy codecs without downmixing as I don't see the point why I have to reduce the volume just to reconstruct the lossy source.

Now with lossy codecs, you got transformation into the frequency domain and stuff and maybe due to errors introduced by the psychoacoustic model, you might get (intersample) peaks here and there asking for a few additional dB of headroom but why this shall still be an issue with lossless sources is beyond me.

By definition, there can be only one correct PCM result and audio level for any given lossless source. If that clipped, so shall the PCM of course. When lowering the decoded audio level for a clipped lossless source, I would expect the decoded result to clip as well at -x dBFS then, gaining nothing but slightly rising the noise level due to the lower SNR.

[nevcairiel]

Its probably just a mistake when encoding. Formats like DTS-HD are a bit more complex than a simple codec like FLAC, they have embedded downmixes for stereo and 5.1 and whatnot (and the decoder performs something called "downmix reversal" to get the 7.1 signal), so when all these features are used, its apparently easy enough to screw something up.

I'm sure there are also lossless encodes where the clipping is part of the original PCM, but in this particular sample it is possible to retrieve the audio without clipping, albeit at a slightly reduced volume to make room for the extra data.

[Thunderbolt8]

just to confirm, the US Blu-ray of Ex-Machina has indeed a 7.1 DTS:X (+ a DTS:X headphone) track. so this is perhaps why anything else than just extracting the track (well at least if this works as it should) could potentially result in non lossless output as the reis probably no support yet for these kind of tracks via libDAC or arcsoft (unless there are recent updates for arcsoft)

I checked a sample provided by Yoshi of this track, and the new libdcadec version produces a bit-identical result to ArcSoft, so as I suspected the clipping issue was already fixed. Just needs an updated version of eac3to with a new dcadec.just asking, does that mean that full DTS:X track processing is possible now with that version? or that just one specific error has been fixed?

[nevcairiel]

There are no decoders for DTS:X outside of the hardware receivers.
Note that such things as "bitexact" don't really apply to concepts like DTS:X or Dolby Atmos, as they are designed to be mixed for the speaker setup of the user, so there is no one correct way to "decode" it.

[Thunderbolt8]

so if you want to have the full Atmos and DTS:X information from an audio track, the only thing which makes sense is to extract the entire track and bitstream it to you receiver, correct?

[NikosD]

@madshi
@tebasuna51
@nevcairiel

Any comments on my TrueHD sample and AC3 conversion ?

Is it a bug of v3.30 ?

[nevcairiel]

so if you want to have the full Atmos and DTS:X information from an audio track, the only thing which makes sense is to extract the entire track and bitstream it to you receiver, correct?

Yes, you should always leave the track intact. Even if some day a software decoder appears, it'll still be the same deal - you need to tell it how your speakers are setup to mix the extra "3D" audio properly, so it should only do this conversion at playback time, and not when re-encoding.

[madshi]

eac3to v3.31 released

http://madshi.net/eac3to.zip

* libDcaDec: updated to latest build
* libDcaDec: decoding only aborts on critical issues now
* libDcaDec: now reports warnings if something isn't 100% perfect
* libDcaDec: proper handling of clipped files (2nd pass etc)
* libDcaDec: proper handling of tracks that switch bitdepth 16 <-> 24
* fixed: TrueHD decoding -> AC3 encoding didn't work properly