DynamicAudioNormalizer

[LoRd_MuldeR]

And all things which are widely employed do become some kind of a standard over time. If you are the only one in the world who does it "correctly", but at the cost of some functionality (like not being able to process long files where an intermediate WAV file is >4GB), and the rest of the world uses "illegal" hacks which actually work even if they are illegal, what have you gained then? Just the warm fuzzy feeling that you have done it correctly, while all the others say "Whatever works...".

Just because something has become a "de-facto standard" after somebody started using it, due to the lack of a proper solution, does not make it a "good" solution that we should advocate.

There are quite a lot of "de-facto standards" that are quite horrible ;)


As far as the original question is concerned:

I have changed the Audio I/O module so that you can explicitly specify the output format, even when writing to a pipe. We no longer force "raw" PCM in that case. But, coming as no surprise, libsndfile fails to write WAV to a pipe:
"Error : this file format does not support pipe write."

Please note that libsndfile supports other file formats (e.g. AU) that can be written to pipe without problem. Actually, the AU formats looks like a promising candidate for this kind of task.


But you uses already this "hack" when write wav files > 4GB hoping that the other side will be able to deal with this situation.

You can output W64 or RF64 files instead "fake" WAV, at least ffmpeg, not all audio soft, can manage them.

These are actually two different problems. The 4 GB file size limit of WAV/RIFF format is because of the 32-Bit size fields. And, indeed, with W64 there exists a proper solution for that issue.

DynamicAudioNormalizer supports W64 (and also RF64) just fine, thanks to libsndfile-based Audio I/O.

The fact that RIFF/WAV (and similarly W64, by the way) do not support streaming, is more fundamental. Those are purely file-based formats, never designed with streaming in mind.


All audio soft have parser to read wav header and is easy ignore some fields, there are many fields ignored, some redundants, in wav headers.

We need only Format (bitdepth), Samplerate and Mask-channels (better than Num-channels than force to assume a default Mask-channel).

It's actually not quite that simple :)

WAV/RIFF is not simply a global header followed by "raw" data. Instead, its a sequence of "chunks", where each chunk has its own header with size + type. To correctly parse such file, you must consider the size of each chunk, because it is the only way to know where one chunk ends and where the next one starts. Also, you must consider the type of each chunk, because you must skip chunks that you don't know (or don't care about) and find the chunk you actually care about.

Surely, you can simply ignore all that and assume that the given WAV/RIFF is the most simple form of a WAV/RIFF file that can exist. Indeed, this will appear to work as long as you really got the most simple form of a WAV/RIFF file. But it is going to fail in all other cases! Be aware that there could be many other chunks (http://www.robotplanet.dk/audio/wav_meta_data/) present in the WAV/RIFF file, before or after the "data" chunk that contains the actual PCM data. See example here (https://i.imgur.com/6Px8VUj.png).

This is also the reason why the "ignore length" hack always needs to be enabled manually by the user - and only when absolutely needed. You could never make that the default behavior, as it would fail on many legit WAV/RIFF files.

[manolito]

This is also the reason why the "ignore length" hack always needs to be enabled manually by the user - and only when absolutely needed. You could never make that the default behavior, as it would fail on many legit WAV/RIFF files.

Do you have any examples where this would fail? (Not just a RIFF chunk list, but a real file)

I always use these "ignorelength" or "readtoeof" parameters in SoX and Aften, now even in FFmpeg and qaac, and I never had any problems (with the exception that the progress indicator might show funny values).


Cheers
manolito

[tebasuna51]

WAV/RIFF is not simply a global header followed by "raw" data. Instead, its a sequence of "chunks", where each chunk has its own header with size + type. To correctly parse such file, you must consider the size of each chunk, because it is the only way to know where one chunk ends and where the next one starts.

Of course, but all wav parsers are ready to ignore all extrachunks between the "fmt " chunk (than define Format, Samplerate and Channels) and the "data" chunk than contain the PCM data.

Is also legal put extrachunks at end of "data" chunk but don't exist a crazy soft than put that in a wav > 4GB or in stdout.

Is really easy make a wav parser to manage the decisions, you have a sample in the source code of wav2w64 (https://forum.doom9.org/showthread.php?t=174279)

With only a few lines of code can read wav "legal" (with any chunk) and "illegal" files or piped (with any chunk between "fmt " and "data"), and output legal w64 files or pipe that header (accepted by ffmpeg even without the -ignorelength parameter).

[tebasuna51]

Do you have any examples where this would fail? (Not just a RIFF chunk list, but a real file)

I always use these "ignorelength" or "readtoeof" parameters in SoX and Aften, now even in FFmpeg and qaac, and I never had any problems (with the exception that the progress indicator might show funny values).

There are legal wav files (<4GB) than can have extrachunks at end of "data" chunk, using "ignorelength" or "readtoeof" the end chunks are considered like pcm data, and you obtain a end click.

Many audio editors can put these extrachunks.
Then the parameters "ignorelength" or "readtoeof" must be used only for wav > 4 GB or piped data.

[LoRd_MuldeR]

Do you have any examples where this would fail? (Not just a RIFF chunk list, but a real file)

A WAV file, by definition, is nothing but "a RIFF chunk list". There is an example file in the link I provided above. Also, the screenshot showing the RIFF chunk tree was taken from that example file. But here is another example of a simple WAV file I saved from Audacity (http://www.audacityteam.org/) audio editor, so nothing special or uncommon at all:
https://i.imgur.com/RIYwzAM.png

Obviously, if here you assume anything after the start of the "data" chunk (to the end of the file) is PCM data, you will get garbage at the end of the track.


I always use these "ignorelength" or "readtoeof" parameters in SoX and Aften, now even in FFmpeg and qaac, and I never had any problems (with the exception that the progress indicator might show funny values).

As said before, the "ignore length" hack is going to work on a very specific type of file or stream which uses a "WAV-format inspired" header followed only by "raw" data. It certainly is not going to work with all valid WAV/RIFF files! Also, even worse, there is no reliable way for an application to distinguish between a valid WAV/RIFF file and "raw" data preceded by "fake" WAV header - even though both cases require very different handling (as we have seen). That is exactly why you have to manually tell the application every single time - via command-line switch - that this file/stream is non-standard and requires special handling.

(For me, the need for this kind of manual workaround is the definition of a "hack" ;))

[manolito]

If it is true what tebasuna said (the worst that can happen will be a click at the end of the file) then I can probably live with it. And when using pipes nothing bad can happen at all.

I would have appreciated a link to a real WAV file which contains such extra chunks after the data so I could do some tests. You say that Audacity produces such files. Do you have to apply special settings in Audacity to obtain such a file, or does this always happen?


Cheers
manolito

[hello_hello]

That is exactly why you have to manually tell the application every single time - via command-line switch - that this file/stream is non-standard and requires special handling.

Then let's do that. ;)
I assume the alternative is having to manually provide the application every single time, via several command line switches, enough information about the raw audio for the application to process it? I can't write software, so my perspective is a user one, but from where I stand it seems a no-brainer.
I'm not sure a fake wave header has ever caused me problems, but once again from a user perspective (using a GUI) will the "proper" method improve or change my user experience in any way. If not... it makes it harder to care about hackery. From a user perspective, having the functionality now instead of not having it while waiting for a non-hacked implementation that mightn't be available for years seems kind of silly.

Personally I think the hack label should be removed for a rebranding. Sure it might still technically be a hack, but maybe a small warning on the packaging will suffice given the current "fake wave header" is almost as universally supported as the non-fake wave header will be, when the proper solution arrives.

[LoRd_MuldeR]

If it is true what tebasuna said (the worst that can happen will be a click at the end of the file) then I can probably live with it.

It is not the worst that can happen. The "click" sound is a symptom of the fact that, if you ignore the length field and read all the way to the end, then any chunks that follow after the "data" chunk (which is perfectly possible and common practice) are interpreted as PCM data - although they are not. The result is that you get an undefined number of corrupted samples (i.e. samples with undefined content) at the end of the file. It could manifest itself as anything between "silence" and a loud "buzz" sound that blows up your speakers. After all, undefined is... undefined. But, in any case, getting corrupted input samples from a perfectly valid WAV file clearly is not acceptable behavior.


And when using pipes nothing bad can happen at all.

This has nothing to do with pipes per se. The "problem" is: When the writing application produces a "fake" WAV header followed by an undetermined number of PCM samples, then the reading application must enabled the "ignore length" hack to deal with this non-standard situation. At the same time, the reading application must not enable the "ignore length" hack when reading normal correct WAV files, because this can cause problems with some perfectly legitimate WAV files.

The reading application's dilemma is: It can not know, from the data (header), whether this is a "fake" WAV header or a real (correct) WAV file, because the "fake" WAV header imitates the header of a real (correct) WAV file... This is why real world applications do not use the "ignore length" hack by default. They do "correct" parsing, unless you explicitly tell them, via command-line option, to enable the "ignore length" hack for this particular input.

(And, nope, you are not "save" when using a pipe. That's because sending a valid standard WAV file over a pipe is also possible - provided that the source application can determine the actual file size beforehand)


I would have appreciated a link to a real WAV file which contains such extra chunks after the data so I could do some tests. You say that Audacity produces such files. Do you have to apply special settings in Audacity to obtain such a file, or does this always happen?

Nope, I did not use any special settings. Just plain standard Audacity with "factory" settings. It will probably include the "id3" tag only, if the source files contained meta information, or if you edited the meta information in Audacity.

FWIW, here you can see a standard WAV file exported from Audacity, then encoded with NeroAAC - once with the "-ignorelength" option set and once without - and finally re-imported into Audacity:

https://i.imgur.com/p0E4d1zl.png (https://i.imgur.com/p0E4d1z.png)

(Version that was encoded with "-ignorelength" option and thus has corrupted samples at the end is at the bottom, if that isn't obvious ^^)

[tebasuna51]

It could manifest itself as anything between "silence" and a loud "buzz" sound that blows up your speakers.
You don't need scare, a extrachunk LIST, like the image show, in a wav 2.0 16 bits 44100 Hz is less than 7 ms, if that can damage your speakers...

(And, nope, you are not "save" when using a pipe. That's because sending a valid standard WAV file over a pipe is also possible - provided that the source application can determine the actual file size beforehand)

Is possible, but I don't know any decoder or other soft, like sox, than send extrachunks at end of audio data.

BTW this is your thread and your soft.
Sorry for disturb you.

[LoRd_MuldeR]

You don't need scare, a extrachunk LIST, like the image show, in a wav 2.0 16 bits 44100 Hz is less than 7 ms, if that can damage your speakers...

Well, the point was that you can get some "garbage" at the end of the file if you ignore the length. And, because you are interpreting something as PCM data that isn't PCM data, it is completely undefined how it will sound like. But it's not going to sound "nice", that's for sure. Some programs (e.g. Audacity) include meta data as "id3" chunk. ID3 tag can contain even artwork as JPEG image. This could easily map to a few seconds of playback time, I suppose.

But the duration of the glitch is not all that important. Even a very short glitch at the end of a track can already be annoying. If I pass my valid input file trough some tool and that tool adds a glitch to the end of the track, I call that a bug.

BTW this is your thread and your soft.
Sorry for disturb you.

Discussion went kind of off-topic, but I want to make it clear again: The CLI front-end relies on libsndfile for audio I/O, which handles all the "nasty" stuff like endianess conversion, float/integer conversion, many different file formats and so on. I don't intend to go away from that. And, as a matter of fact, libsndfile refuses to write WAV format to a pipe, which is not a big surprise. You can select other formats, like AU/SND or FLAC, if you don't like it "raw"...

[manolito]

I tried to reproduce these extra chunks thing with audacity, but I had to use the older version 2.03 because new versions did not run on my non-SSE2 CPU (again,,, what a surprise).

And guess what, I imported an MP3 with lots of tags and then exported it to a standard 16-bit WAV file without removing any tag information. And the resulting WAV had absolutely no extra chunks after the WAV data chunks.

So these extra chunks may be totally legal, but IMO they are absolutely sick and should not be there.

And even if you do get these artifacts at the end of the resulting file, in my work flow the audio will be muxed in the end with some video stream. And if the audio is longer than the video then these artifacts will (hopefully) be cut off.


For the DynAudNorm CLI version I do have my workarounds for handling long multichannel sources. And for general audio format conversions I have completely abandoned LameXP in favor of FFmpeg (using dmMediaConverter as a GUI). Same or better quality, much faster and more options like using a container as the source. And no problems whatsoever using long multichannel sources (where LameXP is broken as soon as a 6-ch source exceeds a length of 2h 4m).


Cheers
manolito

[LoRd_MuldeR]

So these extra chunks may be totally legal, but IMO they are absolutely sick and should not be there.

The RIFF format, and everything based on it (WAV, AVI, etc), is defined as a tree of nested "chunks". Whichever way you look at it, that's just how the format is designed.

Now, if you misinterpret this tree-structure as "global header followed by raw data all the way to the end of the file" (which it really isn't) then things are expected go wrong - and often will go wrong.

So you can't say that something that is fundamental to how to format is defined is "sick and should not be there", just because you don't like it :rolleyes:

Take it: Sending a "fake" WAV header followed by an undetermined amount of raw data is going to work, if and only if the reading application employs the "ignore length" hack - which in turn is asking for trouble when reading normal WAV files!

You simply can not get both things to work correctly at the same time with (pseudo) WAV format. That is the reason why the "ignore length" hack needs to be enabled manually by the user, and only when absolutely needed.

(If, on the other hand, we used a proper "streamable" format - for example AU format - then no such hackery would be needed, because the format actually supports what we are trying to do)

[hello_hello]

I don't see a distinction, as pretty much every encoder I've configured for a foobar2000 preset uses "ignore length" and accepts stdin with a fake wave file.

stdin and "ignore length" work for every source and doesn't require advanced knowledge of the audio sample rate or channel count etc, assuming the encoder understand it..
Without the hack, each input needs to have information added to the command line describing the raw audio. I know which is easier when working with a GUI, but a question (although I appear to be suffering from some sort of invisibility), from a user perspective what's the difference between having to remember not to specify bitdepths and sample rates etc when the hack is used, and remembering not to use "ignore length" when the output is raw. For most applications it's probably in the manual, and you'd hope it says "these options only apply to raw input", just as you'd hope there'd be a description for "ignore length" which includes something like "only for fake wave" or however it's usually described.
The inclusion of raw input command line options proves the encoder isn't supposed to know everything, and without the hack an additional burden on a user who's not to be trusted with "ignore length".

If the audio is being decoded to temporary wave files along the way, there's no longer an option to convert two, four or even more simultaneously as the hard drive wouldn't keep up, and knowing when to use a temporary wave file as input is possibly just as important as knowing when to use the "ignore length" or "raw input only" command line options, I assume.

When you pipe to an encoder with DynamicAudioNormaliser, the temporary wave file count doubles, doesn't it? With the hack, the hard drive is free to write several lossy output files as their processed simultaneously, not having wave creation to worry about.

If you're doing it all by the command line it might be a different story, but I guess you have the official command line compressor if you're using the command line, and ffmpeg with the compressor included for GUI users. Without ffmpeg I think the CLI compressor would be far less practical for GUI use.

Foobar2000 has a command line decoder DSP that gives it the ability to decode with any CLI decoder, which is handy when there's no native decoding for a particular type of audio. I use it to decode EAC3 with ffmpeg. I can pipe and the audio loads into a playlist instantly, but the duration isn't displayed and there's no navigating. Forcing wave output means everything works as usual, but unfortunately the process repeats when you stop and restart playback or when you begin converting. Loading and converting movie length audio files becomes so slow and tedious you might as well be extracting thermfor loading into Audacity to compress and export as AAC etc, so after configuring the decoder initially, I didn't take long to switch to using the hack. Decoding that way is reserved for converting audio foobar2000 can't natively decode anyway (for my use), so the hack is fine. There's only so much coffee to be made while wave file creation is in progress.

[LoRd_MuldeR]

I don't see a distinction, as pretty much every encoder I've configured for a foobar2000 preset uses "ignore length" and accepts stdin with a fake wave file.

Surely, if both sides - the writing application and the reading application - agree on some non-standard format, then it's going to "work" - no matter how obscure their non-standard format is.

But things are going fail, as soon as the source actually is not in the non-standard format (but in the valid standard format), while the reading application still assumes the non-standard format. Similarly, things are going the fail as soon as the writing application produces the non-standard format, while the reading application expects the valid standard format (rather than the special non-standard format).

Thus, in addition to the prerequisite that both applications must support the same non-standard format, there is no way around some "out of band" signaling of when to actually use/assume the non-standard format.


stdin and "ignore length" work for every source and doesn't require advanced knowledge of the audio sample rate or channel count etc, assuming the encoder understand it.

It does not work "for every source". It works only if the writing application generates a "fake" WAV header followed by "raw" data; it works only if the reading application supports the "ignore length" hack; and it works only if the reading application is explicitly made aware (via CLI switch) that, this particular time, the "ignore length" hack is supposed to be enabled. The latter must not be done for normal WAV files, because it could easily produce garbage (see above).

So, after all, you don't gain much with "fake" WAV header, compared to just plain "raw" PCM. In both cases, an additional "out of band" information must be passed from writing application to reading application, i.e. outside of the stream (pipe).

If, on the other hand, you pass a proper "streamable" format over the pipe - for example, AU or FLAC format - then no such hackery and no "out of band" information whatsoever are needed:
DynamicAudioNormalizerCLI.exe -i "E:\Orange and Blue.wav" -t au -o - | ffmpeg.exe -i - final_output.wav



When you pipe to an encoder with DynamicAudioNormaliser, the temporary wave file count doubles, doesn't it? With the hack, the hard drive is free to write several lossy output files as their processed simultaneously, not having wave creation to worry about.

If you pipe from DynamicAudioNormaliser CLI front-end to an encoder, the content is passed in memory. No temporary file is created.

DynamicAudioNormaliser can output any format supported by libsndfile, either to a file or to a pipe. The latter, of course, requires a format works on non-seekable pipes. This includes AU and FLAC format, but not WAV/RIFF format.

...which is not my "idea" or "opinion", but simply how things have been implemented in libsndfile. And it's quite obvious (to any person familiar with WAV/RIFF format internals) why it has been implemented that way.

(Of course, using a temporary file always an option too)

[manolito]

Using a streamable format for piping like AU or Flac is nice, but the application you are piping to neds to understand this format. FFmpeg sure does, but Aften only supports WAV, and for many other encoders this is the same.

And somehow the -t switch is missing in my older (Non-SSE2) version of DynAudNorm... :devil::D

I made a couple more tests, and as a result I will stick to my current workflow. I used an MP3 with a cover picture in the meta data and tried to convert it to a WAV file which also included this cover. The older Audacity 2.03 would not do it, the very old WaveLab 3.0 would not do it, even the current LameXP created a WAV without the cover (in fact without any metadata even though the option to write metadata into the output was activated).

The only way to create such a WAV file with extra chunks after the WAV data was to use TagScanner and import the metadata from the MP3 into the WAV file. And true enough, piping this WAV file into Aften with the ignorelength parameter resulted in an AC3 with an artifact at the end.


I still think that I am not really affected by this. It only happens if the WAV file has these extra chunks after the WAV data, and this does not happen for me. Mostly these intermediate WAV files are created by Wavi, maybe by FFmpeg from an AVS script. No metadata here. The same is true when creating a WAV file or stream from SoX.

And in the very unlikely case that my source is such a WAV file, loading it into WaveLab and saving it again will get rid of these extra chunks. Or I can fire up TagScanner and remove those tags. I still believe that such metadata have no place in an intermediate file or stream. If I want them I will apply them to the final encoded file.


Cheers
manolito

[hello_hello]

It does not work "for every source". It works only if the writing application generates a "fake" WAV header followed by "raw" data; it works only if the reading application supports the "ignore length" hack; and it works only if the reading application is explicitly made aware (via CLI switch) that, this particular time, the "ignore length" hack is supposed to be enabled. The latter must not be done for normal WAV files, because it could easily produce garbage (see above).

As I'm generally using a GUI I guess I have a different perspective. Open a file with foobar2000 and convert it, it's decoded and piped using the hack. Unless an encoder won't accept stdin in which case you can tell foobar2000 to create a temporary input wave file instead.

Here's the list of encoders for which I'm currently using stdin:
Aften, fdkaac, ffdcaenc, ffmpeg, fhgaac, flac, lame, musepack, NeroAAC, ogg, opus, qaac, wavpack, WMAEncode

The list of encoders configured to use a temporary input file:
DynamicAudioNormalizer

So until recently the hack worked for every source because fooobar2000 decodes them all to a "common denominator" and sends it to the encoder. Including standard wave files. "ignore length" still required. The source itself is kind of irrelevant in that respect and I can't see how it'd be likely to work any other way with most GUIs. At least GUI's capable of playback too.

Foobar2000 has presets for many encoders. Select one and it gives you an appropriate GUI for basic settings, adjust them, switch to "custom" and the command line is filled in waiting for you to modify it. The encoder's version of "ignore length" will be included automatically along with stdin.
Creating multiple encoder presets to cover all the sample rate, bit depth and channel count contingencies defeats the purpose, and I'm not sure where that leaves channel mapping, so it's hack or temporary wave file.

No doubt there's times when the user would be setting up an encoder preset from scratch, if the GUI allows, in which case it's the user's responsibility to get the options right, and in relation to this discussion that's pretty much knowing when to use "ignore length" and when not to. Well I have to know now, whereas a few days ago....

If, on the other hand, you pass a proper "streamable" format over the pipe - for example, AU or FLAC format - then no such hackery and no "out of band" information whatsoever are needed:
DynamicAudioNormalizerCLI.exe -i "E:\Orange and Blue.wav" -t au -o - | ffmpeg.exe -i - final_output.wav

The last paragraph had so many flavours of emphasis, after tuning it into a quote I struggled to read it. :)

If an encoder accepts one of those formats as input I can't argue about that one.
(editing out some stuff after realising I misread the above command line)

I did create an encoder preset for DynamicAudioNormalizer to simply output a wave file itself, and in the end a two step process might be easier. Compress and convert a bunch of files to wave, then convert the wave files to the final format with the usual encoder. That's possibly how I would have done it anyway, because then I don't need one preset piping to encoder A, another piping to encoder A with different command line options, a preset piping to encoder B, another piping to.....
It's really just a few extra clicks to do it in two steps, but after creating the first encoder preset I thought I'd give piping a try mainly for the sake of it, and I've learnt a bit as a result.

PS. I noticed the only encoder preset for which I appear to have no "ignore length" equivalent in the command line is ffmpeg. I couldn't find much information on it, but the little I did find seems to indicate ffmpeg decides for itself when to ignore. Is that correct?

[hello_hello]

I used an MP3 with a cover picture in the meta data and tried to convert it to a WAV file which also included this cover. The older Audacity 2.03 would not do it, the very old WaveLab 3.0 would not do it, even the current LameXP created a WAV without the cover (in fact without any metadata even though the option to write metadata into the output was activated).

If you need to know for future reference, foobar2000 will do it.
I don't put cover art in files normally, but I was curious. The option to transfer attached pictures worked when converting an MP3 to wave, as did attaching a pic to a wave file loaded into a playlist (right click Tagging/Attach Pictures menu).
I've been using foobar2000 for so long I've no idea which features are native any more. The picture attaching options probably are, but if not I have the Quicktagger and TagBox DSPs/plugins installed, one of which would be responsible for them.

[manolito]

PS. I noticed the only encoder preset for which I appear to have no "ignore length" equivalent in the command line is ffmpeg. I couldn't find much information on it, but the little I did find seems to indicate ffmpeg decides for itself when to ignore. Is that correct?

No, FFmpeg does have a (poorly documented) "ignore length" parameter. See this post:
https://forum.doom9.org/showthread.php?p=1795583#post1795583


Cheers
manolito

[hello_hello]

Thanks. Should I find myself confused as to why I wasn't having problem without it, or should I just add it and pretend it was always there?
It might explain a couple of log file warnings I didn't understand. I probably should test that later. I might learn a little more.
Cheers.

[hello_hello]

Nothing's ever easy or I missed the obvious, but ffmpeg possibly has the only ignore length command line option requiring an argument, so the full story turned out to be:

-ignore_length true

Testing so far hasn't revealed a difference. The log files are the same aside from acknowledging the extra command line option. With and without -ignore_length, ffmpeg's output flac files have been identical. Same number of samples, same bitrate, same audio md5. I supposed it can't hurt to leave it in the command line though.

[manolito]

The only time when this option can possibly make a difference is when the input is a WAV file which has extra data chunks (like a cover artwork) AFTER the main WAV data. WAV files which have been created as intermediate temporary WAV files (or WAV streams when using a pipe) will never have these extra chunks after the WAV data, so the "ignorelength" parameter will not make any difference in this case.

Cheers
manolito

[hello_hello]

Maybe a mod can split this off into a new thread if it's in the way here, but because I didn't know the hack was a hack until very recently, I played around a little more.

From a GUI perspective again, where the audio is always decoded by the GUI, a couple of encoders seem to assume "ignore length" for stdin and don't have an "ignore length" option (that I could find). They were LAME and MusePack. I'm not sure about the latter but for LAME stdin seems to assume the hack unless you specifically tell it the input is raw.
FFmpeg's "ignore length" probably fits into the above category too, simply because it's not well documented.

For the encoders with an "ignore length" option, many still seem to apply the same rule for stdin. The following encoders produced output files with the correct number of samples either way. FDKAAC, FFmpeg, FhGAC, Flac, Ogg, Opus, QAAC, WMAEncode.
The only anomaly I found there was the Flac file size was slightly larger without it's version of "ignore length" in the command line, yet according to foobar2000 the bitrate, sample count, and audio md5 were the same.

The encoders that failed without it were Aften, ffdcaenc, NeroAAC, and WavPack.
Fortunately though, encoders either exited with an error or they encoded normally, there seemed to be no middle ground, so from that perspective it's a good thing.

Not being able to encode with Nero if I used stdin without "ignore length" (a temporary input wave file worked fine) I tried with Audacity to see what would happen, but I must have been using the wrong incantation, because the progress meter would get to the halfway point each time and Audacity would stop responding, but I'm quite interested to learn why NeroAAC just exits with an error for foobar2000 if I use stdin without "ignore length", if it'll encode for Audacity, however I need to find out why it's not working at all first (I had an older Audacity installed so I upgraded to the latest but the result was the same).

The command line for Audacity I'm using:
"C:\Program Files\foobar2000\encoders\neroAacEnc.exe" -q 0.40 -ignorelength -if - -of %f

[LoRd_MuldeR]

FWIW, here is a new Test version with Python bindings included:
https://sourceforge.net/projects/muldersoft/files/Dynamic%20Audio%20Normalizer/Testing/DynamicAudioNormalizer.2017-02-22.Windows-DLL.zip/download

[LoRd_MuldeR]

Dynamic Audio Normalizer v2.10
• https://github.com/lordmulder/DynamicAudioNormalizer/releases/tag/2.10
• https://www.mediafire.com/folder/flrb14nitnh8i/Dynamic_Audio_Normalizer
• https://bitbucket.org/muldersoft/dynamic-audio-normalizer/downloads

Changelog:
• Core library: Added process() function, i.e. an "out-of-place" version of processInplace()
• Implemented Python API → Dynamic Audio Normalizer can be used in, e .g., Python-based applications
• CLI front-end: Added new CLI option -t to explicitly specify the desired output format
• CLI front-end: Added new CLI option -d to explicitly specify the desired input library
• CLI front-end: Added support for decoding input files via libmpg123 library
• CLI front-end: Implemented automatic/heuristic selection of the suitable input library
• CLI front-end: Properly handle input files that provide more (or less) samples than what was projected
• Windows binaries: Updated the included libsndfile version to 1.0.27 (2016-06-19)
• Windows binaries: Updated build environment to Visual Studio 2015 (MSVC 14.0)

[kolamorx]

Hello.
I've been experimenting with FFmpeg's dynaudnorm filter recently (on movies), and the results are good.
However, the loud parts (music, effects) are sometimes slightly louder than the quiet parts (dialogue).
My goal is to get a completely constant volume, with no need to raise and lower the volume at all.
What options and settings should I use in order to achieve this goal?

[LoRd_MuldeR]

Hello.
I've been experimenting with FFmpeg's dynaudnorm filter recently (on movies), and the results are good.
However, the loud parts (music, effects) are sometimes slightly louder than the quiet parts (dialogue).
My goal is to get a completely constant volume, with no need to raise and lower the volume at all.
What options and settings should I use in order to achieve this goal?

First of all, it is possible that the "quiet" part is so quiet, compared to the "loud" parts, that the maximum gain factor (default: 10×) doesn't suffice to bring it up to the same volume. So you can try increasing the maximum gain factor.

Secondly, it is possible that the "quiet" part is shortly after a "loud" part and the filter takes a moment to adapt. Keep in mind that the filter adapts slowly and smoothly, in order to avoid nasty "volume pumping" effect. Anyway, you can try using a smaller Gaussian filter window size (default: 31) to make the filter adapter faster.

Finally, it is possible that the "quiet" part simply has more dynamics compared to the "loud" part. What it means is that the "quiet" part contains a few signal peaks that prevent it from being amplified even further (without clipping), even though the average amplitude is relatively low. In this case you can give RMS mode a try. Or you can try adding some input compression. Or both ;)

See also:
http://muldersoft.com/docs/dyauno_readme.html#configuration

[kolamorx]

Since I'm not going to check this options for every movie, what I'm looking for is the best general settings to cover these options.

Secondly, it is possible that the "quiet" part is shortly after a "loud" part and the filter takes a moment to adapt. Keep in mind that the filter adapts slowly and smoothly, in order to avoid nasty "volume pumping" effect. Anyway, you can try using a smaller Gaussian filter window size (default: 31) to make the filter adapter faster.
I changed it already, (g=11) and it improved the results.
Finally, it is possible that the "quiet" part simply has more dynamics compared to the "loud" part. What it means is that the "quiet" part contains a few signal peaks that prevent it from being amplified even further (without clipping), even though the average amplitude is relatively low. In this case you can give RMS mode a try. Or you can try adding some input compression. Or both
RMS - I haven't tried it much.
input compression - tried it already, and it causes FFmpeg to freeze right at the beginning. (I use rogerdpack version for XP.)

[hello_hello]

kolamorx,
Try just reducing the frame size. For ffmpeg:

-af dynaudnorm=f=150

or even f=75 and g=11 if it doesn't result in too much "pumping".

Edit see post #133 regarding the compression option and a newer version of ffmpeg (XP friendly)

[LoRd_MuldeR]

input compression - tried it already, and it causes FFmpeg to freeze right at the beginning. (I use rogerdpack version for XP.)

FFmpeg actually doesn't use my "original" code/library, but they re-implemented everything.

So, if you get "freeze" with FFmpeg and DynAudNorm when input compression is enabled, I suggest you try it with the "standalone" version of DynAudNorm:
https://github.com/lordmulder/DynamicAudioNormalizer/releases/latest

If you can re-produce the "freeze" with that, please send my a sample file and your exact options. Otherwise, please send a bug-report to the FFmpeg developers.

[hydra3333]

FFmpeg actually doesn't use my "original" code/library, but they re-implemented everything.
<snip>
... Otherwise, please send a bug-report to the FFmpeg developers.
Goodness me.
How to they deal with updates you may produce from time to time ?

[stax76]

kolamorx,
Try just reducing the frame size. For ffmpeg:

-af dynaudnorm=f=150

or even f=75 and g=11 if it doesn't result in too much "pumping".

Does anybody know where I can find the documentation for this?

edit:

To answer my own question after going through the thread: https://ffmpeg.org/ffmpeg-filters.html#dynaudnorm

[richardpl]

Goodness me.
How to they deal with updates you may produce from time to time ?

There had be no updates to core for very long time. And relevant changes are committed when they appear.

[hello_hello]

RMS - I haven't tried it much.
input compression - tried it already, and it causes FFmpeg to freeze right at the beginning. (I use rogerdpack version for XP.)

As I don't normally use the compression option I didn't remember straight away, but I now recall someone at VideoHelp mentioning the same problem.
I tested with the following ffmpeg command line myself and it works fine using this version of ffmpeg (https://github.com/rdp/ffmpeg-windows-build-helpers/issues/219#issuecomment-306066471), which is probably more recent than the version you're using and it runs on XP.

-af dynaudnorm=f=75:g=11:s=12

[manolito]

If you have trouble getting the desired results with DynAudNorm you can try the "Loudnorm" filter which comes with current versions of FFmpeg, Muxson wrote a nicle little GUI for it:
http://www.muxson.com/winloud


Cheers
manolito

[kolamorx]

I tested with the following ffmpeg command line myself and it works fine using this version of ffmpeg (https://github.com/rdp/ffmpeg-windows-build-helpers/issues/219#issuecomment-306066471), which is probably more recent than the version you're using and it runs on XP.

-af dynaudnorm=f=75:g=11:s=12

Thanks.
Any other suggestions for general settings?

[v0lt]

@LoRd_MuldeR
I have a sound in float format, in which the range is higher than [-1.0, +1.0]. DynamicAudioNormalizer (FFmpeg implementation) can reduce the sound level to [-1.0, +1.0]?

[richardpl]

@LoRd_MuldeR
I have a sound in float format, in which the range is higher than [-1.0, +1.0]. DynamicAudioNormalizer (FFmpeg implementation) can reduce the sound level to [-1.0, +1.0]?

See alimiter filter documentation.

[raffriff42]

From the ffmpeg source, it looks like all audio filtering is done at double-precision floating point, with nominal +1/-1 range - but no clipping outside that range. FFmpeg dynaudnorm default peak volume is 0.95 (-0.4 dB on a +1/-1 scale). It looks to me like it'll normalize any input range without problems; you'll have to try it for yourself though.

[kolamorx]

First of all, it is possible that the "quiet" part is so quiet, compared to the "loud" parts, that the maximum gain factor (default: 10×) doesn't suffice to bring it up to the same volume. So you can try increasing the maximum gain factor.
In some movies, the dialogue is so quiet that even a maximum gain factor of 100× (20 dB) doesn't suffice.
What should I do in this case?

[LoRd_MuldeR]

In some movies, the dialogue is so quiet that even a maximum gain factor of 100× (20 dB) doesn't suffice.
What should I do in this case?

If the original source was encoded in AC3/EAC3 (Dolby Digital) from, then you may try enabling the DRC (Dynamic Range Compression) feature of the AC3/EAC3 decoder already.

Apart from that, you can try the "--compress" option of the DynamicAudioNormalizer. See the manual for details...

[kolamorx]

If the original source was encoded in AC3/EAC3 (Dolby Digital) from, then you may try enabling the DRC (Dynamic Range Compression) feature of the AC3/EAC3 decoder already.

Apart from that, you can try the "--compress" option of the DynamicAudioNormalizer. See the manual for details...
Actually, my idea was to increase the volume before using dynaudnorm. (-af "volume=15dB,dynaudnorm=m=100...")
Is there a limit for the loss factor as there is for the gain factor?

[LoRd_MuldeR]

Actually, my idea was to increase the volume before using dynaudnorm. (-af "volume=15dB,dynaudnorm=m=100...")
Is there a limit for the loss factor as there is for the gain factor?

If the volume can be increased any further using a fixed gain factor without causing clipping in the "loud" sections (i.e. there is some unused "headroom" in the file), then you should be doing this, of course.

However, if you use a tool like DynamicAudioNormalizer, it usually means that "traditional" normalization doesn't work on your file, because the "loud" sections are at maximum signal level already and yet the "quiet" sections are still too silent.

After all, the DynamicAudioNormalizer is not about "increasing" the volume (you could achieve that by simply turning up your speakers). It is all about "equalizing" the volume of "quiet" and "loud" sections ;)

[kolamorx]

So, I guess I don't really understand how this tool works.
I tought that if, for example, the loud parts are 90dB, the quiet parts are 65dB, and you set the target value to 80dB- it decreases the loud parts by 10dB and increases the quiet parts by 15dB.

[LoRd_MuldeR]

So, I guess I don't really understand how this tool works.
I tought that if, for example, the loud parts are 90dB, the quiet parts are 65dB, and you set the target value to 80dB- it decreases the loud parts by 10dB and increases the quiet parts by 15dB.

Yes, when using the DynamicAudioNormalizer, any "frame" whose level is below the target level will be amplified to the target level, and any "frame" whose level is above the target level will be attenuated to the target level – with some trickery (Gaussian filter) to avoid fast fluctuation of the amplification/attenuation factors. In other words, the DynamicAudioNormalizer will "equalize" the volume of "quiet" and "loud" sections.

The more simple "volume" filter will just apply the same fixed amplification factor to all samples. So, if the "loud" sections of your file already are at maximum signal level, then the "volume" filter can not further increase the volume of your file, as this would unavoidably result in clipping (in the "loud" sections). In other words, with the "volume" filter, the "loud" sections may prevent further amplification of the "quiet" sections. The DynamicAudioNormalizer can help in this situation.

If, on the other hand, the "loud" sections of your file are not at maximum signal level yet, i.e. there is some "headroom" in your file, then the "volume" filter can increase the volume of your file – to the extent that brings the "loud" sections to the maximum signal level. But even in that case, the "volume" filter does not "equalize" the volume of "quiet" and "loud" sections! It's just like turning up the volume of your speakers a little more. There is no "dynamic" adjustment.

Still, if you actually do have some "headroom" in your file, I would suggest to apply the "volume" filter first, to get "traditional" normalization. And then, if still required, use DynamicAudioNormalizer to bring up the "quiet" section some more...

[kolamorx]

How do I check how much headroom I have in my file?

[LoRd_MuldeR]

How do I check how much headroom I have in my file?

For example, with the "volumedetect" filter:
https://ffmpeg.org/ffmpeg-filters.html#volumedetect

(it is max_volume that you are interested in)

[kolamorx]

Thanks.
Any chance you will increase the maximum possible value of the maximum gain factor in the future?

[LoRd_MuldeR]

Thanks.
Any chance you will increase the maximum possible value of the maximum gain factor in the future?

I don't think so. An amplification factor of 100× already is equivalent to +40 dB.

If the "quiet" sections really are that much quieter than the "loud" sections that even an amplification by 100× (+40 dB) doesn't suffice, it probably means that the "quiet" sections are already lost in the noise.

Again: Don't use DynAudNorm to eliminate the "headroom" in your file. Use a "traditional" normalizer for that purpose. Use the DynAudNorm to "equalize" the volume of "quiet" and "loud" sections...

[manolito]

Still, if you actually do have some "headroom" in your file, I would suggest to apply the "volume" filter first, to get "traditional" normalization. And then, if still required, use DynamicAudioNormalizer to bring up the "quiet" section some more...

Not so sure about this recommendation...

I downloaded a short random clip with "sane" audio levels and did a few tests:

The source had these levels:
Integrated Loudness: -19.83 LUFS
Sample Peak: -6.07 SPFS

After conversion using DynAudNorm with default settings:
Integrated Loudness: -12.72 LUFS
Sample Peak: -0.45 SPFS

This time using peak normalizing first followed by DynAudNorm with default settings:
Integrated Loudness: -12.64 LUFS
Sample Peak: -0.36 SPFS


The loudness values are almost identical for both conversions. At least for "sane" source audio levels it is not necessary to apply regular peak normalization before DynAudNorm.


Just for fun I converted the same clip using the LoudNorm filter which is part of current FFmpeg versions. Since I do not like the defaults all that much I used the following parameters:
SET LUFS=-18.00
REM Target Loudness in LUFS
SET TruePeak=-1.5
REM Target True Peak value
SET L_Range=14
REM Target Loudness Range

I used 1-pass mode which of course has problems reaching the desired loudness values, but is much faster. The result came out like this:
Integrated Loudness: -16.27 LUFS
Sample Peak: -0.96 SPFS


To my ears this conversion did sound a little better than DynAudNorm, and it has the advantage that a target LUFS level can be specified.


Cheers
manolito

[kolamorx]

I don't think so. An amplification factor of 100× already is equivalent to +40 dB.
I mistakenly thought it's equivalent to +20 dB...
To my ears this conversion did sound a little better than DynAudNorm, and it has the advantage that a target LUFS level can be specified.
I prefer dynaudnorm since it evens out the volume better.