MKVToolNix v24.0.0 released

[mkver]

Ok, so let me rephrase: If I use mkvmerge to mux the webm file, the srt subs and the opus file that you uploaded I am supposed to get a file that has a defect audio track? Because I just did exactly that and the file I got was not defective.

[fadedfedor]

Originally Posted by mkver

Ok, so let me rephrase: If I use mkvmerge to mux the webm file, the srt subs and the opus file that you uploaded I am supposed to get a file that has a defect audio track? Because I just did exactly that and the file I got was not defective.

You're right, I just did the same thing and the audio sounds fine. I noticed when I load the short clips there are two tags in the webm with the VP9 video. When I load the full VP9 and Opus tracks that those clips were cut from, there are no tags included. Could that have something to do with it?

[mkver]

Can you upload the full opus file? Or at the least the first few megabyte of it (it's important that you don't remux the file, but simply extract the first few MB (with a hex editor, not with a tool like ffmpeg); yes, you will probably have a defect packet at the end, but if there is enough material before it one can nevertheless use it to analyze it).
I'd be very surprised if the tags had anything to do with it. And the approval of attachments is very slow on this forum. If you ever want to upload something, then don't do it as an attachment. Not that I need a screenshot to show me that there are tags. I know it, because ffmpeg adds some tags by default.

[fadedfedor]

You meant the file before it's muxed into mkv that had the added clicking noises, right? Stupid question probably.
I split the file with a hex editor, is the first 3MB split okay? I don't mind uploading the full Opus file either, it's only 33MB.

https://a.safe.moe/0iTR2WZ.opus

[sneaker_ger]

I can hear the popping with the mkv remuxed from that sample.

[mkver]

If you have no problem uploading the whole sample, then please do so. Thanks.
For ease of reading: A = the opus file you just uploaded, B = file A remuxed with mkvmerge, C = file A remuxed with ffmpeg to opus
1. I can confirm that in B the packets at 980ms, 1980ms etc. have a DiscardPadding element. A uses page durations of 1s and the DiscardPadding elements in B are attached to the last frame out of A's pages (if I am not mistaken).
2. opusinfo has two complaints about B: WARNING: Samples with negative granpos in stream 1
WARNING: EOS not set on stream 1 (normal for live streams)
The second warning is an obvious result of the fact that the file is truncated; I don't think the first warning is, but to be sure I'd like to check the whole 33MB file.
What is remarkable is that there are no "WARNING: Sample count ahead of granule (49920>49608) in stream 1" and no "ERROR: stream 1 has interior holes or more than one page of end trimming" messages. (These are the errors that opusinfo outputs if one extracts the track from B to ogg/opus and analyzes the output with opusinfo.)
3. When A is decoded with the reference decoder, it also has this popping in it, but only once at about 970-980ms; B has it more often, because it has DiscardPadding elements every second. Consequently the decoded output of A and B diverge gradually; the difference in length is 138216 = 443*312 samples. 443s = 7m23s, the length of the file is about 7m24s.
If A is decoded with ffmpeg's native decoder then it is quite ok (the native decoder doesn't strip the CodecDelay away, so that the output file is actually 624 samples longer than the output of the reference decoder).
The differing behaviour of the reference decoder and mkvmerge might be due to a bug in the latter.
4. Unfortunately I don't know an analyzer that does for ogg/opus what mkvinfo does for Matroska.
5. Given that A is likely buggy I'd like to know exactly how you created it. In particular I'd like to know if you can reproducibily produce files like A.

[fadedfedor]

33.9MB Opus



I used these settings to create it with XMedia Recode 3.4.3.0. I deleted the source I used shortly after encoding because I listened to some of it and was happy with the result until I muxed it. I just checked and the source is no longer available online either. Almost forgot to mention that the source was 5.1 and I didn't use a downmix matrix.

[mkver]

If your defective opus file is all you've got, then the best thing to do would be to remux said file with ffmpeg: ffmpeg -i RObV6XA0h1k.opus -c copy output.opus. As has been said, this seems to repair it.
If the video is no longer available and you still want to help, you can try a few similar videos from the same website. They presumably use one set of tools for all their videos so that this behaviour may also be triggered by other videos.

[fadedfedor]

Thanks once again. That did seem to repair it and it sounds fine when muxed into an mkv now. That was my first time using ffmpeg. I'm sure I've used a couple programs that utilize it. I really appreciate your detailed analysis. I'll definitely try more videos from that same site.

[mkver]

You have definitely already used ffmpeg: XMedia Recode uses it; in fact, RObV6XA0h1k.opus was created by ffmpeg: Lavc58.17.100 libopus encoded RObV6XA0h1k.opus. Lavc stands for libavcodec, the library ffmpeg provides for de- and encoding. And the first opus file you uploaded (wNs6w3z.opus) was created using libavformat, ffmpeg's muxer and demuxer library.
But anyway, I can now produce such files myself: As has already been mentioned, ffmpeg's native opus decoder doesn't strip the Preskip/CodecDelay away; instead it adapts the timestamps: The first pcm sample has a timestamp of -312 (if the preskip was 312 samples, the default for libopus) on an 48 kHz timescale (or -6.5ms if you prefer) so that the actually valid samples start at zero. Unfortunately the invalid samples with negative timestamps aren't stripped away later either: If you output it to wave, the wave muxer includes the invalid samples in the output file; and if you reencode an opus file again (using ffmpeg's native decoder and the libopus encoder) the output file causes opusinfo to raise a "Samples with negative granpos in stream 1" warning and mkvmerge includes these DiscardPadding elements in the whole file.
So you don't need to try other files from the site again. I'm currently reading the opus in ogg specs to find out which tool is buggy (and whether files like RObV6XA0h1k.opus are actually against the spec (there might be a reason that opusinfo only shows a warning for such files)) and inform the relevant people. I'm already pretty sure that mkvmerge has a bug: If I don't use an opus input, but offset the timestamps manually with -itsoffset -0.05, I get a file where the DiscardPadding is more than the duration of the block without DiscardPadding:

| + Block group
|  + Block: track number 1, 1 frame(s), timestamp 00:00:00.940002336
|   + Frame with size 3
|  + Discard padding: 10000000
|  + Block duration: 00:00:00.009999360
| + Block group
|  + Block: track number 1, 1 frame(s), timestamp 00:00:00.960001056
|   + Frame with size 3
|  + Discard padding: 30000000
|  + Block duration: 00:00:00.000000000
| + Block group
|  + Block: track number 1, 1 frame(s), timestamp 00:00:00.979999776
|   + Frame with size 3
|  + Discard padding: 50000000
|  + Block duration: 00:00:00.000000000
...
| + Block group
|  + Block: track number 1, 1 frame(s), timestamp 00:00:01.940000832
|   + Frame with size 163
|  + Discard padding: 10000000
|  + Block duration: 00:00:00.009999360
| + Block group
|  + Block: track number 1, 1 frame(s), timestamp 00:00:01.959999552
|   + Frame with size 173
|  + Discard padding: 30000000
|  + Block duration: 00:00:00.000000000
| + Block group
|  + Block: track number 1, 1 frame(s), timestamp 00:00:01.979998272
|   + Frame with size 166
|  + Discard padding: 50000000
|  + Block duration: 00:00:00.000000000
...

One more thing: The offset in RObV6XA0h1k.opus (equal to 312 samples, the Preskip of the reference opus encoder) lets me believe that this file has been reencoded from an source that was already opus. Is that so?

[fadedfedor]

Originally Posted by mkver

One more thing: The offset in RObV6XA0h1k.opus (equal to 312 samples, the Preskip of the reference opus encoder) lets me believe that this file has been reencoded from an source that was already opus. Is that so?

Yes, the source was Opus. It was an odd 5.1 channel file that I've never seen on that site before.

[kuchikirukia]

I've uploaded a transport stream to your ftp that won't mux. The uncut one is 14GB and when mkvtoolnix tries to mux it it will sit there and read it, slowly filling 14GB of RAM, and then output a 120MB mkv that doesn't work.

If you split it with tsmuxer every piece is broken.
mkvtoolnix doesn't report any errors.

[mkver]

This sounds like a file that doesn't contain a single frame that mkvmerge detects as keyframe. Try the "--engage all_i_slices_are_key_frames" option. If this doesn't solve your problem: Can the transport stream actually be played? And does it contain other tracks than the video stream (can the 120 MB come from an audio stream?)?

[Mosu]

That sample is strange. I've only looked into it for a minute, but ffmpeg isn't happy with it either. Running "ffmpeg -i kuchikirukia\ -\ won\'t\ mux.ts -an -c:v copy ffmpeg.h264" produces an empty file; the similar "ffmpeg -i kuchikirukia\ -\ won\'t\ mux.ts -an -c:v copy ffmpeg.mkv" produces a Matroska file with track headers & tags but without any frame.

As to mkvmerge: the option is called ""--engage all_i_slices_are_key_frames" (additional "_" after "key"), BTW. You can mux the file that way. There are I slices every 500ms which will all be marked as key frames.

If I use mkvmerge with "all_i_slices…", extract the h.264 from that and import into an MP4 with MP4Box, then none of the frames are marked as key frames in the MP4 file (the "stss" atom contains an empty table).

If anyone else wants to take a look: I'll leave the file up here for a while.

[mkver]

The file has 27 I frames, 27 SPS/PPS combinations (in the same access units as the keyframes), but no IDR frame and no recovery_point SEI messages. In other words: This is exactly the type of file for which said engage option has been introduced.
ffmpeg by default strips everything in front of the very first keyframe away. This behaviour can be overriden with -copyinkf[:stream_specifier]. Btw: It seems that ffmpeg ignores the Matroska keyframe flags: If I copy the file muxed with mkvmerge with the engage option, then the output file doesn't contain any frames flagged as keyframes (and no cues either, of course).

[manolito]

Also played a little bit with this sample, it is definitely quite broken IMO...

TSDoctor has no problems with it, no errors and no warnings. Demuxing or Remuxing always ends up without any video, no matter if you use MKVMerge or FFmpeg.

//EDIT//
Sorry I have to correct myself...
I just found out that Avidemux can remux this clip to MKV just fine (without any reencoding). I was using the ancient version 2.6.8 which still works under Win XP. As far as I can tell the resulting MKV seems to be in sync. Not too bad...
//End Edit//

But...
This clip can very well get transcoded, either to the same format or to a different format. FFmpeg does it, and it is also possible using AviSynth. As the AVS source filter only DSS2Mod worked for me, the three different versions of ffms2 I tried all failed.

Too bad that this clip really makes it impossible to judge audio sync, but I guess this could be repaired one way or another.


Cheers
manolito

[mkver]

Both mkvmerge (with the engage option already mentioned) and ffmpeg (with the copyinkf option; although you won't get an index in this case and the I frames won't have the keyframe flag set) can mux this file with video.
Do you have any reasons to believe there could be audio sync problems? (The sound doesn't consist in dialogue or noises that are caused by things we see onscreen, so we can't detect audio sync with it.)
If you want to repair this clip, you should add recovery_point SEI messages to the I frames: 0606018480 in hex without the start code (00000005 in Matroska, 000001 in an elementary stream/transport stream; but keep in mind that you can't simply add something in Matroska or a transport stream without breaking the container). Maybe mkvmerge could offer an option to include said recovery_points so that one can reprocess further with other tools who want to derive the keyframe condition on their own and don't use the Matroska flag?

[kuchikirukia]

I'm using a version of FFMS2 I pulled from somewhere (i needed the 10 bit hack) and it works to encode it from the start, it just can't seek. Trying to open it with MeGUI's preview just makes it churn endlessly since it tries to open from the middle.
MPC-HC plays it just fine. Audio's in sync and it has no problems seeking anywhere.

[Mosu]

MKVToolNix v23.0.0 is out containing a moderate number of enhancements and bug fixes.

About AV1 support: even though the previous release was the first to support AV1, keep in mind that neither the bitstream format nor the method of storage in MP4, WebM and Matroska has been finalized yet. I've therefore decided to disable AV1 support by default. You have to enable it manually by passing `--engage enable_av1` to mkvmerge. mkvmerge can still identify AV1 without that option, but it'll refuse to multiplex it.

Note further that only supported bitstream format is the one that was active on 2018-05-02.

To my users on Debian and Ubuntu: the layout of my APT repositories was changed in April. You'll have to update your `sources.list` entry accordingly. Read more about that in this blog post.

Here are the usual links: the MKVToolNix home page, the Windows installer/portable version & macOS DMG and the source code.

The Windows and macOS binaries are available already. The Linux binaries are still being built and will be available of the course of the next couple of hours.

Here are the NEWS since the previous release:

# Version 23.0.0 "The Bride Said No" 2018-05-02

## New features and enhancements

* mkvmerge: input: format detection uses file-extension to improve performance and to give preference when several formats match.
* mkvmerge: AV1: added support for reading AV1 video from Open Bitstream Unit files.
* mkvmerge: AV1: adjusted the code for the AV1 bitstream format changes made up to 2018-05-02 (git revision d14e878).
* mkvmerge: MP4 reader: if a track has an edit list with two identical entries, each spanning the file's duration as given in the movie header atom, then the second entry will now be ignored. Improves the handling of files with bogus data; see #2196 and #2270.
* MKVToolNix GUI: multiplexer: added options to only enable tracks of certain types by default. Implements #2271.
* MKVToolNix GUI: multiplexer: added an option to enable dialog normalization gain removal by default for all audio tracks for which the operation is supported. Implements #2272.
* MKVToolNix GUI: multiplexer: when deriving track languages from the file names is active and the file name contains the usual season/episode pattern (e.g. "S02E14"), then only the part after the season/episode pattern will be used for detecting the language. Part of the improvements for #2267.
* MKVToolNix GUI: multiplexer: the regular expression used for deriving track languages from the file names can now be customized in the preferences. Part of the improvements for #2267.
* MKVToolNix GUI: multiplexer: the user can now customize the list of track languages the GUI recognizes in file names. This list defaults to a handful of common languages instead of the full list of supported languages. Part of the improvements for #2267.

## Bug fixes

* mkvmerge: MP3 packetizer: removed a memory leak growing linearly with the track's size.
* mkvmerge: VobSub packetizer: whenever a VobSub packet doesn't contain a duration on the container level, mkvmerge will now set it from the duration in the SPU packets. Before it was accidentally setting the SPU-level duration to 0 instead. Fixes #2260.
* mkvmerge: track statistics tags: if writing the `Date` element is deactivated via `--no-date`, the `_STATISTICS_WRITING_DATE_UTC` isn't written either anymore. Fixes #2286.
* mkvmerge, mkvextract, mkvpropedit: removed several small, constant-size memory leaks.
* mkvextract: fixed a crash when mkvextract with a non-Matroska file as the source file. Fixes #2281.
* MKVToolNix GUI: the central area is now scrollable, allowing the GUI to be resized to almost arbitrary sizes. Fixes #2265.
* MKVToolNix GUI: multiplexer: the "copy file title to destination file name" functionality will now replace everything in the destination file name up to the last period instead of only up to the first period. Fixes #2276.

## Build system changes

* build system: MKVToolNix now requires a compiler that supports the following features of the C++14 standard: "user-defined literals for `std::string`". For the GNU Compiler Collection (gcc) this means v5.x or newer; for clang it means v3.4 or newer.
* Windows: linking against and installing shared version of the libraries with MXE is now supported by setting `configure`'s `host` triplet accordingly, e.g. `--host=x86_64-w64-mingw32.shared`.

## Other changes

* mkvmerge: AV1: support for AV1 must be activated manually by adding `--engage enable_av1` as the AV1 bitstream specification hasn't been finalized yet.

Have fun

[LeMoi]

Thanks, many new features are quite useful and were missing