MKVToolNix v99.0 released

[Mosu]

No. Whenever I have free time and feel motivated to tackle it. Thing is, it isn't a trivial change.

[wonkey_monkey]

A warning, as people still seem to be posting directly to threads and may not be aware:

It appears that the forum may have been hacked. There is a suspicious "test" announcement, apparently from tebasuna51 (but probably not), parts of the forum are not working, and there appear to be some malicious javascript files.

I'm not speaking in any official capacity here, but I would recommend, at the very least, NOT entering your password anywhere on Doom9 for the time being.

Please refer to this post: https://forum.doom9.org/showthread.php?goto=newpost&t=176128

[LeMoi]

Sometimes I make a mistake when I encode a video and I forget to crop, I just realize after I mux.
So I do a reencode of the video with the proper crop settings, I remove the muxing job from the queue and re-edit it in MKVToolNix GUI, but it doesn't reload the properties of the files, and if I just remux the files, the ratio of the muxed file is wrong... (ie old 16/9 instead of new 2.35). I have to remove the video track from the list and re-add it to get the new properties.
Is it possible that, when I edit a job and remove it from the queue, MKVToolNix GUI re-checks the properties of the files? while keeping the settings like delay, track name, etc.?

[Mosu]

Is it possible that, when I edit a job and remove it from the queue, MKVToolNix GUI re-checks the properties of the files? while keeping the settings like delay, track name, etc.?

No, that's not possible at the moment, and it would require quite a lot of work to implement. Not something I'm willing to spend time on.

[LeMoi]

OK, no problem, you're the programer, I thought it would be easier to reload the files, I'll wait until you change your mind :)

[Wakaku]

Thank you for implementing "emoji" support. No more "black diamonds with white question marks". All areas seems covered (eg. input of/output to emoji filenames, Header editor tags, retaining XML tags, covers). Thus such metadata can now be preserved. Output MKV is playable without problems in MPC-HC, MPC-BE, PotPlayer, KMPlayer, VLC. :)

[Mosu]

Hey,

Here's MKVToolNix v32.0.0, a really small bug fix release, the most important probably being the handling of Unicode code points > U+ffff (e.g. Emojis). For that to work a bug-fixed libEBML is also needed, which is why libEBML v1.3.7 is now required. It was released earlier today.

Other than that nothing has changed for package managers since v31.0.0.

Here are the usual links: the MKVToolNix home page (https://mkvtoolnix.download/), the Windows installer/portable version & macOS DMG & Linux AppImage (https://www.fosshub.com/MKVToolNix.html) and the source code (https://mkvtoolnix.download/source.html).

The Windows and macOS binaries as well as the Linux AppImage are available already. The other Linux binaries are still being built and will be available over the course of the next couple of hours.

Here are the NEWS (https://mkvtoolnix.download/doc/NEWS.md) since the previous release:

# Version 32.0.0 "Astral Progressions" 2019-03-12

## New features and enhancements

* mkvinfo: when sizes are output the size of the element's data portion is output in addition to the element's total size.
* MKVToolNix GUI: info tool: the element's data portion is shown as an extra column.
* MKVToolNix GUI: multiplexer: added column "Delay" to the track list containing the additional delay to apply during multiplexing. Implements #2506 (https://gitlab.com/mbunkus/mkvtoolnix/issues/2506).

## Bug fixes

* all: fixed handling of Unicode code points > U+FFFF. Fixes #2516 (https://gitlab.com/mbunkus/mkvtoolnix/issues/2516).
* mkvmerge: Windows: mkvmerge was crashing with an exception when trying to identify certain files that can be used on Blu-rays (such as MPEG transport streams of MPLS play list files) and when the file name was given as a UNC path (e.g. `\\servername\sharename\path\to\file.m2ts`). The GUI emitted errors such as "the JSON output could not be parsed" in that case. Fixes #2507 (https://gitlab.com/mbunkus/mkvtoolnix/issues/2507).
* MKVToolNix GUI: the portable mode wasn't detected correctly when the current working directory the GUI was started from wasn't the directory the GUI's executable file was located it. Examples for when this is the case are Windows' "send to" or "open with" functions. Fixes #2501 (https://gitlab.com/mbunkus/mkvtoolnix/issues/2501).
* MKVToolNix GUI: multiplexer: using button to change the current destination directory to one of the recently used ones did not update the file name according to the "make file name unique" setting. Part of the fix of #2519 (https://gitlab.com/mbunkus/mkvtoolnix/issues/2519).
* MKVToolNix GUI: multiplexer: the function "set destination file name from selected file's name" will now only change the destination file's name but not its path. Part of the fix of #2519 (https://gitlab.com/mbunkus/mkvtoolnix/issues/2519).

## Build system changes

* libEBML v1.3.7 and libMatroska 1.5.0 are now required as they fix their handling of Unicode code points > U+FFFF (see #2516 (https://gitlab.com/mbunkus/mkvtoolnix/issues/2516)).

Have fun :)

[hubblec4]

Thanks a lot.

[robena]

Hi,

I'm tying to use mkvmerge in command line mode, inside some scripts I wrote to automatize remuxing MKV files originating from Blu-rays.

I'm working with an overclocked 6 core cpu on Windows 7, and on a RAID system that can read and write at around 500 Mbs.

When I use MKVtoolnixG, Processhacker shows that remuxing runs at about 180MB/s, which is much faster than the drive speed, and remuxing a 60GB file is pretty fast.

If I display the command line, and cut and paste it into a DOS windows, and run it manually, I get barely 25MB/s, and muxing if much much slower.

In both cases, it's exactly the same command running, but it's much faster from the gui.

Why, and how can I get back the full speed from the command line?

[Mosu]

… remuxing runs at about 180MB/s, which is much faster than the drive speed …

I'm pretty sure you're experiencing the nice effects of caching. If you really want to make speed comparisons, you'll have to re-run the same test multiple times in sequence and discard the first run (e.g. in the GUI add the same job three times and then let the whole queue run to finish; only take the time of the second & third run).

The most important thing is to flush all the caches to disk after completing one test and before moving on to the next. Otherwise the OS will still be busy writing data from the first test to disk while you're already running the second test. That will severely skew the results in favor of the first test, obviously.

In fact, real testing should look ideally look something like this:

Flush all cached data to disk
Run the test and…
Flush all cached data to disk directly afterwards

I cannot tell you how to do 1, but mkvmerge itself can do steps 2 and 3 if you add the command line option "--flush-on-close". In that case the time mkvmerge reports will be the real duration of one round of multiplexing. Of course you must add that option both in the GUI as well as on the command line; both tests must use identical options.

Again, the test is worthless if 1. (flushing before muxing) isn't done.

In the end using the GUI and using mkvmerge directly cannot produce different results as the GUI does nothing more than starting mkvmerge with the command-line parameters you can see via "Multiplexer" → "Show command line". There's no secret option, no additional setup the GUI does.

Benchmarking is real hard.

[robena]

In the end using the GUI and using mkvmerge directly cannot produce different results as the GUI does nothing more than starting mkvmerge with the command-line parameters you can see via "Multiplexer" → "Show command line". There's no secret option, no additional setup the GUI does.

Benchmarking is real hard.

That's a very logical answer, but unfortunately it's not a caching problem.

Muxing with the GUI is always at least 5 times faster, whether I start it first or second.

I did multiple tests like:

1) command line
2) GUI
3) command line
4) GUI

and the result are consistent every time, the GUI is much faster.

Besides, these are UHD very large 60GB files, so memory caching would be filled up very fast.

When I copy a 60GB file from my RAID to a SSD, I see first (sometimes) 1.5Gbs speed, but after a few gigabits, the caching ends and it goes down to the real disk speed.

With the GUI, the whole 60GB run at the same fast speed from beginning to end.

With the command line cust & pasted, the whole 60GB run at the same very slow speed, even if I run it after the GUI run. I just redid the GUI first, command line second second test, and the command line was easily 5 times slower.

Edit: I used the sysinternals "sync -r" command to flush all the drives before each run, same results.

Here is the command line:

"C:/Program Files/MKVToolNix\mkvmerge.exe" --ui-language en --output ^"B:\prob\test3\Lord of War UHD ^(1^).mkv^" --no-subtitles --language 0:eng --track-name ^"0:MPEG-H HEVC Video / 71269 kbps / 2160p / 23.976 fps / 16:9 / Main 10 @ Level 5.1 @ High / 10 bits / HDR10 / BT.2020^" --default-track 0:yes --language 1:eng --track-name ^"1:Dolby TrueHD/Atmos Audio / 7.1 / 48 kHz / 4750 kbps / 24-bit^" --default-track 1:yes --language 2:eng --track-name 2:AC3 ^"^(^" ^"B:\prob\test3\Lord of War UHD.mkv^" ^"^)^" --language 0:und ^"^(^" ^"B:\prob\test3\Lord of War UHD.sup^" ^"^)^" --language 0:und ^"^(^" ^"B:\prob\test3\Lord of War UHD-grey.sup^" ^"^)^" --title ^"Lord of War ^(2005^)^" --flush-on-close --track-order 0:0,0:1,0:2,1:0,2:0

I am at a complete loss to explain that.

[Mosu]

Are you writing to the same file in each iteration? If so you're likely incurring a lot of time from simply deleting the existing huge output file. That's a rather slow operation on NTFS.

I can reproduce your results somewhat, but not if each call to mkvmerge writes to a different file that doesn't exist yet.

So… add "remove output file, flush all of Windows' caching (e.g. by rebooting)" to your list of things to do before each run.

Like I said, proper benchmarking is hard.

Edit: just for kicks: the simple act of deleting ~25 GB in two files on my NTFS SDD takes 1m24s (that's deleting via "del" in cmd.exe or via the Explorer — mkvmerge does something similar: it opens the file in write & truncate mode, meaning it will be cut to 0 bytes of size when opening it which practically does the same as deleting it: all blocks allocated for the file are freed at that moment, and that takes a lot of time on NTFS). That's rather significant given that multiplexing _to_ that file took (3m 42s).

[robena]

Are you writing to the same file in each iteration? If so you're likely incurring a lot of time from simply deleting the existing huge output file. That's a rather slow operation on NTFS.

I can reproduce your results somewhat, but not if each call to mkvmerge writes to a different file that doesn't exist yet.

So… add "remove output file, flush all of Windows' caching (e.g. by rebooting)" to your list of things to do before each run.

Like I said, proper benchmarking is hard.

I did both, either writing the same file after deleting, or letting the GUI increasing the (N) number.

You probably did not see my edit while writing your answer, I found out how to flush all the drives without rebooting with the sync command.

Also, aside these specific tests, I made scores of command line remuxes that were all always extremely slow, and every time I try the GUI, it's 5 time faster. At first I though I was writing poorly my scripts (which I did with the doc, not by looking at the command line in thr GUI), and rewrote them to have the exact same syntax as the GUI. It did not help, and I think that the use of the sync command and of different file names rules out caching.
I also tried to read on one drive and write on another, with exactly the same difference in speed. My RAID is very good at parallel accesses (unlike regular hard drives), I barely notice a difference when using the 2 drives strategy.

Anyway, thanks for spending time on that subject.

[robena]

Another thing: this seems to happen only with UHD HEVC video files.
I did not notice my scripts being slow with 50GB regular HD AVC files. Even weirder!

[robena]

So, nobody has an idea why this is happening, and how to correct it?

[Dulus_No]

Hi!
I've encountered a strange behavior of SPS/PPS duplication when merging and extracting an avc stream.
Used commands:
mkvmerge -o ldaQLGOoJW0_1.mkv ldaQLGOoJW0.avc
mkvextract ldaQLGOoJW0_1.mkv tracks 0:ldaQLGOoJW0_1.avc
mkvmerge -o ldaQLGOoJW0_2.mkv ldaQLGOoJW0_1.avc
mkvextract ldaQLGOoJW0_2.mkv tracks 0:ldaQLGOoJW0_2.avc
mkvmerge -o ldaQLGOoJW0_3.mkv ldaQLGOoJW0_2.avc
mkvextract ldaQLGOoJW0_3.mkv tracks 0:ldaQLGOoJW0_3.avc
H.264 analysis: https://imgur.com/a/rPWNZdm
7z archive with source and result files: https://mega.nz/#!J1hASKpA!VLINH3Ie2vigG5f_8XT6pUCS_NYlA8q9fbA3xiOBZGI

[Mosu]

Yeah, my tools are definitely not trying to deduplicate SPS & PPS (or VPS for HEVC). Instead, they try to achieve best playback compatibility (including seeking to arbitrary places) while not complicating the code too much.

[Mosu]

So, nobody has an idea why this is happening, and how to correct it?

One last idea: when starting mkvmerge manually, do you perchance start it with a lower process priority (e.g. because the process you start it from, such as cmd.exe, is already running with a lower process priority)? Or do you start the GUI (and therefore mkvmerge-started-from-the-GUI) with a higher one? See Windows 10's task manager's "detail" page, right-click on the column header & enable the "base priority" column.

[Perenista]

It's odd. I tried to add a dubbing to a Matroska video and told MKVToolnix to add + 7000 ms as delay, to sync said dubbing audio (AC3) with the video. This time didn't work. However when I open MPC-HC (and both files at the same time) and add 7000 ms manually the audio matches. Now it seems 7000 ms is too much because the audio is still 0.5 second or 1 second out of sync.

(I know the best way to do this is adding these +7 seconds in the dubbing audio and then adding to the MKV without any delay option enabled in MKVToolnix. The problem is that I might have to do this with subtitle streams, too, so it would be a hassle if this didn't work in all cases.

Maybe this time didn't work because 7 seconds is too much?

[robena]

One last idea: when starting mkvmerge manually, do you perchance start it with a lower process priority (e.g. because the process you start it from, such as cmd.exe, is already running with a lower process priority)? Or do you start the GUI (and therefore mkvmerge-started-from-the-GUI) with a higher one? See Windows 10's task manager's "detail" page, right-click on the column header & enable the "base priority" column.
Thanks, I already tried that. I also verified the processor affinity.

I start it 2 ways: normally using a REXX interpreter that I wrote (my scripting language of choice, having worked with an IBM mainframe in the 80s) which does a direct CreateProcess() call without using the Windows cmd.exe. My interpreter has a routine allowing to set the priority, and it changes nothing. I integrated the calls into the shell, so I just have to right click on a file and choose the aspect ratio to create sup files with the command line version of tsmuxer correctly placed at the bottom of the video, and remux then automatically. I get details from the processed files with (again) the command line version of mediainfo. It's a one click process, and why I want to use the command line. Without a script to handle every thing, the GUI would be much easier, but very far from a one click process.

And in case it was my interpreter the culprit, by cutting and pasting the command line as is from the GUI into a DOS window, and checking the priority of the mkvmerge process running with the task manager.

So, still a headscratcher. :)

[mkver]

It's odd. I tried to add a dubbing to a Matroska video and told MKVToolnix to add + 7000 ms as delay, to sync said dubbing audio (AC3) with the video. This time didn't work. However when I open MPC-HC (and both files at the same time) and add 7000 ms manually the audio matches. Now it seems 7000 ms is too much because the audio is still 0.5 second or 1 second out of sync.

(I know the best way to do this is adding these +7 seconds in the dubbing audio and then adding to the MKV without any delay option enabled in MKVToolnix. The problem is that I might have to do this with subtitle streams, too, so it would be a hassle if this didn't work in all cases.

Maybe this time didn't work because 7 seconds is too much?
Is your audio an AC3 elementary stream (that doesn't have explicit timestamps, so that its first audio frame gets a timestamp of zero by default) or a Matroska file (or another container)? If so, what is the lowest timestamp of said audio file? And what about the timestamps of the other file (only the first few timestamps are of interest here)? See my answers here (https://gitlab.com/mbunkus/mkvtoolnix/issues/2174) for a possible reason why this is happening.

[robena]

One last idea: when starting mkvmerge manually, do you perchance start it with a lower process priority (e.g. because the process you start it from, such as cmd.exe, is already running with a lower process priority)? Or do you start the GUI (and therefore mkvmerge-started-from-the-GUI) with a higher one? See Windows 10's task manager's "detail" page, right-click on the column header & enable the "base priority" column.

I use 3 kind of drives: a QNAP basic NAS (600/260 speed), an SSD (500/500) and a SAS RAID Areca DAS (500/500).

Using "sync -r" before each test, remuxing a 60GB UHD movie in command line mode takes 1872s on the Areca, 2053s on the SSD (its cache is full and slows it down) and 523s on the QNAP! With the GUI, the Areca takes 350s. The GUI on the QNAP takes 420s.

These are completely reproducible results. The command line is almost as fast as the GUI over a 10Gbs networked NAS, but almost 6 times slower on a Sata or SAS hard drive (the Areca RAID is seen by Windows like a single SAS drive).

Thoughts?

[mkver]

What about the actual CPU time used by the underlying mkvmerge process for each task? Does it make a difference when started from the GUI?

[Perenista]

Is your audio an AC3 elementary stream (that doesn't have explicit timestamps, so that its first audio frame gets a timestamp of zero by default) or a Matroska file (or another container)? If so, what is the lowest timestamp of said audio file? And what about the timestamps of the other file (only the first few timestamps are of interest here)? See my answers here (https://gitlab.com/mbunkus/mkvtoolnix/issues/2174) for a possible reason why this is happening.I did more tests and 19 others (and different ones) were perfectly fine. There were times I even added 7000 or more than 8000 ms in terms of delay.

The file number 20 was the one with this issue. However I managed to figure out the correct delay to insert: 6350 ms. I checked at the beginning and after 58 minutes and now it's 100% OK. Not sure what went wrong before.

The audio stream I was trying to correct the sync (to be added in my Matroska video) is AC3:

Format : AC-3
Format/Info : Audio Coding 3
Commercial name : Dolby Digital
File size : 352 MiB
Duration : 1 h 49 min
Overall bit rate mode : Constant
Overall bit rate : 448 kb/s

Audio
Format : AC-3
Format/Info : Audio Coding 3
Commercial name : Dolby Digital
Duration : 1 h 49 min
Bit rate mode : Constant
Bit rate : 448 kb/s
Channel(s) : 6 channels
Channel layout : L R C LFE Ls Rs
Sampling rate : 44.1 kHz
Frame rate : 28.711 FPS (1536 SPF)
Bit depth : 16 bits
Compression mode : Lossy
Stream size : 352 MiB (100%)
Service kind : Complete Main

Info about the MKV:
https://pastebin.com/U4beAwL1

[mkver]

1. This is actually not all the data I wanted, because it doesn't tell me anything about the first few timestamps of Matroska file. Use mkvinfo -s -r <output log> <Matroska file> for this; only the first few (say, the first second) of timestamps is of interest here.
2. And you used the elementary stream during determining the offset? Not the stream remuxed into Matroska or so? (Seeking in elementary streams that don't provide timestamps is a PIA, so it is often not instantaneous.)

[dREV]

Hello, not sure if this has been brought up or not. I wanted to bring an issue that I wanted to clarify or just ignore. I made a post on mkvtoolnix page a few months ago asking about this but was told to come here instead to ask my question. Not sure if I should also post this on HEVC's thread as well?

I use an older version of mkvmerge (prefer simple gui then the new one) and also tried the latest portable of version 32.0.0 "Astral Progressions" 64 bit and both still got the same results.

My issue is concerning HEVC's implementation of the code when viewed in Media Info. It's "total-frames=". As I cut frames with Avisynth then append them together to fix problematic scenes, however, when muxing I would receive these messages only on HEVC.

#version 8.3 "Over the Horizon" 64 bit version

mkvmerge finished with a return code of 1. There were warnings, or the process was terminated

Warning: The track number 0 from the file 'C:\Users\Public\# 000\4003, 4146) #banding\# 000 4003, 4146) #banding.mkv' can probably not be appended correctly to the track number 0 from the file 'C:\Users\Public\#000\0, 4002) #Combined\# 000 0, 4002) #Combined.mkv': The codec's private data does not match (lengths: 2277 and 2278). Please make sure that the resulting file plays correctly the whole time. The author of this program will probably not give support for playback issues with the resulting file.

# version 32.0.0

--- Warnings emitted by job 'Multiplexing to file "# 000 - done.mkv" in directory "C:\Users\Public"' started on 2019-03-29 02:14:27 ---The track number 0 from the file 'C:\Users\Public\# 000\4003, 4146) #banding\# 000 4003, 4146) #banding.mkv' can probably not be appended correctly to the track number 0 from the file 'C:\Users\Public\# 000\0, 4002) #Combined\# 000 0, 4002) #Combined.mkv': The codec's private data does not match (lengths: 2277 and 2278). Please make sure that the resulting file plays correctly the whole time. The author of this program will probably not give support for playback issues with the resulting file.

I understand there's no support for the older mkvtoolnix version but as I wrote it's carrying on in the latest. If I try to mess with the total frame line via command line's in MeGUI for HEVC it'll stop working.

When appending then muxing them together regardless of how many frames there are in total now it'll only go by the first mkv cut (ex: frames 0, 4003) and will display that under media info ignoring the appended frames added.

# HEVC x265 3.0_Au+14-c7e5878bdd31

Encoding settings : cpuid=1111039 / frame-threads=3 / numa-pools=12 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=3 / input-res=1440x1080 / interlace=0 / total-frames=4003

So my question is, should I just ignore this as thus far there's been no issues during playback. Is this even an issue or not? It's kinda odd that it's able to recalculate the "Bits/(Pixel*Frame)" for the complete frame count of the video (at least I believe it's so) likewise with H264 but not do the same with HEVC's "total-frames=". There's no issues when doing these cuts/appends on H264 since it doesn't exist there.

Hopefully devs know what I am talking about concerning this and clue me in. :confused:

[sneaker_ger]

You can ignore this in this case as long as the rest of the x265 settings are identical. It's not an issue. The "encoding settings" line you see in MediaInfo is x265's custom "SEI". It's only for information, not part of any standard. Mkvmerge will never alter it in any way. The bitrate on the other hand is taken from Matroska statistic tags (when present) which are in fact updated when appending.

When appending mkvmerge compares the header data of both tracks. When they are not identical it displays a "warning" that there might be a problem. It doesn't actually check whether or not the files are compatible in any meaningful way.

[manolito]

This is my experience,too, using H.264 streams.

I often add 5 seconds of blank video with silent audio to the beginning and the end of my converted MKV files, and as long as I create the "Silence" file with exactly the same settings as the movie then MKVMerge has no complaints.

But recently I noticed that after replacing x264.exe with a newer build I do get these warnings about the codec's private data not matching. The weird thing is that the lengths are identical so I really do not see any mismatch.

The combined file plays perfectly, but just to be on the safe side I reencoded the "Silence" file using the newer x264 build, and the warnings disappeared.

[sneaker_ger]

"sky", "oil". 2 words that are not identical but both have length of 3 letters. ;)

[mkver]

You can use e.g. ffmpeg's trace_headers bitstream filter to see what is in the relevant parameter sets: ffmpeg -i <Inputfile> -c copy -bsf:v trace_headers -t <appropriately short time like 0.1> -report -f null -

[robena]

What about the actual CPU time used by the underlying mkvmerge process for each task? Does it make a difference when started from the GUI?

No, it's pretty low in both cases, below 20%. None of the 6 cores goes over 50%.

[mkver]

I actually meant the accumulated CPU time, i.e. for the whole remuxing, not at a single instant in time. If the average CPU utilization is the same in both cases, but the CLI version uses it longer, then that would be really strange.

[dREV]

You can ignore this in this case as long as the rest of the x265 settings are identical. It's not an issue. The "encoding settings" line you see in MediaInfo is x265's custom "SEI". It's only for information, not part of any standard. Mkvmerge will never alter it in any way. The bitrate on the other hand is taken from Matroska statistic tags (when present) which are in fact updated when appending.

When appending mkvmerge compares the header data of both tracks. When they are not identical it displays a "warning" that there might be a problem. It doesn't actually check whether or not the files are compatible in any meaningful way.

OK. :thanks: Should I post this in HEVC thread to let them know about this? Seems like an oversight for those who cut frames.


This is my experience,too, using H.264 streams.

I often add 5 seconds of blank video with silent audio to the beginning and the end of my converted MKV files, and as long as I create the "Silence" file with exactly the same settings as the movie then MKVMerge has no complaints.

But recently I noticed that after replacing x264.exe with a newer build I do get these warnings about the codec's private data not matching. The weird thing is that the lengths are identical so I really do not see any mismatch.

The combined file plays perfectly, but just to be on the safe side I reencoded the "Silence" file using the newer x264 build, and the warnings disappeared.

I too use .h264 streams and don't receive that. At least with version 8.3 64 bit. I don't get those when appending files together either in .h264 streams or mkv. Even when I've accidentally muxed in the wrong frames it gave me no warning about that. So that's odd you are receiving that. Have you by accident changed the x264 settings from one set of frames to the next? I've accidentally done that and would receive that message as well and as sneaker_ger wrote seems to be so. The x264 I'm using is x264 core 152 r2851kMod ba24899. http://komisar.gin.by/.

I think I tried updating to the newer ones but I think MeGUI couldn't use it since I'm using an old version (ver 2525) in order to still use .h264 streams via DGI as lsmash takes a bit longer to index.

[sneaker_ger]

OK. :thanks: Should I post this in HEVC thread to let them know about this? Seems like an oversight for those who cut frames.
Let who know? It's not a bug.

[robena]

I actually meant the accumulated CPU time, i.e. for the whole remuxing, not at a single instant in time. If the average CPU utilization is the same in both cases, but the CLI version uses it longer, then that would be really strange.
Well, I not sure how to measure it precisely, but since the CLI lasts 6 times more with about the same average, the accumulated should be higher.
If you know a tool to measure the accumulated time of a process, I'll make the measurement.
Anyway, I have decided to replace my DAS by another NAS, more powerful with a 20Gbs aggregated Ethernet link, so the problem will be soon moot I hope.

[hello_hello]

This is my experience,too, using H.264 streams.

I often add 5 seconds of blank video with silent audio to the beginning and the end of my converted MKV files, and as long as I create the "Silence" file with exactly the same settings as the movie then MKVMerge has no complaints.

But recently I noticed that after replacing x264.exe with a newer build I do get these warnings about the codec's private data not matching. The weird thing is that the lengths are identical so I really do not see any mismatch.

The combined file plays perfectly, but just to be on the safe side I reencoded the "Silence" file using the newer x264 build, and the warnings disappeared.

I always include --stitchable in the x264 command line. I never know when I'm going to need to split off a bit off an encode and re-do it to fix a problem and/or do some appending.

For reasons I don't understand, when x264 is writing directly to MKV, or you're appending MKV's, the warning is more likely to occur.
I've never understood why sometimes I can extract the raw h264 streams from each MKV to be appended (after MKVToolNix produced the codec private data warning), append the raw streams, and if the encoder settings were identical sometimes that'll make the codec private data warning go away, even if --stitchable wasn't used.

I'm happy to learn why... should anyone would care to explain.

Cheers.

PS. I'm using x264 core 155 r2901 7d0ff22 (with 32 bit MeGUI), as last time I tried to upgrade it, the newer x264 wouldn't run on XP. Is that still the case with the latest version?

[sneaker_ger]

Mkvmerge displays a warning if "CodecPrivate" element of 2 (or more) mkv files you wish to append aren't identical. "CodecPrivate" is an element of a Matroska container (https://matroska.org/technical/specs/index.html#CodecPrivate). So if you append 2 raw H.264 streams there is no "CodecPrivate". So there is no warning. The underlying problem remains the same, though.

Note that x264cli behaves a bit different depending on how you output:
to .mkv: H.264 headers (SPS/PPS) only at beginning of file
to .264: SPS/PPS at every keyframe
The latter can mitigate problems when appending files with different headers because the decoder can always access the correct PPS/SPS.


Note that not using --stitchable doesn't necessarily make files incompatible. But I think it is indeed a good idea to activate in case you want to make editing later.

[manolito]

PS. I'm using x264 core 155 r2901 7d0ff22 (with 32 bit MeGUI), as last time I tried to upgrade it, the newer x264 wouldn't run on XP. Is that still the case with the latest version?

The behavior I was talking about happened with LigH's latest build r2969 under Win7-64. My XP computer does not really handle HD sources. The latest x264 version which runs under XP and does not need SSE2 is 0.148.2579.0 from 2015.

I never used the --stitchable parameter in x264 because at the time when I created this "Silence" clip I used the same x264 build which was used for the movie conversion. The warning started to appear after I updated x264 to a newer build. For the movie conversion the parameters were absolutely identical. So I believe that every new build probably creates different "private data" even when identical parameters are used.

Maybe mkvmerge should be a little less strict when it issues this warning...


Cheers
manolito

[dREV]

Let who know? It's not a bug.

To the HEVC developers.

I also did a test by doing two different frames. Cutting one with (0, 1) and the other with (2, 900) and the info showed total frames= 899 so I wanted to know what the documentation means by https://x265.readthedocs.io/en/latest/cli.html

If --total-frames is 1, then a stillpicture variant will be signaled, but this parameter is not always set by applications, particularly not when the CLI uses stdin streaming or when libx265 is used by third-party applications.

So wanted to let them know for those who cut frames and append and see if it wasn't an oversight.

The behavior I was talking about happened with LigH's latest build r2969 under Win7-64. My XP computer does not really handle HD sources. The latest x264 version which runs under XP and does not need SSE2 is 0.148.2579.0 from 2015.

Didn't know you use XP. I'm on Windows 7 so I see why I am not expiriencing the same issue as you're going through.

[mkver]

Well, I not sure how to measure it precisely, but since the CLI lasts 6 times more with about the same average, the accumulated should be higher.
If you know a tool to measure the accumulated time of a process, I'll make the measurement.
Anyway, I have decided to replace my DAS by another NAS, more powerful with a 20Gbs aggregated Ethernet link, so the problem will be soon moot I hope.
The easiest way to measure this is by using the Taskmanager: Make it show the CPU Time of your processes and take a look at the value of this counter right before the process ends.
(This is not a perfect benchmark: If a process takes (say) 10 % of a core's CPU time, then after 10s the CPU time will be 1s higher -- regardless of whether your CPU is currently running at its maximum/default frequency or not.)

[filler56789]

PS. I'm using x264 core 155 r2901 7d0ff22 (with 32 bit MeGUI), as last time I tried to upgrade it, the newer x264 wouldn't run on XP. Is that still the case with the latest version?

When you try to run x264.exe on its own, i.e., "outside of" MeGUI,
what is the error message(s) that Windows XP returns? :confused:

[robena]

The easiest way to measure this is by using the Taskmanager: Make it show the CPU Time of your processes and take a look at the value of this counter right before the process ends.
(This is not a perfect benchmark: If a process takes (say) 10 % of a core's CPU time, then after 10s the CPU time will be 1s higher -- regardless of whether your CPU is currently running at its maximum/default frequency or not.)

I used something more precise, the command:

typeperf "\Process(mkvmerge)\% Processor Time" -si 10 -o mkvmerge.txt

For the command line, directly cut pasted form the GUI, I got a file that looks like:



"(PDH-CSV 4.0) (Romance Standard Time)(-60)","\\NT\Process(mkvmerge)\% Processor Time"
"03/30/2019 19:58:15.817"," "
"03/30/2019 19:58:25.822","96.552356850679971"
"03/30/2019 19:58:35.824","96.859476872593547"
"03/30/2019 19:58:45.826","98.107263033370458"
"03/30/2019 19:58:55.828","97.327396077620293"
.....
"03/30/2019 20:28:15.975","97.483370443431511"
"03/30/2019 20:28:25.977","98.107263033370458"
"03/30/2019 20:28:35.978","98.112168754568174"
"03/30/2019 20:28:45.981"," "
"03/30/2019 20:28:55.983"," "

The sum of all not NULL times was: 16144.6679175611
Elapsed time was 19:58:25 -> 20:28:45, 32m20s

Then, I ran the GUI, after a "sync" command and got:


"(PDH-CSV 4.0) (Romance Standard Time)(-60)","\\NT\Process(mkvmerge)\% Processor Time"
"03/30/2019 20:51:55.756"," "
"03/30/2019 20:52:05.760","35.717906573357453"
"03/30/2019 20:52:15.762","31.818571794606637"
"03/30/2019 20:52:25.774","28.514617588198405"
.....
"03/30/2019 20:57:25.883","34.470119444157191"
"03/30/2019 20:57:35.885","37.121667093707742"
"03/30/2019 20:57:45.888"," "


Total CPU time was: 1017.40205400394
Elapsed time was 20:52:05 -> 20:57:45, 5m40s

As you can see, both the elapsed time and the CPU usage are way lower from the GUI.

[hello_hello]

The behavior I was talking about happened with LigH's latest build r2969 under Win7-64. My XP computer does not really handle HD sources. The latest x264 version which runs under XP and does not need SSE2 is 0.148.2579.0 from 2015.

I never used the --stitchable parameter in x264 because at the time when I created this "Silence" clip I used the same x264 build which was used for the movie conversion. The warning started to appear after I updated x264 to a newer build. For the movie conversion the parameters were absolutely identical. So I believe that every new build probably creates different "private data" even when identical parameters are used.

I assume the timeline went something like this:

1: x264 doesn't optimize global stream headers
2: x264 does optimise global stream headers
3: Users complain about not being able to append encoded video
4: --stitchable added to prevent x264 from optimising global stream headers
5: Fewer users complain about not being able to append encoded video

--stitchable became a thing in 2013 (https://github.com/yixia/x264/commit/8c8b46f1fbcf1c4b684afff9160d074077909965), so if you created the original silence clip before then, it'd make sense. If not, I don't know.........

I've definitely had problems appending encoded video in the past, even when the same x264 build was used and the same settings. Although --stitchable has always prevented the warning when appending constant frame rate encodes. For variable frame rate encodes it's still a little hit and miss. That could be because x264 is writing directly to MKV (for reasons I don't understand) as if MKVToolNix has warned about the codec's private data when appending VFR video in MKVs, extracting the raw streams and appending those instead has always worked.

An SSE2 requirement isn't an issue for me, but if x264 could find that missing entry point in kernel32.dl and obsess over bcrypt.dll a little less, I wouldn't complain. That started sometime after build 2091.

Cheers.

[sneaker_ger]

You already asked about fps in the past. It changed headers.
https://forum.doom9.org/showthread.php?p=1822942

[filler56789]

An SSE2 requirement isn't an issue for me, but if x264 could find that missing entry point in kernel32.dll and obsess over bcrypt.dll a little less, I wouldn't complain. That started sometime after build 2091.

Just my wild guess... the problem is not the recent revisions of x264, it's whether the .EXE uses win32thread or not. If you compile x264 without win32thread, then it should run normally on Windows XP
(or at least I think so).
I don't have a XP machine anymore, so I cannot test my hypothesis...

[hello_hello]

Sorry filler56789,
I missed your earlier post.

I can't remember if Windows offered an error message as such, but the first hurdle was x264 not being able to find bcrypt.dll. I downloaded it just to see what'd happen, then the next hurdle was a procedure entry point in kernel32.dll, so I gave up. That's much the same as the latest ffmpeg compiled without XP support (and MKVToolNixGUI after version 10), and while I don't know what win32thread is just yet, I suspect you're on the right track.

Cheers.

[hello_hello]

You already asked about fps in the past. It changed headers.
https://forum.doom9.org/showthread.php?p=1822942

I completely forgot about that.

Thanks.

[lvqcl]

I can't remember if Windows offered an error message as such, but the first hurdle was x264 not being able to find bcrypt.dll. I downloaded it just to see what'd happen, then the next hurdle was a procedure entry point in kernel32.dll, so I gave up. That's much the same as the latest ffmpeg compiled without XP support (and MKVToolNixGUI after version 10), and while I don't know what win32thread is just yet, I suspect you're on the right track.


From x264 configure file:
External library support:
--disable-avs disable avisynth support
--disable-swscale disable swscale support
--disable-lavf disable libavformat support
--disable-ffms disable ffmpegsource support
--disable-gpac disable gpac support
--disable-lsmash disable lsmash support

So, it looks like x264 is linked with some ffmpeg libraries by default. r2901 contains string "FFmpeg version git-2018-01-18-81c09ce34d", while r2935 contains "FFmpeg version git-2018-09-23-77c673920b".
Apparently, newer x264 -> newer ffmpeg -> no WinXP support.

(sorry for offtopic)

[Mosu]

Hey,

another month, time for another release of MKVToolNix: v33.0.0. This is still a rather quiet time for MKVToolNix; hence this is another small bug fix release.

There've been no changes regarding packaging since the previous release.

Here are the usual links: the MKVToolNix home page (https://mkvtoolnix.download/), the Windows installer/portable version & macOS DMG & Linux AppImage (https://www.fosshub.com/MKVToolNix.html) and the source code (https://mkvtoolnix.download/source.html).

The Windows and macOS binaries as well as the Linux AppImage are available already. The other Linux binaries are still being built and will be available over the course of the next couple of hours.

Here are the NEWS (https://mkvtoolnix.download/doc/NEWS.md) since the previous release:

# Version 33.0.0 "A Little Bit of Madness" 2019-04-12

## Bug fixes

* mkvinfo: when using the `--size` option, mkvinfo will now report the correctly if an element has an unknown size. Part of the fix of #2530 (https://gitlab.com/mbunkus/mkvtoolnix/issues/2530).
* MKVToolNix GUI: info tool: clusters with an unknown size will now be read and displayed correctly. Part of the fix of #2530 (https://gitlab.com/mbunkus/mkvtoolnix/issues/2530).
* MKVToolNix GUI: multiplexer: Windows: trying to open Blu-ray index or playlist files failed when the path to the files contained symbolic links (e.g. when mounting a drive in a sub-folder via Windows' disk management utility). Fixes #2522 (https://gitlab.com/mbunkus/mkvtoolnix/issues/2522).
* MKVToolNix GUI: multiplexer: if a destination file names ends with a number in parenthesis (e.g. a year such as "(2017)"), that number will not be stripped anymore during the process of ensuring the destination file name is unique. Only those suffixes added automatically in prior attempts to make the file name unique will be removed. Fixes #2521 (https://gitlab.com/mbunkus/mkvtoolnix/issues/2521).
* MKVToolNix GUI: multiplexer: Windows: the GUI will let the user change the drive letter part of the destination file name freely again and only verify its validity right before starting to mux/adding to the job queue. Before it tried to force that into something valid, often resulting in unintentional paths (such as "C:\users\…\DC\files\…"). Fixes #2527 (https://gitlab.com/mbunkus/mkvtoolnix/issues/2527).

Have fun :)

[stax76]

Thanks for the release. Minor cosmetic request for you or anybody else that posts markdown: BB code instead of MD code would be easier to read. It could be converted with a script that reads MD from the clipboard and copies BB back and for convenience the script could be started with a launcher like Alfred or Wox or via command line.

before: https://forum.doom9.org/showpost.php?p=1870861&postcount=88

after: https://forum.doom9.org/showpost.php?p=1870995&postcount=90

node script: https://pastebin.com/ru5xe4EE