Matroska Overhead Calculation

[Atamido]

The code that was originally being developed to calculate overhead in Matroska has been put on hold until further notice. The developer is working on an MPEG-2 to MKV transcoder. At the moment the best overhead calculator comes from alexnoe in his AVI-Mux GUI (http://www-user.tu-chemnitz.de/~noe/Video-Zeug/AVIMux%20GUI/en_mkv_status.html). The source code is available, but is not the most recent version.

It has not been publicly announced yet, but a new lacing scheme has been developed that allows for a much lower overhead. However it also changes the calculations for overhead. Once the overhead calculation is completed in the new AVI-Mux GUI, then the Gordian Knot developers are encouraged to use it. Current source code is available on the above page and is useful in that overhead will not increase with the new lacing system.

Please take a look at the new overhead comparison for an idea of the new overhead required by Matroksa. overhead comparison (http://www-user.tu-chemnitz.de/~noe/Video-Zeug/AVIMux%20GUI/en_overhead_comparison.html)

[len0x]

Huge development team of GK (i.e. me and tHe gLouCh) personally would like to thank members of matroska team for providing prompt information about progress of bitrate calculator for the last 4 month :D
Jokes aside - I just don't get one thing: having info about how matroska works (I assume that matroska team knows that) it should take not more than a couple of hours to produce a pure math formula (even rough estimation) of the overhead. I never asked for code to be written for me - in fact I always though that it's a doomed project to write bitrate calculator and provide it via dll to others. All I need now is just a math formula (it can be even experimental estimation).

From now on I will not be looking at source code of others to find what formula they're using. So if you could extract a formula from say avimux and post here, I'd appreciate that. Otherwise I just don't have time to that myself...

P.S. Thank god I haven't done much on mkv overhead - I would be really upset if you told me that overhead will change after I actually implemented everything...

[stax76]

I'm using this formula atm

mkv: overhead in kb = Frames * 0.013

so far nobody complained, OGM seem to be more problematic. I changed OGM lately because of complains. Damn it, I know I changed the OGM formula lately but when I now look at my source I see the old formula :scared:

[len0x]

Thanks, I'll try to use that actually for MKV. Is this total overhead including audio or just container overhead?

Regarding OGM - so far we got pretty good results with the following: usual avi overhead (with no audio) + 3 mb per 700mb of target size...

[alexnoe]

If you care to have a look at my overhead page, you'll see that only the output parameters can make the difference between 10,6 and 2,6 MB of overhead. So any formula like this is just nonsense.

Better working overhead code is now available on the download page of the matroska version of avi-mux gui. But note that due to the nature of EBML lacing, it is impossible to promise an accuracy better than +/- 0.5 bytes/audio frame.

[len0x]

Originally posted by alexnoe
If you care to have a look at my overhead page, you'll see that only the output parameters can make the difference between 10,6 and 2,6 MB of overhead. So any formula like this is just nonsense

that's what most ppl used to say about OGM overhead. And yet some ppl constantly getting good estimations in real life :)

[alexnoe]

The difference might be that I *demonstrated* those numbers

[Atamido]

Please follow the link in the first post as the source has now been updated with the overhead calculations for the new lacing scheme.

[alexnoe]

What you need is found in Muxing.cpp (begin of MKV muxing thread), Matroska.h and Matroska.cpp (top of file)

[len0x]

Btw, there is one more issue - I need overhead estimations with the parameters which are used in VDubMod 1.5.1.1a by default. Coz the the version that is used now...

P.S. Personally I would wait until developmen of both mkv and VDubMod was in synch and constant for this matter...

[alexnoe]

Just look into the settings of VDubMod. Some params are also limited my libmatroska (which I don't use)
-> Clusters of 5 secs, but can be adjusted
-> lacing either off or 8 frames per lace
-> currently afaik only xiph lacing available
-> one cue per keyframe
-> won't write PrevClusterSize or ClusterPosition

You will need the audio frame duration, and in case of xiph lacing also the audio frame size, in any case

[len0x]

It will still take me a little while to dig through the code... Dunno when I have time though. Thanks for efforts anyway.

[alexnoe]

I sometimes do not care too much about the readability of my code :devil:

[ChristianHJW]

Originally posted by len0x Jokes aside - I just don't get one thing: having info about how matroska works (I assume that matroska team knows that) it should take not more than a couple of hours to produce a pure math formula (even rough estimation) of the overhead. I never asked for code to be written for me - in fact I always though that it's a doomed project to write bitrate calculator and provide it via dll to others. All I need now is just a math formula (it can be even experimental estimation).

Toni, again sorry for that, but unfortunately its not as easy as it may seem in a first glance. matroska is a pretty powerful container, and as such it can use different lacing shemes for different audio formats, depending on whether it has a constant ( MP3, AC3, AAC, etc. ) or a variable ( Vorbis ) number of samples per frame/block. Not to mention that there are a number of parameters like max. cluster lenth, etc. that are user configurable and have direct influence in the overhead, even if they are not so big.

I was telling the guys ever since that some basic formulas to get a good approximation is enough, but these guys are real technicians, they wnat to have it precise or not .... and as nobody had the time to do it exactly, it was NOT done at all ;) ..... :sigh: .... technicians .... :sigh: ....

[stax76]

Is this total overhead including audio or just container overhead?


it is video overhead only, there is also a little bit audio overhead
but I think it's not much (50-150KB each stream) so I don't care about
it atm


So any formula like this is just nonsense


it seem to be in a range within +- 100 KB if you don't change
the mkv settings in VDM. I'm still afraid a more precise calculation
could be too time consuming or too hard to figure out and since
a estimation works pretty good I'm not sure it's worth the effort

note: I'm only a lame VB programmer ;)

[alexnoe]

it seem to be in a range within +- 100 KB if you don't change
the mkv settings in VDMYes, if you don't change any settings....but since AVI-Mux GUI does not include limits set by libmatroska (such as lace length), there is some thing to change ;)

Look at my source code :D BTW, reading the matroska specs also helps to find out how to estimate the overhead...or at least to understand why it really depends on muxing settings
Btw, there is one more issue - I need overhead estimations with the parameters which are used in VDubMod 1.5.1.1a by default. Coz the the version that is used now...I hope you don't speak about any build earlier or equal to 2092. The version of libebml used in those builds was b0rked

[len0x]

Originally posted by alexnoe
I hope you don't speak about any build earlier or equal to 2092. The version of libebml used in those builds was b0rked

I checked source forge - 1.5.1.1a has build number 1155, which basically rules out usage of mkv at the moment anyway :)

[alexnoe]

And you can't generate AVI-Mux GUI or mkvmerge scripts and use those for muxing because....?

[len0x]

Originally posted by alexnoe
And you can't generate AVI-Mux GUI or mkvmerge scripts and use those for muxing because....?

it's large architectural change which I have no time for...

[Christos]

quote: Originally posted by alexnoe:
And you can't generate AVI-Mux GUI or mkvmerge scripts and use those for muxing because....?

quote: Originally Posted by len0x:
it's large architectural change which I have no time for...

@alexnoe + @len0x
Maybe am being stupid but here is a crazy idea... :)
Could alexnoe maybe help a little?

@alexnoe
I know GK is in Delphi/Kylix but.. maybe your help would speed things up and it wouldn't take as match time as it would...

@lenox
We all can see that matroska has what it takes to become the future standard container but it could use some help...
And GK supporting it would be a great help...

[Tuning]

Originally posted by Christos
And GK supporting it would be a great help...

len0x has already mentioned support of Matrosk/OGM in later release.(May be 0.29...).

@len0x,
Is there a plan of including mp4(or any other container...) in future release.

[bond]

as i understand len0x, he will not use any other muxing tool than virtualdubmod in the near future, so container-, audio-formats, features which arent included in virtualdubmod will not be available in gknot...

[alexnoe]

it's large architectural change which I have no time for...The only architectural change in AVI-Mux GUI for you is that setting manual split points using scripts is not compatible to how it is done in 1.14. I hope you did not seriously think about pressing buttons using WM_LBUTTONDOWN messages....

[len0x]

Originally posted by alexnoe
The only architectural change in AVI-Mux GUI for you is that...


I was never using AVI-Mux within GK. When I said architecture I meant that the part of the job in GK consist of execution of VCF script by VDubMod. Scripts are differently constructed but always executed in the same way. Execution of any other external program is kinda incompatible with this flow (not impossible of course, but time is needed).

P.S. 2All: I'm leaving VDubMod build 2178 within GK package atm, so mkv has a good chance of being supported in a first public alpha of 0.29.

[alexnoe]

Good. Then you won't write b0rked AVI files...

[ChristianHJW]

Originally posted by len0x P.S. 2All: I'm leaving VDubMod build 2178 within GK package atm, so mkv has a good chance of being supported in a first public alpha of 0.29.

:) :) !!!!!! With this build you will write beautiful MKV files :) ! Great decision !!

[pixolex]

there any news about the calculation of overhead in the alfa versions? is any thing implemented? :)

Thanks!

[len0x]

well, if you look in the next thread in this forum - you'll see what i'm doing atm :) as soon as i get gkif4u stable enough i'll port calculations from avimuxgui to gk and make public alpha version. in fact public alpha may come before that but i cannot promise anything before i've done some stuff for gkif4u...

[pixolex]

thanks for fast response!

And good work with Jr. GK :D

[jkwarras]

I don't know if people have found the same thing that me, but Matroska Overhead is almost inexistent :D I mean, when muxing (even a very large file) a video and audio file (mp3/vorbis...) for me the results it almost the same as: video size + audio size, no overhead, and very strange but true sometimes the resulting muxed file is SMALLER than the sum of the files without muxing!

I use VirtualdubMOd (last version) and right now mkvmerge last release and get always this results (audio: cue none; video: cue only i-frames).

I love this container :cool:

[Hiro2k]

That's because mkvmerge uses new lacing schemes. I don't think that VirtualDubMod supports that new lacing scheme, because the new lacing method is incompatible with old libs. I don't think that build 2178 supports the new lacing yet.

[alexnoe]

EBML lacing is virtually unpredictable. You would have to do 2-pass muxing :D

Here's a protocol from a muxing process with avi-mux gui that should demonstrate where the problem is:

19:02:01.968 estimated overhead: 3,443,884 bytes
19:02:01.968 estimated raw size: 1,277,240,320 bytes
19:02:01.968 Begin muxing...
19:02:02.000 new file at 0:00:00.000 (frame 0):
I:\Filme\CDs\Starship Troopers.mkv
19:29:06.171 4,958 clusters written causing 104,008 Bytes of overhead
19:29:06.171 created 1,507 cue points for video stream
19:29:06.171 track 1:
EBML lacing used in 11,657 blocks containing 167,852 frames
Size of lace headers: 179,509 bytes
19:29:06.187 track 1:
fixed lacing used in 8,159 blocks containing 117,732 frames
Size of lace headers: 8,159 bytes
19:29:06.484 overhead written: 3,038,177 bytes
19:29:06.500 done!