Overhead in OGM and MKV?

[SLider3]

Hello!!
I´m coding my own bitrate calculator but I need to know the formulas to calculate the overhead of ogg, mp3 and ac3 in OGM and in MKV.

Can someone tell me that?

[stax76]

http://forum.doom9.org/showthread.php?s=&threadid=52670&highlight=overhead

[ChristianHJW]

spyder is still fighting with a defect mobo, so he cant do any development on Windows right now .... hopefully in the course of next week he will have his box restored, and be able to start developing his DLL ....

[spyder]

Hopefully I will be back to coding this weekend :)

Starting Thursday I should be able to code until Monday :) This means we will likely have a overhead calc routine this weekend. And also, to any RV9 users, please please please bug the Real people to get back to us ASAP on supporting RealMedia in Matroska.

[SLider3]

@spyder

I´ll be waiting for your reply.:D

[spyder]

Well, I have started planning this code. This will be more complicated than you would think ;) I will try to make the calculation as simple and accurate as possible. To help though, please give me an idea of what information is available to you for the calculation.

[stax76]

bitrate, size, length and type of the stream (no advanced things like B-Frames)

[ChristianHJW]

What you need :

- number of streams, stream types
- number of clusters ( estimated )
- number of frames
- lacing yes/no

anything i forgot ?

[stax76]

what is clusters and lacing?

[Atamido]

Clusters are a way of grouping blocks together. If you are using an accuracy of 1ms for the timestamp on frames (the default) then the timestamp field on the Block only has an accuracy of about 64 seconds. This was done to increase the efficiency of the Block design. But, by grouping together Blocks in Clusters, you overcome this limit. It also increases seeking speed. (I think it is mostly the same as Atoms in MP4)

Currently the Cluster size can be set by both timecode and bytes. IE, if the next block exceeds 5 seconds, or 7 megabytes for this cluster, create a new cluster.

Refering to the Diagram (http://cvs.corecodec.org/cgi-bin/cvsweb.cgi/~checkout~/matroska/doc/website/specs/diagram/index.html) may help you to understand the cluster concept better.


Lacing is a technique to decrease overhead by storing several audio samples in a single Block. It is mainly useful for low bitrate audio. Overhead adjustments for this will be difficult because the Matroska library automatically determines when to use this by calculating on the fly when it will decrease overhead.

[stax76]

I was hoping for a approximately method without analysing the streams knowing nothing but the file extension, size and length

[ChristianHJW]

Originally posted by Dolemite
I was hoping for a approximately method without analysing the streams knowing nothing but the file extension, size and length ... forget it, sorry ..... overhead of a matroska file, when looking at 2 extreme cases for the same file, could vary by factor 100 easily ...

[stax76]

I have to calculate the overhead before encoding anything anyway

[Atamido]

If you leave everything as with the standard settings, and you know the framerate and average bitrate, then you should be able to get a pretty good guess at the overhead. The amount of overhead can vary drastically though depending on the options you select.

[SLider3]

Originaly Posted by spyder
Well, I have started planning this code. This will be more complicated than you would think ;) I will try to make the calculation as simple and accurate as possible. To help though, please give me an idea of what information is available to you for the calculation.
Sorry the late. I think they have already answered, but here i go...
The information avaiable for the calculation is:
- number of streams and stream types
- number of frames
- frame rate
- no need advanced thinks (at least, not now)
- i think i forgot something :D


For avi there is formulas to calculate overhead very accurate. Is so hard to find this formulas for OGM and Matroska overhead?

[spyder]

Well, the problem with calculating Matroska overhead is that it can vary depending on the data in the file. Because Matroska uses variable length coding of the fields and elements, the overhead is not perfectly calculated without the full details that are only available after encoding. I just ignore the overhead alltogether since it is very slight. I know this is not the best solution for some of you so I am making this C DLL. Hopefully it will be available tomorrow. Maybe before then if I can get undistracted by OpenGL. ;)

Spyder

[spyder]

Ok, I have preliminary code for the DLL. It's not a DLL yet but it's all coded in C. I am writing a test app to check it with and then I will release it once it is halfway accurate ;) The interface is simple and should not change between revisions. Currently I have 3 methods:


//This method initializes the KaxOverhead structure (Call first)
//This takes a pointer for a YET TO BE ALLOCATED KaxOverhead struct as a paramter,
//not an actual structure.
int KaxOverheadInit(KaxOverhead* ovh, int numStreams);

//This method frees the data used by the structure. (Call last)
//After this call the pointer will be NULL.
int KaxOverheadDestroy(KaxOverhead* ovh);

//This method performs the calculation and returns the estimated overhead size in bytes.
unsigned long int KaxOverheadCalculate(KaxOverhead* ovh);


I know this could have been doen in C++ but I prefer C. The way this works is that you declare a pointer for a KaxOverhead structure and then initialize it with KaxOverheadInit. Next you fill in the parts of the structure that you can. There are many fields in it but few are required. Many are optional and only make the calculation more accurate. For example, if you know the number of key frames in the output, the Cue data can be more accurately calculated if you set that in the structure. The number of keyframes shouldn't be hard to determine for an XviD encoder since the stats file stores the decisions I think. Anyway, it's still missing some important estimation code and I have yet to figure out how I will estimate the number of blocks in a vorbis stream. The code also allows calulation with subtitle data and with streams of different lengths :)

Once I have debugged it and made sure it is decently accurate, I will post the code here under GPL and probably make a DLL and some interface code for you all :)

Spyder

[geoffwa]

So far I've been playing around with OGM/MKV muxing one video, two vorbis, and one SRT stream. Generally MKV winds up being smaller (by about 10KB for 3 hours), although having said that seeking is still better in OGM (hoping that gets fixed).

[spyder]

Ok, i have worked on mp3 overhead and the results are pretty close:

Original overhead file 1: 20374
Projected overhead file 1: 20194

Original overhead file 1: 25187
Projected overhead file 1: 27727

As you can see, VBR MP3 makes accurate prediction difficult ;)

I am trying to find a way to test the video overhead.

Spyder

[spyder]

OK, I have a pretty accurate routine now. It only handles video(tuned for XviD currently) and MP3 audio at the moment. It is pretty accurate and gets pretty close with VBR mp3 files turned to mka. For the video size, I got a 20KB difference in the predicted size and the actual overhead on a 640MB 1hr 18min clip. That's not bad to me :) This code currently uses no cue data. I will work on that and try testing it on a 2 stream file. Also, I have no idea how i will find the overhead of Ogg on a vorbis stream. That is one thing I am missing still, along with AAC and AC3 audio types. Then I will need to break video into a few types. Then I can work on sub stream calculations too. For the sub calculation, you will need to define a number of frames for it. If this is a problem, you will have to estimate from the filesize. This keeps me from having to maintain the code too much after the first stable release.

[ChristianHJW]

Great work spyder !! Wonder when len0x will show up here ...

[tHe gLouCh]

Originally posted by spyder
I have no idea how i will find the overhead of Ogg on a vorbis stream.
Perso I use this formula in GordianKnot GL

For 1 stream ogg:

Overhead=(Frames*190/1024)
Overhead=Overhead + ((23.086+(Frames*.19140625))* oggFactor)
Overhead=Overhead/8;

I have tested different values for oggFactor, and it seems that 0.72 is good.
[Edit] I mean when the container is OGM !

[SLider3]

@spyder
Great work ;) When you get some very accurate formulas, share it so we can make some tests...

@tHe gLouCh
Thanks for your help ;)

[spyder]

@SLider3:

I am trying to complete the code first to include Vorbis,AC3, and AAC.
Then I will release it.

[bond]

dont forget good old .mp3 ;)

[spyder]

@bond: keep up son, mp3 was working a while back :)

[bond]

ups should read everything carefully before posting :D

[SLider3]

@spyder
Can you please post here de results you get till now. :confused:

[ChristianHJW]

Originally posted by SLider3
@spyder :Can you please post here de results you get till now. :confused:

What results ?

To make this clear one more time, there are no 'rule of thumb' overhead calculations for matroska. Thats why he made the .exe he sent to len0x now, to be able to make a precise calculation for every type of video/audio stream, sampling rate, etc ...

[spyder]

The code I sent len0x is not complete. It needs a little work to be cleaned up etc. I will not be able to work on it much for the next week as I will be gone most of it. When I get back, maybe I can clean it up and add precise Vorbis support and maybe sub support. Until then, len0x has the newest code. I am not releasing it to anyone else yet because I don't think I will need it is ready for public use.

[GlitchOfDoom]

I am hoping to aid in the development of a bitrate calculator for OGM. I recently muxed a OGM movie Lord of the Rings the final size before mux was 1,314,652Mb after mux was 1,420,366Mb. My target size was 2 cd's (1.4Gb). Post back if you need anything else.

Here is the Calculator log:

Calculator:
VideoSize: 1309229 kbyte.
Audio1Size: 108207 kbyte.
Audio2Size: 0 kbyte.
Overhead: 16163 kbyte.
AviSize: 1433600 kbyte (1400 mbyte).
FilesSize: 0 kbyte.
TotalSize: 1433600 kbyte (1400 mbyte).
Interleaving: vbrmp3
Bitrate: 1002 k(=1000)Bits/s
Frames: 256560
FPS: 23.976
Duration: 258:21
Quality: 0.255 bits/(pixel*frame).
CompCheck3: 0.000 bits/(pixel*frame).
CompCheck5: 0.404 bits/(pixel*frame).
Resolution: 640 x 256

[len0x]

Finally bond pointed me to this thread.
I've implemented basic version of MKV calculator in GK, but I still have some questions (which I asked here: http://forum.doom9.org/showthread.php?s=&threadid=56964 )

Basically main point at the moment is that GK doesn't have any control over MKV parameters (like lacing etc) and I have no idea what are the defaults used by VDubMod...

When spyder have some more code ready I'll port it to GK (it's very simple calculations anyway).

Regarding OGM overhead - I saw Koepi's estimations (in OGMCalc). Will try to start from there.

Although it's getting quite complicated since we have so many type of audio/video for containers...

[stax76]

does anybody know the formula XviD uses for overhead, I don't understand why XviD calculates overhead, isn't there a difference between container formats

[stax76]

approximately calculation seem to work pretty
good. My current approach is the code below,
test build can be found here

http://www.planetdvb.net/board/index.php?act=ST&f=22&t=498&st=0&#entry3158


Public Shared Function GetOverheadKB() As Integer
Dim Overhead As Double
Dim Frames As Integer = CInt(p.TargetSeconds * Target.GetRate)

Select Case p.Container
Case "avi"
Const VBR As String = "0x00000202,1"
Overhead += Frames * 0.024
If Contains(p.Audio0.VD, VBR) Then
Overhead += Frames * 0.04
ElseIf p.Audio0.VD <> "" Then
Overhead += Frames * 0.023
End If
If Contains(p.Audio1.VD, VBR) Then
Overhead += Frames * 0.04
ElseIf p.Audio1.VD <> "" Then
Overhead += Frames * 0.023
End If
Case "ogm"
Overhead += Frames * 0.069
Case "mkv"
Overhead += Frames * 0.013
End Select
Return CInt(Overhead)
End Function

[Nic]

"does anybody know the formula XviD uses for overhead"?
You mean the overhead calculated in the MiniCalc thing? That assumes AVI and uses the same numbers as GKnot uses for AVI overhead calculation

-Nic

[stax76]

I'm not sure, if you enter the kb in the XviD dialog you will get this size if you use AVI because XviD takes care of AVI video overhead for you (while DivX doesn't), XviD should produce undersize for mkv because mkv overhead is very small

[Nic]

Oh, I see what you mean, interesting point. Ill look into that and get back to you.

-Nic

ps
Ok just looked it up:
(2pass.c)
// compensate for avi frame overhead
desired -= frames * 24;

Is added to account for AVI Frame overhead. Hmmm, Maybe we should consider taking that out now, but most people encode to AVI first before muxing. Good point of interest though Dolemite :)

[stax76]

but most people encode to AVI first before muxing


although encoding directly to a new container format with muxing or encoding audio at the same time works great :)

[darth rosenberg]

desired -= frames * 24; pure open-dml avi would be 16 instead of 24.