OGMuxer v OggMux output discrepancies

[Kompressor]

Hi! This would be my first thread since joining the forum :)

My issue is regarding the differences between outputs from OggMux and OGMuxer. Both are great tools and without them, I'll still be stuck in dark age of avi containers :eek:

I'm hoping that Suiryc and Koepi can spare a little time comparing the my log files (and if the differences between the outputs could limit how other OGGtools in the future can interact with the OGG/OGM files presently made).

I had a movie (Assault on Precinct 13, cool movie :D) in an AVI container format muxed with an AC3 track. Video component was encoded with GKnot, with the audio muxing done separately in Nandub afterwards. The movie plays fine w/o any problems in its avi form.

Part of my movement of migrating towards OGG/OGM was for the convenience of instant seek times, and I decided to have my movie converted to an OGM container. At the moment, I can see this achievable in at least 3 ways:

1) Using OggMux and convert the entire avi movie file to an OGM.
2) By first separating the AC3 track from the movie using Nandub, and then mux the two separate video/audio components into OGM (using OggMux). With this method, I will have to leave my extracted AC3 with the wave header I assuming, which I did.
3) Same as step 2, except I'll remove the wave header this time from the extracted AC3 file, and instead I used OGMuxer to mux them together (which can handle AC3 files without wave headers).

Ideally, each OGM container created by the above methods should play identically, and even more ideally have the exact same structural integrity and content. But in reality, I believe it is not an easy task to achieve the 2nd part (having identical internals), due to the different forms of the A/V sources that we use in muxing for the output.

Due to my (pain-in-the-butt :D ) extremist/perfectionist approach, I wanted to know which method gives the best playable file in terms of generally compatibility with future OGG applications developed (knowing that OGG/OGM is a very losely defined container class).

Using each of the above methods, I've generated 3 OGM container files. Besides each file appearing to play identically with each other, I've also noticed a number of differences between the 3 files.

On the surface they all had different file size (+-500Kb)

Using OGMInfo, I generated log files from each of the three OGMs. While the majority of the information given were identical, there were some figures that were strikingly different.

I'll attach the 3 log files for comparison in the following post. Below were some of the figures that were different:

Video Stream:
Stream Header - Buffer Size
Packet Bytes

Audio Stream:
Stream Header - Size
Stream Header - Buffer Size
Stream Header - Bits per sample
Duration (!!)
Estimated Bitrate

(part 1)

[Kompressor]

(Method 1 - AVI/AC3 already muxed in nandub, throwing the whole avi container as an input into OggMux):

2 Streams : 0 Vorbis, 1 Audio, 1 Video, 0 Text, 0 Old, 0 Unknown, 0 Invalid

=== STREAM 0 (VIDEO 1) ===
Stream Header :
Type : video
SubType : DX50
Size : 56
Time Unit : 400000 (in reference time)
Samples per Unit : 1
Sample Rate : 25.000
Default Len : 1 (in media unit)
Buffer Size : 71101
Bits per Sample : 24
Width : 704
Height : 384
End Of Stream reached : yes
Number of Pages : 164782 (0 skipped)
Number of Packets : 130829 announced / 130829 found (0 skipped), 130827 containing data (not header nor comments)
Number of Samples : 130827 announced / 130827 found
Samples per Packet : 1.0
Packets per Page : 0.8
Bytes : 7317513 (headers) + 707019416 (bodies) = 714336929 (681.24MB)
Percentages : 1.024% for headers + 98.976% for data
Packet bytes : 707019416
Bytes per Page : 44 (header) + 4290 (body) = 4335
Bytes per Packet : 5404
Duration : 01:27:13.080
Estimated BitRate : 1080.8 kbps
Frames per Second : 25.000 fps

=== STREAM 1 (AUDIO 1) ===
Stream Header :
Type : audio
SubType : 2000
Size : 74
Time Unit : 10000000 (in reference time)
Samples per Unit : 48000
Sample Rate : 48000.000
Default Len : 1 (in media unit)
Buffer Size : 1920
Bits per Sample : 0
Number of Channels : 2
BlockAlign : 1
Average Bytes per Second : 24000 (192.0 kbps)
End Of Stream reached : yes
Number of Pages : 30529 (0 skipped)
Number of Packets : 65416 announced / 65416 found (0 skipped), 65414 containing data (not header nor comments)
Number of Samples : 251185920 announced / 251187840 found
Samples per Packet : 3840.0
Packets per Page : 2.1
Bytes : 1347593 (headers) + 125790282 (bodies) = 127137875 (121.25MB)
Percentages : 1.060% for headers + 98.940% for data
Packet bytes : 125790282
Bytes per Page : 44 (header) + 4120 (body) = 4164
Bytes per Packet : 1922
Duration : 01:27:13.039
Estimated BitRate : 192.3 kbps
Codec : 2000 (AC3)


Total bytes : 8665106 (page headers) + 832809698 (page bodies) = 841474804 (802.49MB)
Percentages : 1.030% for headers + 98.970% for data

[Kompressor]

(Method 2 - AC3 track separated from Nandub with wave header intact, together with the soundless video (audio stripped with Nandub) -> OGM using OggMux)

2 Streams : 0 Vorbis, 1 Audio, 1 Video, 0 Text, 0 Old, 0 Unknown, 0 Invalid

=== STREAM 0 (VIDEO 1) ===
Stream Header :
Type : video
SubType : DX50
Size : 56
Time Unit : 400000 (in reference time)
Samples per Unit : 1
Sample Rate : 25.000
Default Len : 1 (in media unit)
Buffer Size : 71101
Bits per Sample : 24
Width : 704
Height : 384
End Of Stream reached : yes
Number of Pages : 164782 (0 skipped)
Number of Packets : 130829 announced / 130829 found (0 skipped), 130827 containing data (not header nor comments)
Number of Samples : 130827 announced / 130827 found
Samples per Packet : 1.0
Packets per Page : 0.8
Bytes : 7317513 (headers) + 707019416 (bodies) = 714336929 (681.24MB)
Percentages : 1.024% for headers + 98.976% for data
Packet bytes : 707019416
Bytes per Page : 44 (header) + 4290 (body) = 4335
Bytes per Packet : 5404
Duration : 01:27:13.080
Estimated BitRate : 1080.8 kbps
Frames per Second : 25.000 fps

=== STREAM 1 (AUDIO 1) ===
Stream Header :
Type : audio
SubType : 2000
Size : 74
Time Unit : 10000000 (in reference time)
Samples per Unit : 48000
Sample Rate : 48000.000
Default Len : 1 (in media unit)
Buffer Size : 6000
Bits per Sample : 0
Number of Channels : 2
BlockAlign : 1
Average Bytes per Second : 24000 (192.0 kbps)
End Of Stream reached : yes
Number of Pages : 29554 (0 skipped)
Number of Packets : 20935 announced / 20935 found (0 skipped), 20933 containing data (not header nor comments)
Number of Samples : 251184000 announced / 251187840 found
Samples per Packet : 11999.6
Packets per Page : 0.7
Bytes : 1300336 (headers) + 125656859 (bodies) = 126957195 (121.08MB)
Percentages : 1.024% for headers + 98.976% for data
Packet bytes : 125656859
Bytes per Page : 43 (header) + 4251 (body) = 4295
Bytes per Packet : 6002
Duration : 01:27:12.999
Estimated BitRate : 192.1 kbps
Codec : 2000 (AC3)


Total bytes : 8617849 (page headers) + 832676275 (page bodies) = 841294124 (802.32MB)
Percentages : 1.024% for headers + 98.976% for data

[Kompressor]

(Method 3 - AC3 extracted with Nandub, wave header removed with BeSlice/BeSplit, together with soundless video -> OGM (using OGMuxer)

2 Streams : 0 Vorbis, 1 Audio, 1 Video, 0 Text, 0 Old, 0 Unknown, 0 Invalid

=== STREAM 0 (VIDEO 1) ===
Stream Header :
Type : video
SubType : DX50
Size : 56
Time Unit : 400000 (in reference time)
Samples per Unit : 1
Sample Rate : 25.000
Default Len : 1 (in media unit)
Buffer Size : 71110
Bits per Sample : 24
Width : 704
Height : 384
End Of Stream reached : yes
Number of Pages : 164782 (0 skipped)
Number of Packets : 130829 announced / 130829 found (0 skipped), 130827 containing data (not header nor comments)
Number of Samples : 130827 announced / 130827 found
Samples per Packet : 1.0
Packets per Page : 0.8
Bytes : 7317513 (headers) + 707019425 (bodies) = 714336938 (681.24MB)
Percentages : 1.024% for headers + 98.976% for data
Packet bytes : 707019425
Bytes per Page : 44 (header) + 4290 (body) = 4335
Bytes per Packet : 5404
Duration : 01:27:13.080
Estimated BitRate : 1080.8 kbps
Frames per Second : 25.000 fps

=== STREAM 1 (AUDIO 1) ===
Stream Header :
Type : audio
SubType : 2000
Size : 56
Time Unit : 10000000 (in reference time)
Samples per Unit : 48000
Sample Rate : 48000.000
Default Len : 1 (in media unit)
Buffer Size : 3849
Bits per Sample : 2
Number of Channels : 2
BlockAlign : 1
Average Bytes per Second : 24000 (192.0 kbps)
End Of Stream reached : yes
Number of Pages : 30665 (0 skipped)
Number of Packets : 163535 announced / 163535 found (0 skipped), 163533 containing data (not header nor comments)
Number of Samples : 251186688 announced / 251186688 found
Samples per Packet : 1536.0
Packets per Page : 5.3
Bytes : 1482089 (headers) + 126084065 (bodies) = 127566154 (121.66MB)
Percentages : 1.162% for headers + 98.838% for data
Packet bytes : 126084065
Bytes per Page : 48 (header) + 4111 (body) = 4159
Bytes per Packet : 770
Duration : 01:27:13.056
Estimated BitRate : 192.8 kbps
Codec : 2000 (AC3)


Total bytes : 8799602 (page headers) + 833103490 (page bodies) = 841903092 (802.90MB)
Percentages : 1.045% for headers + 98.955% for data

[Kompressor]

Sorry for the very long post! Whoever reading this far, thanks for your interest ;)

Now hoping the gurus can shed some light :cool:

Kompressor

[Suiryc]

Hi

I can give you some answers here.

Before starting you should know that OGMuxer could also have used the AVI+AC3 file made with Nandub, or the AVI(video only) + Waved AC3 file, but that is not the point here ;)

First of all OggMux use OggDS (DirectShow filters) made by Tobias Waldvogel. So most of the information OggMux/OggDS use while muxing comes from DirectShow itself.
OGMuxer do not use any DirectShow filter and parse each of the inputs to obtain the necessary information to mux.
This mean that some of the information gathered by each tools may differ (especially when it comes to the number of samples that the input contains).

So let's start :
Video Stream:
Stream Header - Buffer Size
The buffer size indicate the size of the buffer that should be used while playing the stream. It generally equals the maximum size of the data that can contain the stream (in the case of OGM, it is the maximum size of data in a Packet).
The fact here is that on the contrary to Tobias I don't know exactly how works OggDS while rendering the stream. As I noticed recently that a buffer size to small would result in data being purely ignored (in my case all the audio data were in the OGM file, but with a buffer too small the stream was not properly rendered), I decided to "maximize" this value.
In other words I considered the maximum size of the Packets in the stream, which is the size of the data contained in the Packet + the size of the header of the Packet (the header for data consists of 1 byte for the type - data - + a maximum of 8 bytes coding the number of samples that represents those data).
So for me "Buffer Size = Maximum Size Of Data in Packet + Maximum Size Of Header in Packet = Maximum Size Of Data in Packet + 9"
Hence your result here :
With OggMux -> Buffer Size : 71101
With OGMuxer -> Buffer Size : 71110=71101+9
Of course I could have made more tests and remove the +9 ... but I was too busy in adding features ;)

Packet Bytes
Sorry but here I don't understand why there is a difference :confused:
I can only say that as I stated above OggDS get its information from DirectShow, and OGMuxer from the VirtualDub routines ...


Audio Stream:
Stream Header - Size
The size field determines the size of the header.
As OggDS.h (containing stream header definition) only states a stream header of 56 bytes, I don't know what other structure uses Tobias in this case.
And as presently with the "standard" structure everything seems to work, maybe this is not a problem.

Stream Header - Buffer Size
First there is the same explanation as above, plus some other things.
We have 3 different cases here :
1. An AC3 stream muxed in an AVI container by Nandub
2. A waved AC3 file
3. A "normal" AC3
And each case made the tools behave differently.
DirectShow will rely on the structure of the files it renders.

For case 1 it seems that Nandub artificially grouped audio data by groups of 3840 samples. And that's the way DirectShow will read them.
As for a 192kbps AC3 stream 1 byte = 2 samples, then the maximum size of data in Packet will be of 3840 samples * 0.5 byte / sample = 1920 bytes, hence the
Buffer Size : 1920

For case 2 it seems that DirectShow will read data by group of 12000 samples (remember the AC3 stream is not anymore in an AVI container, and as the stream is alone DirectShow don't have problems for rendering such things). So the maximum size of data in Packet will be of 12000 samples * 0.5 byte / sample = 6000 bytes, hence the
Buffer Size : 6000

For case 3 OGMuxer get the "exact" structure (i.e. each frame) of the AC3 file. If I remember well for a 192kbps stream each frame is 768 bytes long.
Anyway each frame of an AC3 stream exactly contains 1536 samples. As I wanted it like that, OGMuxer will only put one frame per Packet (i.e. 1536 samples). So you would have said 1536 samples => 768 bytes. So why not a buffer size of 768 (+9)?
Simple : OGMuxer don't parse the entire AC3 file before muxing. It only verify the first frame is correct, and get the stream information from this frame.
However you can have a VBR AC3 stream (since each frame states its bitrate, length, ...). So for the buffer size I take the maximum size of an AC3 frame, which is 3840 bytes, hence the
Buffer Size : 3849=3840+9

Stream Header - Bits per sample
The bits per sample field should indicate the number of bits necessary to code one sample (for one channel). But this field don't seem to be used afterwards (in WAVEFORMATEX structure it is said this field is generally ignored).
That may be the reason why it is set to 0 with OggDS.
In OGMuxer I apply a formula that stands for (normally) PCM audio :
ByteRate = SampleRate*NumChannels*BitsPerSample/8
=>BitsPerSample=BitRate/(SampleRate*NumChannels)
In our case BitsPerSample=192000/(48000*2)=2 hence the
Bits per Sample : 2
Well as I said this formula apply to PCM stream, and according to the doc, all the values should be computed according to manufacturers specifications in other cases (thanks Microsoft to help this much ;)).

If anyone know what I should use for those values (especially nBitsPerSample and nBlockAlign) please tell me :)


Duration (!!)
Duration is computed according to the number of samples that contains the stream, and the sample rate.
As I stated before there may be differences between information given by DirectShow and what I get in OGMuxer hence the
Number of Samples : 251185920 announced / 251187840 found in case 1
Number of Samples : 251184000 announced / 251187840 found in case 2
Number of Samples : 251186688 announced / 251186688 found in case 3
As you can see the number of samples found in case 1 and 2 are the same (normal since the AC3 streams are the same).
But it is not the same value in case 3. I may have an explanation here : I noticed that Nandub sometimes "cut" the first AC3 frame while muxing it in an AVI container. In this case this "first" "invalid" AC3 frame cannot be used by OGMuxer (as there are no more headers for this frame, I cannot know what it contains). So OGMuxer will jump to the next one (hence the "Skipped ... bytes to synch on first frame of AC3 stream in [...]" OGMuxer message).
This frame is not a problem for DirectShow since it uses the WAV headers to process the stream. That could explain why there is a difference of 1152 samples (exactly the same number than in my test BTW :/) between case1&2 and case 3.

Estimated Bitrate
The estimated bitrate is computed according to the size (in bytes) of the stream, and the duration.
The size of the stream is presently equals to the sum of all Packets in the stream. Unfortunately I also count the Packet header (1 byte for the type + n bytes coding the number of samples).
As OGMuxer produce smaller Packets than OggMux in thoses cases, it also produce more Packets (look at the number of Packets in the streams).
So, let's do some maths here :

In case 1 we have 65414 packets containing data. Each Packet should contain 3840 samples (except the last one). We need 2 bytes to code 3840 => Packet header = 1 + 2 = 3 bytes.
We have 251187840 samples => 251187840/2 = 125593920 bytes
So total bytes in data packets = 65414*3 + 125593920 = 125790162 bytes
Then there is the first Packet that contains the Header (74 bytes), and the Packet that contains your comments. (and then we have "Packet bytes" value).
So the estimated bitrate is 8 bits per byte * Packet bytes * 48000 samples per second / number of samples = 192.2kbps
Anyway the real bitrate should be 8 bits per byte * 125593920 bytes for data * 48000 samples per second / 251187840 samples = 192kbps

In case 2 we have less Packets, so less headers, so a smaller Packet bytes value, and so a smaller estimated bitrate (192.1kbps)

In case 3 we have more Packets, so more headers, so a bigger Packet bytes value, and so a bigger estimated bitrate (192.8kbps)


Hope this helps :)

[Kompressor]

Wow ;) That was awesome Suiryc!

Sure it took me some good digesting to get it down. But it did make me a lot clearer and confident on what I'm doing (rather than confusing me even more :D )

Regarding case 3, I've noticed the audio and video streams were a little bit out of synch. I was initially skeptical about my raw judgement, and I compared the OGMs created in these 3 cases to hopefully find out which is the most 'correct' method in doing the mux. My logical guess would be the one created in method 3, but the out of synch 'suspicion' was prompting me otherwise...

The light that you shed for the duration bit suddenly made it all sensible. I'll now know not to strip the waveheader in such a haste before deciding how I want to mux my movie. I've saved your detailed explanations in a text file when I want to compare outputs in the future.

Thanks to you and Koepi for creating these very friendly tools for us. I'm working my way up the steeeep slope now, and hopefully join you guys someday in writing such essential tools.

Kompressor

[Kompressor]

Greetings ;)

I've encoded and muxed another film today - Back To The Future (I). The video was rather grainy, but perhaps thats how the director wanted them(??).

Anyway, I got kind of addicted with OGMinfo lately.. so I ran another 2 logs comparing the OGMs that I got from OggMux and OGMuxer.

What is the significance of the value "End of Stream reached: (yes/no)" within the VORBIS streams as seen in the log files? In this case I have within my OGM container 1 x video stream (avi), + 2 x audio stream (ogg).

In OggMux's case, both Vorbis audio streams have "no" as the value.
In OGMuxer's case, both Vorbis audio streams were left with "yes" as the value.

I've also noticed the duration part for both audio streams were different between the 2 OGMs. The audio streams of OGMuxer's were each 1.12x seconds longer than those of OggMux's. Does this mean one of the OGMs will be very slightly out of synch compared to the other? Or do these raw values have little significance and perhaps some kind of mechanism during playback would takeover and correct both OGMs to the same state of synch? If someone can point me to a tool which can read and show the duration of an AC3 file, I'll be able to compare the oggs with the original AC3 track (still on hdd).

Since Suiryc last post I became aware that both muxers had different mechanisms into what 'seemed' like a simple process (definitely not after reading some of Suiryc's breakdown of the log file :D ). I'll keep the unmuxed files as well as the vobs if I should have to go back to the original files.

Kompressor

[Suiryc]

As you may know (see Xiph site (http://www.xiph.org) for more information), an Ogg stream is made of Pages (which contains the well known Packets ;) ) which size cannot exceed 65kB.

The header of those Pages indicate some information. Among those are the Page number, the Page version, the granular position (absolute position) of the Page in the stream, and some flags like :
- is this Page a BOS (Beginning Of Stream, i.e. first Page of the stream, that should then contain the header Packet giving usefull information on the stream)
- is this Page an EOS (End Of Stream, i.e. - normally - last Page of the stream)
- is this a "continued" Page (i.e. does the previous Page couldn't contain an entire Packet, and so this Packet have been spanned over multiple Pages)

So normally an Ogg stream contains a BOS (first one) and an EOS (last one) Page.
A stream that does not contain a BOS Page cannot be rendered (since there are no headers and so we don't know what contains the stream).
A stream that does not contain an EOS is still good to be rendered. The fact there is no EOS would mean in this case that the stream has been cut (and so that the last Pages have been dropped).

Now as you surely understood the "End Of Stream reached" value in OGMInfo tell you if an EOS Page was found for the stream.

In the case of OGMuxer there should allways be an EOS Page (at the end of the process I generate such a thing if not yet done), even if the stream have been cut.

This could explain the difference between duration you obtained in the resulting OGM files.
If OggMux (i.e. OggDS&DirectShow) somehow cut the Vorbis stream (because it thought it longer than the video maybe), then the last Pages of the stream (including the EOS Page) have been dropped. This also apply to the tools you used to generate the ogg file.

In this case the difference between duration is normal and does not mean any out of synch problem.

Anyway with a 1.12x seconds difference I think you should hear if there is an out of synch problem in the OGM files ;)