STD Buffer Overflow

[Sir Didymus]

My hope is to be able (maybe in the next weeks...) to evidence a direct relationship between STD Buffer Overflow events (that are actually produced BY SURE by DVD-RB) and the reported dropouts. For the moment lets's consider the observed overflows as a stand alone topic of discussion.

The way to see the overflows is to use the attached stdbuflog.exe program. The source code is also included.

It works logging the real time simulation of the STD Buffer filling and voiding. Filling events are basicaly produced by the addition of the packet payloads to the buffer. Voiding events are produced when the system clock reach the DTS time of each MPEG2 picture.

The program usage is as follow:

// ---------------------------------------------------------------
// Usage: StdBufLog [-VIDxxxxx] [-CIDyyy] [[-v1]...[-v4]] filename
// ---------------------------------------------------------------

program arguments should be ordered and are defined as follows:
-VID --> VOB ID [1...65535] of the cell(s) to log. Default -1 logs every VOB ID
-CID --> CELL ID [1...255] of the cell(s) to log. Default -1 logs every CELL ID
Verbose levels:
-v0 [default] print out only time of STD Buffer Changes and Buffer Size.
This mode is the most synthetic and useful for importing the output
in other application for the graphical representation of the simulation
i.e. Microsoft Excel or GnuPlot.
-v1 print out also packet payload sizes (when filling the Buffer) and
frame extraction events (when DTS time elapses), together with reporting
overflows, underflows and the dimensions of these anomalous events.
-v2 print in addition all simulation timings (also when no changes happen
to the STD Buffer size. This is basically a compatibility mode to report
the same file organisation of the Philips DVD-Video Verifier.
-v3 prints also other useful information, like events of MPEG2 Sequence,
GOP, and Picture Start in the MPEG2 stream.
-v4 logging for debugging purposes [not suggested, and file size may become huge].
When used at v0, its output could be easily imported in Excel or in other plotting programs.

When used at v1, its output is especially useful for understanding why STD Buffer Overflow events (that are systematically produced by DVD-RB: original titles are normally not affected) are generated.

Cheers,
SD

EDIT of 4/11/2004:
- corrected many mistakes, mostly in the proper handling of small packets
- added NTSC support (still not properly working)
- added multiple vob files support (it starts with the selected filename and go on up to the last in the same vts)
- fully tested on many PAL cells and on a couple of PAL titles

EDIT of 6/11/2004:
- added support to NTSC and PAL (just frame codes of 3 and 4), that is more than enough for testing pourposes... The mode is automatically selected by the VOB contents.
- added -VID and -CID arguments. Very useful for testing a single cell without demuxing the titleset...
- I consider this one as the "last" release, meaning that just debugging and further testing could lead to change it...
- I say again: for testing pourposes it is more than sufficient...

[robot1]

Thanks for your researches.
I've just downloaded your apps and I will do some tests.

[Trahald]

@Sir Didymus
Nice going!!

[berrani]

Ciao, Sir Didymus, thanks very much for your research. :cool:
We all (with audio drops problem) want to use as soon as possible this great program! :D

Tony

[DVD Maniac]

I tried the app in command line just now.

I seem to be getting lots of output lines for our old problem child movie but when it finishes I only seem to have a couple of screens worth of data and cannot scroll up to find all the output.

Sorry - I am not a command line specialist

Need some help to get all this data into Excel / whatever!

[Sir Didymus]

Dear all,
thanks for the compliments

Sorry to inform I just discovered in the model I wrote a couple of bugs (all due to my fault: I did very few testing with the application...).

First bug is SERIOUS, and it is related to the implicit support of PAL only vob files (in the code the frame time is fixed at 40 ms.).

The second is related to the improper handling of the transition among cells.

Sorry for the mistakes...
I will post an update as soon as possible...

For the moment please use it just on single cell PAL vobs...

@DVD Maniac

For import into Excel, the program should be used in its simplest form from a dos window:

Stdbuflog filename > ppp

Where ppp is the log file generated from the program.
Then open Excel

File --> Open [filetype "all files"] --> ppp

Then you select "A" and "B" columns, then click on the Graph Generation tool, selecting DISPERSION (XY) type, choosing a graph type without data indications...

Cheers,
SD

[psdos]

what kind solution has it? if there is, thanks.

[Sir Didymus]

Just in order to state it plain words:

At this stage the program I posted is nothing else than a testing tool, capable of performing a simulation in order to make evident one problem (i.e. overflows of the STD buffer) that is sometime present in the DVD-Rebuilt titles.

So, the application is not aimed at curing anything.
Even the relevance of this problem and its connection with dropouts problem are far to be demonstrated. This is the reason why I would prefere, for the moment, not asking Jdobbs to do any specif actions or changes in the code of DVD-RB. In the next weeks, if these relationship will be demonstrated it will be discussed.

Of course any comments/suggestions is well accepted (and in some way necessary!!!) from everybody, including the boss... ;)

I say again: the usage of the program is only suggested for the generation of a data file that should be imported in Excel in order to "see" STD buffer overflows events.

I posted the program just for showing that it is possible to write some code (that is complex, but not impossible to manage) including the real time simulation of the buffer status.

In the future I am planning to write another (different) application for changing some timings (the SCR values) of a single cell in a VOB file in order to avoid these overflows. Only at that time it will be possible to carry out some tests to check for the visible benefits/drawbacks of this approach.

For the moment much better (at least this is my suggestion) taking the whole matter with some calm: I would say that the risk of all of this stuff will demonstrate to be silly is not so low...

[psdos]

could to change jdobbs something to limit the value of SCR?

[Trahald]

Originally posted by psdos
could to change jdobbs something to limit the value of SCR?

psdos - please, for now just let sir didymus do the experiment, dont try to make guesses about things

[psdos]

I only want to help with any probe, anything all. I do a probe with a change in ECL of rebuilder how say robot1,

After "prepare", Change in Rebuilder.ecl all lines
fix_gop_length=0
with
fix_gop_length=1

Save and run encode phase.

but it is same.

[Sir Didymus]

I just uploaded a new release of the StdBufLog program. I'd like to add some little more information to explain "where" all this work is expected to arrive...

1. The actual release of the program is matching very well with the timing information (where it'l take much more time to collect) of different DVD profilers (especially the Philips DVD-Video Verifier). It is fundamental to be totally aligned with these tools in order to establish and guarantee the reliability of the program and of the whole approach.

2. Using the same program scheme, it will be possible (in the next weeks) to build other programs to check that the output of DVD-RB is compliant with many timings constraints. For example it will be very easy to see if the DTS of all of the encoded pictures are precisely matching the DTS of the original.

3. Using the same program scheme, it will be possible (also in the next weeks) to build another program to modify the output of DVD-RB (especially the SCR values) in order to be compliant with the video buffering verifier constraints.

4. After these modifications (that hopefully will leave the title fully playable...) it should be possible to verify in the practice if the compliance with the vbv rules will produce benefits or drawbacks...

The nice thing is that even in case of totally bad results, the posted tools could still be kept as nice instruments for the verification of DVD material...

All the best,
SD

[jdobbs]

Great little program. I've been using it to tweak my STD buffer calculations. The next version (0.67) of DVD-RB should show complete compliance with specs.

Now for the warning. I beseech you... please don't release code to modify the DTS and/or SCR values -- if you do I PROMISE I WILL ABORT THIS PROJECT and no longer continue to support DVD-RB, as it will be unsupportable.

You have no idea what you are getting into. Have you considered the temporal position of the individual frames, for example?

DTS will NEVER match the original and neither will SCR --- it's not supposed to, it has been reencoded and reauthored.

[robot1]

Originally posted by jdobbs
Great little program. I've been using it to tweak my STD buffer calculations. The next version (0.67) of DVD-RB should show complete compliance with specs.
Since you wrote the code, I agree with you that this tool should let you find the possible problems, not to tweak at the risk to screw things...
I hope the next version will be at Scenarist level in muxing. ;)

[berrani]

Originally posted by jdobbs
Great little program. I've been using it to tweak my STD buffer calculations. The next version (0.67) of DVD-RB should show complete compliance with specs...
[/B]

Great news, jdobbs :D , I'm very happy to read it!

I wait for your fantastic work :cool:.

[jdobbs]

It remains to be seen whether it actually has any impact -- it doesn't on mine. But, of course, I've never experienced audio dropouts.

[Sir Didymus]

Originally posted by jdobbs
Great little program. I've been using it to tweak my STD buffer calculations. The next version (0.67) of DVD-RB should show complete compliance with specs.


Thanks for the compliment, boss. 0.67 news is cool. If you accept this, and if you think it could be useful, as soon as I have some findings on the matter will report some...


Now for the warning. I beseech you... please don't release code to modify the DTS and/or SCR values -- if you do I PROMISE I WILL ABORT THIS PROJECT and no longer continue to support DVD-RB, as it will be unsupportable.


Well, first of all I want you to know I PROMISE I WILL STRICTLY OBEY TO YOUR WARNINGS: I am stopping immediately all the work based on changing anything of the RB output. That's all. The possibility you are mentioning of stopping your work on dvd-rb because of my testings is simply terrific and totally unwanted for me. I simply do not understand how the programs I was planning (and I say again I am immediately stopping) to develop could represent a trouble for you: I wrote them exactlely with the opposite pourpose (that is to be supportive). It's clear it should be seemed offensive respect to your work. Sorry for this.


You have no idea what you are getting into. Have you considered the temporal position of the individual frames, for example?


You are totally right...
I mean, I am learning, so I realise I don't know everything on the matter. About the temporal position of the single frames, I am sure I considered it (I wrote StdBufLog starting from scratch, and you may see it is explicitely calculated; the source code was included specifically for verifications...).


DTS will NEVER match the original and neither will SCR --- it's not supposed to, it has been reencoded and reauthored. [/B]

As I told before I am learning, so it is very clear to me why SCR values should not match. It is less obvious the reason why DTS and PTS shouldn't too match: especially for PTS, even thought the frames have been reencoded and reauthored, their presentation positions and times should correspond, right ? But I have clear indications that these correspondance in DVD-RB are not guaranteed in all situations (for example if you cut the last let'say 50 frames from one of the rebuilt cells, this BIG mismatch between original and rebuilt is not reported at all as an error...). About DTS I can understand, that due to the reencoding and reauthoring the relative decoding times can be different...

Best regards,
SD

[jdobbs]

SD,

I'm not trying to be a jackass (although that doesn't take much "trying"). I'm trying to save myself an inordinate amount of work.

If you could take a stream and sort it in temporal order -- yes, each frame would have the same PTS in "real world" terms -- but you will find that often the SCR is reset, changing the basis for the timestamps.

More importantly, in MPEG the order of the frames in the stream doesn't match the order in which they are presented. For example you will often find that the first I-FRAME in a GOP is actually the third frame to be presented (because of backward referencing) -- so the PTS in the header is adjusted for that fact. Since the position of the I-FRAME and its temporal order is determined by the encoder (not the authoring software), it could be just about anything within a range -- and rarely would it match that of the original stream.

If you somehow sorted all the frames output by CCE and organized by DVD-RB based upon their temporal position -- and made adjustments for the values of the SCR you'd find that every single frame in fact does play at the exact same time in its adjusted temporal domain.

DTS is even more complicated. The I-Frame I mentioned above that is presented as the third temporal frame, has to be decoded before the first one -- so it can be referenced in the reverse direction.

This is only one scenario, and there are hundreds of other considerations as well, such as the implication of changing the values on the tables in the NAVPACKS that are used for forward and reverse movement, not to mention ILVU.

The problem with writing code that changes all this is that most people (bless their hearts) think they know more about this stuff than they really do. And they will create incredible problems, be convinced it is DVD-RB that is causing it (because they couldn't possibly be at fault), and start posting wild hypotheses about how DVD-RB might fix it. If you think that's not true, just go through the DOOM9 pages and count the number of times I've asked "have you preprocessed?" only to hear:

"No. Well except for removing a bunch of stuff and modifying the IFOs, VOBs, and running IFOEdit against it."

Imagine the difficulties I'd run into if I couldn't even trust the basic clocks and timing any more?

I wasn't being alarmist when I said I'd stop the project. I'd have to just to keep my sanity.

[Sir Didymus]

It remains to be seen whether it actually has any impact -- it doesn't on mine. But, of course, I've never experienced audio dropouts.
That's the real point. As I wrote at the very beginning of this thread, to be able to evidence some relationship between STD Buffer Overflow events and dropouts is just an hope. Also in my experience the reported overflows have no impact. The playback of my backups is perfect, and I am no experiencing dropouts.
I'm not trying to be a jackass (although that doesn't take much "trying"). I'm trying to save myself an inordinate amount of work.
That another big point: I am very seriusly worried about the possibility you write an huge amount of code just for arriving at the conclusion that "Yes ok. Now the compliance with spec. nr. xxxxx is obtained, but that has no impact". Or worse, it may cause the generation of new reported troubles.
Since this path has been already followed in the past I supposed it might be a good idea to propose to write some testing code (at my charge) for postprocessing the dvd-rb output. It was just intended to take all of these risks on my side. I understood it is not a good idea. As stated by others, since you wrote the code, it's better you own these responsibilities, avoiding all risks to get things screwed...
More importantly, in MPEG ...
Imagine the difficulties I'd run into if I couldn't even trust the basic clocks and timing any more?
I wasn't being alarmist when I said I'd stop the project. I'd have to just to keep my sanity.
Thanks a lot for your kind explainations. I think all of them are very useful for people wanting to have a deeper understanding of DVD and MPEG2. Since I have promised I will not release anything apart StdBufLog, may I interpret what you are stating as you will not stop your project because of my testings, right ?

Ok. Now let me switch to the bad news, i.e. that 0.67 seems to me not to comply with vbv verifyer specs. 138182 Annex C.

Overflows events exceeding 300.000 bytes are still present in the DVD-RB output.
Overflows events exceeding 250.000 bytes are frequent.

If you think it could be useful (I have now some doubt about my work is accepted as supportive or it is considered as a noisy and disturbing pushing...) I can post in some more descriptive items on the evidence I am observing.

SD

[jdobbs]

Interesting. I used your software to test it and couldn't find anything over about 200K. I'll look at it.

[psdos]

In another test with RBv0.67 and the Christ passion continuan the lost ones of audio, concretely in time 00:28:10 for example.

[Sir Didymus]

JDOBBS SORRY FOR MY PREVIOUS POST.

I wrote the output of 0.67 is actually totally not compliant with vbv specs. This IS WRONG (or better it was wrong at the time I wrote): it was due to my big faults in looking at the files in a wrong directory...

I say again: I WAS MISTAKEN. In my new tests I see the difference respect to previous versions of DVD-RB, and many of the cells rebuilt with 0.67 are compliant with vbv constraints. In these cells the buffer allocation is constantly below 200.000 bytes, as you stated.

Nevertheless many other files shows that overflow events are still present in the files generated by DVD-RB, with overflows sometimes exceeding 300.000 bytes.

BIG SORRY FOR THE PARTIALLY WRONG INFORMATION I GAVE PREVIOUSLY, AND FOR THE TIME YOU ARE SPENDING on the matter, but for the moment I confirm the non compliance of 0.67.

SD

[jdobbs]

What DVD(s) are you using for testing? I'm really surprised that it could spike that high -- especially considering the video is always smaller than the original.

[robot1]

Originally posted by jdobbs
I'm really surprised that it could spike that high -- especially considering the video is always smaller than the original.
What if "locally" the re-encoded video is larger than the original (CCE allocated more bits for a scene...)?
This could be possible if reduction is low enough (compressing at 85-90% of the original)

[jptheripper]

Thoughts and a Possible solution

If cce is allocating more bits the the original (which it may be as there are few constraints on it) then these spikes could happen, especially if the added compression is low

Maybe all that is needed is a comparison of pre and post bitrates by cell.

Also, if this is the case, the solution might be easy (relatively conceptally with 0 implications to to feasability in coding)

It might only be that in the bitrate allocation calculations a maximum be added per cell based on the existing maximums per cell, instead of a global maximum.

question remaining is how strict is cee with its min/max upon encoding?

Just some random thoughts

[Sir Didymus]

It is Kill Bill vol2, R2, PAL.

If I may add some information, the movie is obviously completely un-preprocessed, encoded using DVD-RB 067, all standard settings, with CCE SP 0.67, stripped two of the three audio tracks (one DTS and one AC3).

Source size of the DVD folder is 7.16 GB. Output folder is 4.33 GB. Playback on PC is perfect. Didn't burn to verify on standalone.

This is the whole plot of the first cell of the movie (V1, C1), that has been obtained with the command:

>StdBufLog -VID1 -CID1 -v1 vts_01_1.vob > cell1.log

And then used Excel to render the data in a graph.

http://img127.exs.cx/img127/8043/Cell1.gif

The time scale unit (X axis) is 100 us, so 200000 means 20 seconds.
The Y axis is the reported buffer occupancy, in bytes.

It follow a little bit of analytical work (in order to help me understanding if you share my findings).

The very first overflow events happen just at the beginning of the cell. The first lines of the file cell1.log are containing the following information:

31 2012 +f 2012
48 4037 +f 2025
64 6062 +f 2025
...

First column is the time of the reported event (31 means 3.1 ms.). At that time the buffer size (second column) is 2012, and this happened because the payload of the first MPEG2 packet in the video stream (third column, +f 2012) was added to the buffer contents, that was void at the beginning.

In correspondance of the first overflow event, the following lines are produced:

...
6276 227429 +f 2025
6292 229454 +f 2025
6308 231479 +f 2025
6325 233504 +f 2025
6341 235529 +f 2025
6357 237554 +f 2025
6373 239579 +f 2025 !!!!! OVERFLOW by 2011 bytes !!!!!
6390 241604 +f 2025 !!!!! OVERFLOW by 4036 bytes !!!!!
6406 243629 +f 2025 !!!!! OVERFLOW by 6061 bytes !!!!!
6422 245654 +f 2025 !!!!! OVERFLOW by 8086 bytes !!!!!
6438 247679 +f 2025 !!!!! OVERFLOW by 10111 bytes !!!!!
6471 249704 +f 2025 !!!!! OVERFLOW by 12136 bytes !!!!!
6472 230840 -B 18864
6487 232865 +f 2025

The time of 6472 (647.2 ms after the cell start) is the DTS time of the individual B frame (whose size is reported as 18864 bytes) that is supposed to arrive in time for voiding the buffer. If the six packets (timings from 637.3ms to 647.1ms) would have SCR timings just a little bit higher (9.8 ms, corresponding to 882 90KHz ticks) this overflow events would have been avoided...

I have some evidence that this strategy (i.e pourposely generating little SCR increase of values), and the explicit inclusion of a limiting model of the buffer behaviour is adopded by some reference authoring package...

This evidence could be easily observed by running StdBufLog on the output of these packages (Some of these tests are reported in the thread "Dropout: Jdobbs don't bother reading...").

P.S. In case you have the feeling StdBufLog could be not precise (or not reliable) in its calculations, this is perfectly understanded. In any case I will repeat tomorrow the same test with the Philips tester, in order to double check my findings.

It will take the whole day to do the test, but this will help myself to verify if I am able to properly calculate the DTS of individual frames in a VOB stream...

All the best,
SD

[Sir Didymus]

Originally posted by robot1
What if "locally" the re-encoded video is larger than the original (CCE allocated more bits for a scene...)?
This could be possible if reduction is low enough (compressing at 85-90% of the original)

It could be a possibility, but it does not happen on this specific situation, that arise on the first GOP of the cell: the original have the following GOP structure & size, in bytes (note it is a 12 frames GOP):

I 9220
P 342
B 342
P 22733
B 7455
B 14943
P 43655
B 25700
B 30402
P 59284
B 40640
B 45480

The rebuilt GOP have the following structure (11 frames)

I 9332
P 456
B 456
P 9008
B 4828
P 17472
B 11060
B 12996
P 25860
B 17516
B 18864

Note (in my previous post), the overflow event timing is on the last frame of the first GOP in the rebuilt cell.

Cheers,
SD

[Sir Didymus]

Originally posted by jptheripper
Thoughts and a Possible solution

If cce is allocating more bits the the original (which it may be as there are few constraints on it) then these spikes could happen, especially if the added compression is low

Maybe all that is needed is a comparison of pre and post bitrates by cell.

Also, if this is the case, the solution might be easy (relatively conceptally with 0 implications to to feasability in coding)

It might only be that in the bitrate allocation calculations a maximum be added per cell based on the existing maximums per cell, instead of a global maximum.

question remaining is how strict is cee with its min/max upon encoding?

Just some random thoughts

Hi, jptheripper :)

Your thoughts are not matching at all with mine!!!

It has been widely understood that the video bitrate is totally unrelated with the STD budder overflows. That's sure, at least if you define it in the commonly intended way. If you mean (like I do) the instantaneous number of bytes per second that is entering in the buffer, then I agree. See my previous post where it is analysed what is happening to generate the overflow.

Cheers,
SD

[jptheripper]

that would just be my ignorancee of the buffer as opposed to the bitrates..

sorry..

back to the more intelligent comments..

[Sir Didymus]

@jptheripper

I just want to add one thing for you, that is you have not at all to say sorry, and your comments are not at all improper or not intelligent. At the contrary they are really appreciated (at least I can say this IMHO).

It's me that I have to ask sorry to you: yesterday I was a little bit in a hurry and I gave to you a very very quick answer, with a very poor explaination, and also with information which is just partially right.

Let me take some little time now to "repair" this fault. The video bitrate is a property (the number of bit/s) of the MPEG2 stream. It is the responsibility of the adopted Encoder to comply (and in which terms, i.e. instantaneously, at the GOP level, or in a given timeframe) with the specified min., max. and average bitrate parameters. The lack of some precise definition on "how" this bitrate is calculated, is frequently the source of some incoherence and disagreement among different tools while plotting or reporting this video bitrate information.

After the MPEG2 streams (in the m2v files) have been generated, before arriving to the DVD structure, the authoring software (among the many other activities) should perform two fundamental operations: the generation of the video stream packets, i.e. crunch of 2048 bytes each, and the interleaving (muxing) of these packets with the other streams (navigation, audio, subpicture) that are present in the VOB structure. In these two operation it is fundamental to control all of the involved timing information in such a way to keep the compliance with the many constraints present in the DVD specifications.

The SCR (System Clock Reference) is a timing information, which is present at the packet level (each packet has its own SCR). This info is basically the "time" of the packet, meaning that it could be assumed that the information carried by a given packet is not available before its SCR value is elapsed.

The DTS and PTS (Decoding Time Stamp and Presentation Time Stamp) are time stamp information attached just to some of the MPEG2 pictures after the packetizing (normally to the I frames heading the GOP structures). Due to the MPEG2 way to handle the pictures (forward and backward references), a reordering of the pictures is necessary, after the decoding process, for the proper (and ordered) presentation of the pictures.

Said this (didn't want to give any stupid lesson, it is just to explain why I wasn't totally right yesterday) in my previous post I wrote:
--------
If you mean (like I do) the instantaneous number of bytes per second that is entering in the buffer, then I agree.
--------

Now (I hope) it should be clear why what I wrote is just partially true: the problem of the overflows is not generated alone by the instantaneous number of bytes per second that is entering in the buffer [this value is fixed by the muxing rate, which is 10.08 Mbit/s] but by the fact that the instantaneous number of bytes per second that is leaving the buffer (that is fixed, since it is guided by the DTS values of the individual frames), is sometimes not sufficient to compensate the payload of the entering packets.

I am sure, looking at the log info posted before, this type of event should result much more easy to understend than by reading my description of it...:confused:

Cheers,
SD

[jptheripper]

wow, a scholar and a gentleman

thanx for taking the time to explain the intracacies (sp?) of the bitrate timings, it makes so much more sense now.

If i can ask if a couple assumptions are true:

1. The encoder does all the frame by frame bitrate assignments, but does so it a way that is unaware of the subpicture, audio, etc.. packets so that it can only work with min/max/avg settings.
2. The authoring package (in this case RB) does all the muxing plus the timings (SCR, DTS, etc..) in a way as to hopefully comply with dvd specs.

IF these are both true, then i only see 2 options.
1. Make sure RB builds to specs. I am sure this is jdobbs' prefered method and he is obviously looking at it
2. As a back of the envelop (sic) approach or jerry rigged method, might we be more convservative in our values to the encoder in an attempt to provide more wiggle room (sorry for the colloquialisms but i hope you get the idea)?

I guess i am just surprised this is an issue at all. At times we are cutting the instantaneous bitrate nearly in half (or more) and I doubt that we rarely reach 90% of the preexisting bitrate on most encodes. How is it then that we are spiking the buffer with more information than the original (to spec) disk did?

I can only surmise that while our peaks are lower they are broader providing for a net larger buffer size for a given number of cells (i.e. lower peak but before and after cells are larger as the encoder smoothes the output).

sorry for the rabbling.. just some analytical thoughts

[Sir Didymus]

Ok I have the double check of the Verifier.

Here is the output of StdBufLog:

31 2012 +f 2012
48 4037 +f 2025
64 6062 +f 2025
...
6276 227429 +f 2025
6292 229454 +f 2025
6308 231479 +f 2025
6325 233504 +f 2025
6341 235529 +f 2025
6357 237554 +f 2025
6373 239579 +f 2025 !!!!! OVERFLOW by 2011 bytes !!!!!
6390 241604 +f 2025 !!!!! OVERFLOW by 4036 bytes !!!!!
6406 243629 +f 2025 !!!!! OVERFLOW by 6061 bytes !!!!!
6422 245654 +f 2025 !!!!! OVERFLOW by 8086 bytes !!!!!
6438 247679 +f 2025 !!!!! OVERFLOW by 10111 bytes !!!!!
6471 249704 +f 2025 !!!!! OVERFLOW by 12136 bytes !!!!!
6472 230840 -B 18864
6487 232865 +f 2025

And here is the same from the profiler:

1.644 0 0 . Mux_rate changed to 10080000 bits/sec !
1.644 0 237568 . Buffer size changed !
1.644 0 237568 .
3.233 2012 237568 +f 2012
3.256 2012 237568 .
4.856 4037 237568 +f 2025
4.878 4037 237568 .
6.478 6062 237568 +f 2025
...
627.633 227429 237568 +f 2025
627.667 227429 237568 .
629.267 229454 237568 +f 2025
629.289 229454 237568 .
630.889 231479 237568 +f 2025
630.911 231479 237568 .
632.511 233504 237568 +f 2025
632.544 233504 237568 .
634.144 235529 237568 +f 2025
634.167 235529 237568 .
635.767 237554 237568 +f 2025
635.789 237554 237568 .
637.389 239579 237568 +f 2025 !!!!! OVERFLOW by 2011 bytes !!!!!
637.411 239579 237568 .
639.011 241604 237568 +f 2025 !!!!! OVERFLOW by 4036 bytes !!!!!
639.044 241604 237568 .
640.644 243629 237568 +f 2025 !!!!! OVERFLOW by 6061 bytes !!!!!
640.667 243629 237568 .
642.267 245654 237568 +f 2025 !!!!! OVERFLOW by 8086 bytes !!!!!
642.289 245654 237568 .
643.889 247679 237568 +f 2025 !!!!! OVERFLOW by 10111 bytes !!!!!
645.544 247679 237568 .
647.144 249704 237568 +f 2025 !!!!! OVERFLOW by 12136 bytes !!!!!
647.167 249704 237568 .
647.267 249829 237568 +p 125 !!!!! OVERFLOW by 12261 bytes !!!!!
647.267 230965 237568 -B 18864
648.767 232865 237568 +R 1900
648.789 232865 237568 .


It is possible to see that the time resolution of the Verifier is 1 us., while the one of StdBufLog is 100us. Apart this, I would say that the matching between the two (at least in this case) is, as it should, very very good...

SD

[Sir Didymus]

@jptheripper:

Oh... Good material to start some brain activity...

Your assumptions
1. Agreed.
2. Agreed.

Your Options
1. Hopeful, but in practice impossible. The MPEG2 and the DVD specifications are so many and so complex that it is impossible to guarantee to comply with everything. The real guess it to understand what it should be not wise to leave. See also some posts of Jdobbs and myself on the matter in this thread:
http://forum.doom9.org/showthread.php?s=&threadid=84740

2. I think I understand your point, but it seems to me you are not fully understanding the mine: it is like placing a block of iron of 1 Kg. on the surface of the sea and trying to avoid it goes down by reducing its weight to the half. The encoder is already doing perfectly its job in producing a stream which is compliant with the specs. It should just be packetised and muxed in such a way to keep the involved timings making an ordered (and limited) usage of the decoding buffer. I tryed to encode a m2v stream at 4200 Kbit/s Ave. 5000 Kbit/s Max. to see if it could be benefical, but it isn't. The problem is generated at a different level than the one of the encoder. See this thread (picture 3) for details on this test:
http://forum.doom9.org/showthread.php?s=&threadid=84221


I guess i am just surprised this is an issue at all. At times we are cutting the instantaneous bitrate nearly in half (or more) and I doubt that we rarely reach 90% of the preexisting bitrate on most encodes. How is it then that we are spiking the buffer with more information than the original (to spec) disk did?


Good question. But easy to answer: again the video bitrate has no impact on the decoder's buffer overflows. There is less information but it is inserted into the buffer too quickly. In addition it is impossible to "adapt" the original timings in this operation: the authoring is new (at least partially), the audio streams and the subpictures may be different, and so on, so the SCR, DTS, PTS should be produced "ex novo".

All the best
SD

[jptheripper]

totally understood..

except for the floating lead.. as although we are cutting half the weight, it is from a block of lead that is already floating !

i understand they are independent.. now im just bein an ass on symantec's.

[TheSeeker]

@Sir Didymus

Hey I was just wondering if you could point me in the direction of a website or a book or something that explains the inner-workings of the mpeg2 standards? also dvd compliance and all that. i would like to know more about std buffers, and scr timings and dts, pts all of it. So just wondering if you knew of a good informational website I could take a look at. Thanks!!

[Sir Didymus]

Hi TheSeeker! :)

I am happy I can say I know THE BEST SITE IN THE WORLD on the subject... But since I can read your post I assume you too have this information... Never heard about Doom9 ? ;)

Well, apart the very obvious suggestion to try to have access to the documents about the MPEG2 and DVD standards, I have really to leave the answer to some of the masters on the subject that are populating this forum. I know by sure Jdobbs and Trahald will read this post. Maybe you don't mind I redirect to them your kind question...

SD

[Trahald]

My knowledge of that is limited.. jdobbs is definately better to answer. This thread http://forum.doom9.org/showthread.php?s=&threadid=81526 contains some links to some specifications as a start.. its hard to find the full dvd book specs.

[TheSeeker]

Sorry to continue this off topic conversation but their really seems to be a lack of reliable technical information out there on mpeg2 standards and dvd structure white papers. Maybe I just dont know where to look but the best thing I could find was a semi-useful white paper on mpeg2 streams put out by pinnacle explaining the nuts a bolts of the compression technology. (I, P, B, frames and such). I already was familiar with this. I want to know more about the dvd video structure I guess you could say. Like all the stuff that the authoring program takes care of, (ie, scr timings, std buffers, PES, DTS, PTS) Maybe I will actually have to spend some money to get good information on the subject. Jdobbs, would you be able to point me in a good direction? How did YOU learn all that you know?

[jdobbs]

Actually DTS and PTS are assigned by the encoder. The authoring package may change them -- but only in a relative manner.

Here is my favorite reference site:

http://www.mpucoder.com

In comparison to mpucoder I am a complete simpleton. Actually in comparison to a lot of the folks here at DOOM9 I'm a complete simpleton. Hmmm... now that I think about it I am a complete simpleton.

Okay back on track. Another really good references is:

http://www.dvddemystified.com/dvdfaq.html

MPEG standards are spelled out in the ISO 13818 series -- search the internet and you'll find lots of stuff, the official documents cost $$$$, though.

There's also a lot of good information in the book "Video Demystified - A Handbook for the Digital Engineer" by Keith Jack -- including MPEG-1 & MPEG-2.

[TheSeeker]

Yea I saw that book Video Demystified.. I might just have to go pick it up. I also noticed that the official ISO documents cost money, which is too bad... Those sites seem to have a great wealth of information. Thanks much for the references.

[Trahald]

I decided to post this here since it isnt worthy of its own thread. It is a 'compliancy' thing that despite that may not have an impact on playback...

A rule with ILVU is that the interleaved sections are not to cross layer boundaries. (if you ever look at movies with ILVU they tend to have 1 or 2 odd sized vts files in the movie vts set). again.. this may not have an impact on playback. just thought id mention it. (not as important on dvd5 projects)

[Sir Didymus]

Hi. :)

Sorry for keeping back alive this old thread...

Since I am continuing to test the new rebuilder releases against the compliancy with the STD buffer limits, the recent progresses of DVD-RB are so relevant and nice to see that I decided to share the good news, that is consisting in the continuous and constant improvement of the buffer allocation profile. Please consider that in the most cells of the same movie the spikes are already well below the buffer size limit...

http://img157.exs.cx/img157/9936/stdprofile0640sw.th.gif (http://img157.exs.cx/my.php?loc=img157&image=stdprofile0640sw.gif)

http://img157.exs.cx/img157/6766/stdprofile0672tf.th.gif (http://img157.exs.cx/my.php?loc=img157&image=stdprofile0672tf.gif)

http://img154.exs.cx/img154/3065/stdprofile0681eq.th.gif (http://img154.exs.cx/my.php?loc=img154&image=stdprofile0681eq.gif)

http://img157.exs.cx/img157/1127/stdprofile0697jm.th.gif (http://img157.exs.cx/my.php?loc=img157&image=stdprofile0697jm.gif)

Cheers,
SD

[TheSeeker]

So when the spikes go above like 300 or so we might see an audio dropout? Or what is the dvd compliant value for the std buffer?

[Sir Didymus]

Hi TheSeeker.

Compliancy is between 0 and 237568.

My point of view (maybe it is surprising, but I am a very practical person) is that the compliancy in itself means nothing.

I mean, if the cost (in term of further time and fatigue) for obtaining the full compliancy is limited, then maybe Jdobbs can consider it as a good investment, since it seems to me it is difficult not to see the correlation between the SCR improvements and the reported reduction of dropouts.

On the other hand, maybe the path for reaching the full compliancy with the specifications is too long and complex (I simply don't know...). In this case better to stop the improvements when the reported dropouts becomes almost absent (or very rare, and it seems to me this condition is already pretty much obtained). No need, IMHO, to spend time and work just for formal improvements...

All the best,
SD