Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.


Go Back   Doom9's Forum > Video Encoding > MPEG-4 ASP

Closed Thread
Thread Tools Search this Thread Display Modes
Old 5th May 2002, 16:17   #1  |  Link
Acaila's Avatar
Join Date: Jan 2002
Location: Netherlands
Posts: 1,529
Undersized Files. Why do I get them, and what do I do about it?

Since a lot of people have this problem with DivX5 I decided to write an explanation of why it happens and what to do if it happens to you.

Before the days of DivX5 it was quite common to get either undersized or oversized files with DivX4. The reason for this was that DivX4's size prediction algo's weren't functioning very well. The difference between predicted and achieved filesize for DivX4 was somewhere between 0-10MB. The new DivX5 no longer possesses this fault, its size prediction is quite accurate now. So most of the time when you enter a target bitrate the codec will hit this size very closely. However with some movies you end up with a final filesize that is over 50MB too small.

What's the deal here?

Once you get a file that is more than say 50MB under your target filesize, there's a very big chance the movie is 'maxed out'. This means that the codec has used every bit you gave it and has produced a file that is at the maximum quality, and therefor maximum filesize, and it just can't make it any bigger. In this case increasing the bitrate will not make the file any bigger, because the codec has already reached the max. The codec uses the bits you gave it to determine which quantizer to use to compress each frame. The higher the quantizer the more the frame will get compressed. With a so-called 'saturated' movie the codec will have used the lowest possible compression, which is a quantizer of 2, and it will still be compressed too much for the bitrate you entered. But because the codec can't use a quantizer lower than 2 the file will end up too small.

Some of the most common factors that play a part in getting a saturated file:
- Resolution
- B-frames
- Low motion movie
- Dark movie
- Span a movie on >1 CD's
- Filters

The lower the resolution of a video, the fewer detail it has and the fewer bits are required to compress each frame. Increasing the resolution will increase the amount of detail in each picture and thus the need for extra bits increases also.

This one of the main reasons why more people get saturated movies with DivX5 than with DivX4. Since B-frames get their information from two surrounding frames instead of just one surrounding frame for a P-frame, they will have a lower compression error. Because of this, each frame that is targeted to become a B-frame will be compressed much more than if it were a normal P-frame. So, even if the codec wants to use a quantizer of 2 on a frame (to keep the size as large as possible), if the frame will become a B-frame it will get a higher quantizer, namely 2x2=4. A file at maximum quality with B-frames will have a quantizer of 2 for P-frames and 4 for B-frames.
Disabling the Bidirectional Encoding feature will give you a bigger file, because now every frame will be get a quantizer of 2.

Low Motion:
DivX5 uses the MPEG compression scheme to get as little a file as possible with high quality. To achieve this it tries to limit the size by only encoding the differences between each frame instead of the entire frame. The more the difference between one frame and the next, the more will be encoded. So it stands to reason that a high action movie will need much more bits to get equal quality compared to a low motion movie. This is determined by the movie content, so there's nothing we can do about it.

Dark Movie:
Black is one of the most easy to compress colors, because each color component (at leat in RGB) is 0. Zero's compress easily. The more black in a movie, the easier it can be compressed. Again this is something that is part of the movie content, so there's also nothing we can do aout it.

Span a movie on >1 CD's:
Because you use a much higher bitrate you'll reach the maximum quality (minimum compression) sooner. Simply cut down on the number of CD's you use for the movie and you should be fine.

Some encoding filters incease the compressability of a movie. Things like blurring and deinterlacing make things easier on the codec. Also the type of resizing filter has much inpact on filesize. Bilinear for instance increases compression a great deal, Bicubic on the other hand decreases compression. The sharper the resizing filter, the greater the filesize will be. So if you get undersized files with one type of resizing, try a sharper method to get a bigger file.

If anyone has anything useful to add here please reply so I can make it as clear to everyone as possible. This is not an invitation for questions, post them under normal threads (after searching around if your question hasn't been answered before).
Acaila is offline  
Old 26th March 2005, 16:52   #2  |  Link
Eternally Junior Member
jon.schaffer's Avatar
Join Date: Apr 2004
Location: France
Posts: 230
Oversized/Undersized explanations

Many posts in the XviD forum are related to a very same problem: people are getting XviD encoding that are too big or to small compared to what they aimed at. This thread attempts to explain what is the cause of these issues and what are the way to correct them.

So, if you experience any problem of file size when encoding to XviD then read the following explanations before posting any questions.
Let's start with the basis you must understand - I strongly recommend you read it first: understand the problem is nearly solve it.

The basis

1. Quantization
XviD is a lossy codec, which means that it loses some picture details that it considers useless.
Each frame is encoded at a certain quantizer which is in a way a quality divider. The higher is the divider, the lower is the quality and the lower is the frame-size.
The quantizer with a value of 1 ("Q1") produces maximum quality pictures and very big files (high bitrate).
The quantizer with a value of 31 ("Q31") produces minimum quality pictures and very small files (low bitrate).

You know what the goal is: maximum quality and minimum size...
=> we could consider that Q1 is excessive since the final bitrate is much too high (if backing-up DVDs, better keep the original MPEG-2 stream...)
Q2, on the contrary, provides a very good picture quality, which is far more smaller (the relation "quality/bitrate" is not linear). Since MPEG-4 is made for lower bitrates, than e.g. MPEG-2, we can assume that Q2 delivers the maximum quality for a "normal" XviD usage.

2. About the size
At the same quantizer (i.e., same level of compression), different video sources will lead to very different file size because of their different contents which are more or less easy to compress (long movies, high resolution, lot of details, high motion movies, grainy films are considered as hard to compress).

3. The two-pass encodings
In order to achieve maximum quality for a given file size, XviD can use a two-pass encoding.
- the first pass is run at a fixed quantizer = 2.
- Second pass: the first pass encoding is used as a reference (via a stat file) - the codec constantly adjusts the bitrate (bitrate control) to optimize the overall quality.

Now, the key to understand the problem. Here are 3 scenarii that can be got with the default settings and depending on the movie compressability:
first case: the 1st pass creates a movie which size is bigger than the desired final size, then the codec has to lighten the encoding - the 2nd pass will use higher quantizers, so the final encoding will have a correct size and a maximum quality for this size (quality which will be lower than the fixed Q2 quality).
second case: the 1st pass creates a file which is smaller than the desired final size. In other words, since Q2 can be considered as maximum quality, the first pass was already perfect - just smaller as wanted... But, during the second pass, the codec is able to fatten the file, using lower quantizer frames, i.e. Q1. Theorically, the quality should be perfect: many Q2 frames + some Q1 frames.
third case: the 1st pass creates a file which has the exact target size. Statistically negligible, plus should not be a problem (have to be rigorous )

Now, the problem

Some movies are so much easy to compress that the codec has to add too much Q1 frames to fatten the file. It disturbs the bitrate control which does not manage to balance these frames. Consequence: you get an oversized file.

At last... some remedies against oversizing

If you get an oversized file, there are two solutions in order to get perfect file size:

1. Overflow treatment values tweaking
These settings are used during the second pass and tell the bitrate control which strength it can use to adjust the bitrate. Higher values allow him to be more efficient and to balance these Q1 frames.
So... recommended values are ("Encoding type" > Two-pass - 2nd pass > "more..."):
- overflow control strength: try 10 or 20
- Max overflow improvement: try 10 or 20
- Max overflow degradation: try 10 or 20

You'll get a correct final file size.

2. Quantizer capping
The second solution is to prevent XviD from using Q1 frames. To do that, just replace the minimum frame quantizer with "2" for each frame type [I/P/B] ("Advanced option" / quantization dialog - in latest XviD releases : "Quality preset" > Userdefined > "more..."), and leave the maximum at 31.
You will have no more oversized. But... since you have read the explanations above, you must have understood that, doing this, you will get an undersized file, since there are no Q1 frames to fatten it. But, once more, it will be at maximum quality!
In this case, consider it as a good occasion to make your encoding better. Since you have "free space" to fill, use it by increasing the resolution, increasing the sound quality, using high quality custom matrices, etc.

About the undersizing
See above

A way to verify that your case fit in this issue... and to prevent it

During the first pass, the codec creates not only a fake file (or a real one if you disabled "Discard first pass") but also a log file (called ".pass" or ".stat" file too). This file "describes" the movie for the second pass process. Within the informations it contains, one is interesting for us: the size of the first pass file, i.e. the size of a constant Q2 encoding.
This way, you can see just after the first pass is finished, if your file must be lightened or fattened during the second pass. If this first pass size is very small comparing to what you finally want, then the risk of oversizing is high...

How to see this "size" information?
-> Launch the StatReader (included in official releases), open the stat/pass/log file and see the Size (MB)...
This way you can prevent this issue... and restart maximizing the quality of the video...

------ End
[EDIT by Koepi]: Cleaned up the discussion stuff. This is a sticky, not a discussion
[EDIT by JS]: Perfect!

[EDIT: added the use of the StatReader / typos / new buttons for settings in new releases / small modifications]
"Methinks it is like a weasel" (Hamlet in Hamlet, by Shakespeare)

Last edited by jon.schaffer; 24th August 2006 at 22:35.
jon.schaffer is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +1. The time now is 03:11.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.