over-sized XviD video

[SoundScape]

Hello,

I am having an "over-sized video file" problem while trying to encode a DVD rip to XviD. XviD codec version used is 1.1.0 Beta 1 (Koepi's build - 16/01/2005) and Gordian Knot version is 0.35.0. Here are the movie stats and the Gordian Knot/XviD configuration used during encoding:

Movie Info:
=========

Frames/Duration: 131025/1h27m21s
Credits Start At Frame: 122997
Video Type: PAL
Video AR: 16:9
Audio: 6ch AC3 - 448kbps


Gordian Knot:
==========

Audio A: 286617KB (280MB)
Target File Size: 1433600KB (1400MB)
calculate Frame Overhead: ON - 1 Frames
Average Bitrate: 1783kbps
Video Size: 1140807KB (1114MB)
Resize to: 640 x 256
Bits/(Pixel*Frame) = 0.435
Resizing Filter: Bicubic (Sharp)
De-Interlacing: Not Needed
Filters: NONE
Re-Calculate Bitrate If Needed: ON


XviD Configuration:
==============

First Pass:
===========

Profile @ Level: AS @ L5
========================

Quantization Type: H.263
Adaptive Quantization: ON
B-VOPs: ON
Max Consecutive BVOPs: 2
Quantizer Ratio: 1.50
Quantizer Offset: 1.00
Packed Bitstream: ON
All other options: OFF

Encoding Type: Twopass - 1st Pass
=================================

Full quality first pass: OFF
Discard first pass: ON

Zones
=====

1st Zone: Start Frame 0
Weight: 1.00
Modifiers: NONE
BVOP Sensitivity: 0
2nd Zone: Start Frame: 122997
Quantizer: 20.00
Modifiers:
Begin with keyframe: ON
Grayscale encoding: ON
Cartoon Mode: OFF
Chroma optimizer enabled: OFF
BVOP Sensitivity: 0

Advanced Options
================

Motion:
Motion search precision: 6 - Ultra High
VHQ Mode: 4 - Wide Search
Use VHQ for B-Frames too: ON
Use chroma motion: ON
Turbo: OFF
Frame drop ration: 0
Maximum I-Frame interval: 250

Quantization:
Min I-Frame: 1
Max I-Frame: 31
Min P-Frame: 1
Max P-Frame: 31
Min B-Frame: 1
Max B-Frame: 31
Trellis quantization:ON

Second Pass:
============

Profile @ Level: AS @ L5
========================

Quantization Type: H.263
Adaptive Quantization: ON
B-VOPs: ON
Max Consecutive BVOPs: 2
Quantizer Ratio: 1.50
Quantizer Offset: 1.00
Packed Bitstream: ON
All other options: OFF

Encoding Type: Twopass - 2nd Pass
=================================

All options at their default values
Target bitrate: 1783kbps (Target size: 1140807KB)

Zones
=====

1st Zone: Start Frame 0
Weight: 1.00
Modifiers: NONE
BVOP Sensitivity: 0
2nd Zone: Start Frame: 122997
Quantizer: 20.00
Modifiers:
Begin with keyframe: ON
Grayscale encoding: ON
Cartoon Mode: OFF
Chroma optimizer enabled: OFF
BVOP Sensitivity: 0

Advanced Options
================

Motion:
Motion search precision: 6 - Ultra High
VHQ Mode: 4 - Wide Search
Use VHQ for B-Frames too: ON
Use chroma motion: ON
Turbo: OFF
Frame drop ration: 0
Maximum I-Frame interval: 250

Quantization:
Min I-Frame: 1
Max I-Frame: 31
Min P-Frame: 1
Max P-Frame: 31
Min B-Frame: 1
Max B-Frame: 31
Trellis quantization:ON

Unfortunately, after encoding, target size of video+audio was not met. Instead of getting 1433600KB (1400MB), I got 1494743KB (approx. 1459MB). Or, instead of a desired bitrate of 1783kbps I got a birate of 1827kbps. In other words, an over-sized file...

Could anyone tell me what went wrong here? Did I miss something? Also, another thing: Why all XviD videos have one frame less than the original video? Is this a bug or some kind of a trick? :)

Finally, are there any settings that need to be turned OFF in the XviD encoder without affecting image quality much? Should I use MPEG quantization since my bitrate is quite high? Or do you happen to know of any custom matrix that can perform "better" than the standard one (by "better" I mean a sharper and more detailed image)?

Thank you in advance for your help, I hope I didn't bore you with so many questions...:)

[Didée]

Was discussed and explained a bazillion times already.

1:27 is rather short, 640*256 is rather small, and 1114 kB is much under these circumstances. Naturally, the encoding would have come out undersized (too little data to encode, to reach destination size @ max quality). Thus a lot of q1-Frames were used. This disturbs rate control. Towards the end, while XviD was still in the middle of struggling to compensate for those silly q1-frames, the credits zone was reached. This one was set to encode @ fixed q20, therefore no further rate control was possible: the overflow that was actually present at the start of the credits got "frozen".

Search for "oversize" and "q1", and get ashamed by the mere number of search hits youll get ...

[jon.schaffer]

SoundScape... may I draw your attention on the first sticky of the list... ? :rolleyes:

[SoundScape]

Didée,

I wasn't aware that this has been discussed so many times before...:o I guess I needed a quick answer without looking any further. And no, I wasn't ashamed by the number of posts I found! :D

jon.schaffer,

Sorry, must have missed that...:D

No one has answered my other questions yet...hmmm...probably they were asked too many times as well? Have a look anyway :)

Why all XviD videos have one frame less than the original video? Is this a bug or some kind of a trick?

Finally, are there any settings that need to be turned OFF in the XviD encoder without affecting image quality much? Should I use MPEG quantization since my bitrate is quite high? Or do you happen to know of any custom matrix that can perform "better" than the standard one (by "better" I mean a sharper and more detailed image)?

After reading the sticky and some posts that address the problem, I'll get back to you guys. Most likely you'll want to know what went wrong...Thanks again.

[jon.schaffer]

Why all XviD videos have one frame less than the original video? Is this a bug or some kind of a trick?

Not sure about this. Two ideas came to my mind:
1) this is related to the software you use. I don't know GordianKnot, but I know that VirtualDub has a particular way to do its selection.

2) More probably (?): this may be related to the use of B-frames (try without B-frames and compare). Could it be linked to the fact that the B-frames are stored in a "hacked" way in P-frames, so that some of them (the last ones) seem not to exist?

It's guess from my part. Not expert enough in that part. Another could answer you more... technically.


For the 2nd question: difficult to answer. You'd really better try for yourself.
Some will say that you should disabled AQ, others will say you should use less or more B-frames.
As for the matrix... you'll get different answers depending on the people preferences...
And, once again: SEARCH. This post is only 4 steps far from yours...:

http://forum.doom9.org/showthread.php?threadid=90954

SiXXGuNNZ compiled custom matrices. "_matrices_info_.txt" on his site will provide you good informations.

[SoundScape]

jon.schaffer,

Thank you for your reply. You are right, I have to try and see for myself. Quality is very subjective!

One more thing (and I know it may be hard to answer too :D ): Do different XviD builds have major differences? I am using Koepi's builds and I was wondering if there are others that may work better. Is there an official binary I can download from xvid.org? Although I searched the website, all I found was source code...

Thanks in advance :)

[jon.schaffer]

Although I searched the website, all I found was source code...
Yes, indeed... only sources at xvid.org.

I don't dig into the _very latest_ builds too much and I prefer wait for some more _official_ builds.
There are different builds which may be more CPU optimized, or that correct some minor bugs (cartoon mode with GMC lately, IIRC). Quality wise, I think it's minor. So you could consider latest Koepi's beta compilation (1.1) as the "best" official one, waiting for the next official release.
Concerning the differences between versions: xvid.org -> change log...

[Elias]

@topic creator
Change the quantizers to 12 on all maximum, and 2 on all minimum. Problem solved, case closed... fact is, XviD should come installed with default settings using these quantizers.

[Manao]

> Change the quantizers to 12 on all maximum

No. There's no need to do that. Not ever except in very specific cases where the user know what he's doing. Max quantizer should stay defaulted to 31.

> 2 on all minimum

Maybe. But then, users will complain of getting undersized files. And there'll be more complaints than with minimum sets to 1, because in most cases, quantizer 1 doesn't create issues.

[Elias]

Originally posted by Manao
> Change the quantizers to 12 on all maximum

No. There's no need to do that. Not ever except in very specific cases where the user know what he's doing. Max quantizer should stay defaulted to 31.

> 2 on all minimum

Maybe. But then, users will complain of getting undersized files. And there'll be more complaints than with minimum sets to 1, because in most cases, quantizer 1 doesn't create issues. Why? DivX 5.02 had these settings, and they work pretty good. I've NEVER EVER had any problems with these settings. The size is perfect, and the quality is almost exactly as good as the 1-31 settings.

[Koepi]

That is YOUR case where these settings work out fine, Elias.

It's not the GENERAL case. So please don't think your settings are better than the defaults - in this matter they are not, most likely they don't even affect your encoding at all - but for others this would cause the trouble that filesize will get too big or some scenes look really bad or VBV buffer overflows or .... or ...

I hope you understand that.

Cheers
Koepi

[jon.schaffer]

@Elias:
Once I used DivX 5.02. I already had an undersized "problem" (that's not really a problem) with it. So this can happen.
There's a sticky about this issue in the DivX forum... I guess it's not for nothing.

[SoundScape]

Guys, guys...I used the settings listed in my first post but I changed quantization to MPEG instead of H263. Guess what...Target filesize was reached :D No need to mess with min/max quantizers. Video quality is absolutely stunning, most of the detail of the source was preserved, thank you XviD!!!

By the way jon.schaffer, your sticky is excellent. Helped me a lot :cool:

P.S. I think I'll say "bye bye" to DivX. No more smoothing with XviD...

[jon.schaffer]

MPEG quantization matrix preserves more details, thus the 1st pass create a bigger file... easier for the codec to reach the desired size.
Unfortunately, this does not work in all the cases :(

[Blue_MiSfit]

Originally posted by SoundScape
P.S. I think I'll say "bye bye" to DivX. No more smoothing with XviD... [/B]

Hooray! You've seen the light (or details?):)

[SoundScape]

Guys, I have a question regarding MPEG Quantization in XviD. Is this type of quantization based on MPEG-2 or MPEG-4? Also, I have come across some matrices that are made for "Ultra Low" bitrates. Why use MPEG-Custom quantization when H263 produces better results at low bitrates?

Thanks in advance :D

[Elias]

Originally posted by SoundScape
Guys, I have a question regarding MPEG Quantization in XviD. Is this type of quantization based on MPEG-2 or MPEG-4? Also, I have come across some matrices that are made for "Ultra Low" bitrates. Why use MPEG-Custom quantization when H263 produces better results at low bitrates?

Thanks in advance :D Good questions. I don't see any reason for using customized matrices though. H.263 kicks ass imo :)

[SoundScape]

It seems that I solved the over-sized video problem but another tiny problem surfaced...my video file is 5MB under-sized :D So, why make such a fuss? Well, I am a perfectionist, bad for me I guess...

Maybe the following creates the problem, I am not sure so any ideas are welcomed. Target size of movie (1h48m8s) is 1400MB (1433600KB) with an AC3 audio (448kbps). Gordian Knot calculates a bitrate of 1353kbps for the video. Also, it reports a video size of 1046MB (1071158KB). The Frame-Overhead calculation gives 7MB (7645KB) when 1 Frame is used. Of course, adding all that up gives a total size of 1400MB indeed. On the other hand, when using XviD's bitrate calculator, the Overhead is calculated at 4118KB and the bitrate at 1356kbps. Movie size is 1074670KB, different than that of Gordian Knot. I noticed that if number of frames in the latter is changed from 1 to 16, the frame- overhead becomes equal to that of XviD's calculator, i.e. 4118KB and the bitrate (in Gordian Knot) is almost identical, 1357kbps.

Can anyone explain to me why all the above are happening? Can Frame-Overhead calculation be responsible for the small under-sizing of the final video? Is it safe to use a frame number of 16 in Gordian Knot? Forgot to tell you that the encoded video has a bitrate of 1347kbps-0.36% overhead (according to VirtualDubMod) and a bitrate of 1350kbps according to GSpot.

Again, many thanks for your thoughts and help! :p

[Blue_MiSfit]

Originally posted by Elias
Good questions. I don't see any reason for using customized matrices though. H.263 kicks ass imo :)

H.263 is an excelent general purpose quantization type that will produce acceptable results in almost all cases (assuming you don't starve it for bitrate). However, the cool thing about having custom quantization matricies is that you can select one that is optimized for higher bitrates.

For example - Didee's superb 6of9 matrix is designed for high bitrate. I do 1/3 to 1/2 DVD-R sized backups, so this means around 1.4 gigs for just the video stream in most of my encodes. That kind of bitrate would saturate h263, and I would get loads of undersize. To avoid this and use all the space I have allocated in the most effecient way, I can use a high bitrate matrix - like 6of9 or perhaps sharktooths V3UHR. This will produce a higher average quantizer than h263 at high bitrates, but will preserve much more details and sharpness.

Sorry if I got a little bit long winded. In all honesty, the difference between h263, mpeg, and a custom matrix is usually pretty small, and it takes trained eyes to notice a huge difference. Try some custom matricies, and if you like the result use them, and if you cant tell a difference, use h263 for low bitrate and MPEG for high bitrate - it's as simple as that :)

Give it all a chance :)
~misfit

[Elias]

Originally posted by Blue_MiSfit
However, the cool thing about having custom quantization matricies is that you can select one that is optimized for higher bitrates.Yes, but they will break the MPEG-4 compliance, right? MPEG-4 compliance means everything to me when encoding to MPEG-4 :)

[Sharktooth]

WTF? Break what?
Custom Matrices are part of the MPEG4 ASP.
They wont break anything.

[Elias]

Originally posted by Sharktooth
WTF? Break what?
Custom Matrices are part of the MPEG4 ASP.
They wont break anything. Cool down man, I was asking a question. I've read somewhere by bond or doom9 that using for instance MPEG-custom breaks the specs a little. Is it true?

[Sharktooth]

no.

[jon.schaffer]

Originally posted by Elias
MPEG-custom breakes the specs a little. Is it true?

Originally posted by Sharktooth [master of the short answers - do not forget it ;)]
No

IIRC, only the switching between two matrices in one movie is not MPEG4-compliant (there was such an option in a previous XviD version which allow the alternate use of H263-quant and MPEG mtx). But was it really not compliant... I think it's the reason this option does not exist anymore...

[Elias]

Originally posted by jon.schaffer
IIRC, only the switching between two matrices in one movie is not MPEG4-compliant (there was such an option in a previous XviD version which allow the alternat use of H263-quant and MPEG mtx). But was it really not compliant... I think it's the reason this option does not exist anymore... You're right, that's the one I was talking about. Couldn't remember exactly what the fuss was about though. But it's just a good thing that they removed it. I hate things that breaks the compliance :)

[SoundScape]

Thanks for the cool info guys :)

But I need some answers about 2 things I asked some posts above. First: is MPEG Quantization based on MPEG-2 or MPEG-4? Because if it's based on the former, then why not encode using CCE (for example) in the first place?

Second: It's the question concerning the under-sized video by 5MB and the Frame-Overhead issues...Read my previous post if it's not much trouble :)

Also, something else I've noticed when NOT using the "Packed Bitstream" option. If this option is unchecked, VirtualDub shows the following message, just before the 1st frame is played: "Warning: Nothing to output. B-Frame decoder lag". Is this normal and something not to worry about? Video plays normally in Media Player though. I apologise if this is answered somewhere.

Many thanks in advance. Your input on the above is much appreciated!

[unskinnyboy]

Originally posted by SoundScape
Also, something else I've noticed when NOT using the "Packed Bitstream" option. If this option is unchecked, VirtualDub shows the following message, just before the 1st frame is played: "Warning: Nothing to output. B-Frame decoder lag". Is this normal and something not to worry about? Video plays normally in Media Player though. I apologise if this is answered somewhere.Asked and answered about 6473432 times so far.:)

Check here (http://forum.doom9.org/showthread.php?s=&threadid=84018) and here (http://forum.doom9.org/showthread.php?s=&threadid=78329) and more and more and more (http://forum.doom9.org/search.php?s=&action=showresults&searchid=1418226&sortby=lastpost&sortorder=descending)

[Sharktooth]

Originally posted by SoundScape
But I need some answers about 2 things I asked some posts above. First: is MPEG Quantization based on MPEG-2 or MPEG-4? Because if it's based on the former, then why not encode using CCE (for example) in the first place?
That's a nonsense.
Whatever matrix (including MPEG) or whatever quantization type you use with a MPEG-4 codec will result in a MPEG-4 encode.

[SoundScape]

unskinnyboy,

Thanks for the links, my question has been answered. I think it's best to turn "Packed Bitstream" OFF. Would this result in incompatible video if it is played on a DivX standalone?

Sharktooth,

Sorry if my question seems nonsense to you. To me sounds perfectly good considering my limited knowledge on XviD. Thanks for the answer anyway. And being polite in your replies wouldn't hurt much, would it?

[jon.schaffer]

Originally posted by SoundScape
... the question concerning the under-sized video by 5MB and the Frame-Overhead issues...Read my previous post if it's not much trouble :)
1) You solved your oversized issue by using H263 quant instead of MPEG mtx (unless I missed something). This solution does work for your encoding but is not a general solution.

2) 5 MB are not such a big undersizing. It's a quite common "undersize error" (by "error", I mean "variation", not "bug"). What container do you use? An AVI is bigger than a MKV with the *exact* same content (cf. overhead...).

3) You are a perfectionist... Try applying the advices you read in the sticky: keep MPEG mtx, for instance, and tweak the overflow treament values (to 10/10/10 or even 10/5/10 - this could ease the bitrate increasing in order to compensate these 5 MB). Then, you may get a closer size to what you expect.

EDIT: typos

[Sharktooth]

Originally posted by SoundScape
Sharktooth,

Sorry if my question seems nonsense to you. To me sounds perfectly good considering my limited knowledge on XviD. Thanks for the answer anyway. And being polite in your replies wouldn't hurt much, would it?
Sorry, i didnt meant to sound harsh.
It's just too much stress.
However a quantization matrix does only define how quantization is done.
Using the MPEG-2 quant matrix in a MPEG-4 encoder does not mean the result will be a MPEG-2 encode.

[unskinnyboy]

Originally posted by SoundScape
unskinnyboy,

Thanks for the links, my question has been answered. I think it's best to turn "Packed Bitstream" OFF. Would this result in incompatible video if it is played on a DivX standalone?
Depends on your standalone player actually.

Some are known to have issues -with- packed bitstream and some -without-. If your player is DivX certified, then it should handle packed bitstream since it is part of the DivX profile. But for other standalones which plays MPEG-4 but are not DivX certified, you need to test it out for yourself. Try playing both and see. Sometimes both might work.

I always consider it a safe bet to use packed bitstream. If ever it is seen to cause a problem, it can be unpacked using MPEGModifier (http://www.moitah.net/download/latest/MPEG4Modifier.zip) , but never the other way around unless you re-encode.

[SoundScape]

jon.schaffer,

Always a pleasure and a great help getting answers from you. I'll keep what you suggested in mind. Many thanks :D

Sharktooth,

Apology accepted :D Thanks for your reply.

unskinnyboy,

I don't have an MPEG-4 standalone yet. I am just asking cause you never know. I wouldn't want all my future XviD encodings to be incompatible and unplayable ;) Anyway, thanks for your reply!


Thank you all for taking the time to answer my questions. You make Doom9 what it is: THE place to be!!!