XviD overshooting target file size

[Katie Boundary]

Is there anything, other than user error, that could cause a 2-pass XviD encode to exceed its intended file size? In particular, will it overshoot if the target file size is "too low" for the video's resolution, frame rate, and amount of motion? I just had to set the target amount of video data to 280 mb in order to get the actual amount of data down to ~310 mb (with audio adding another ~40 mb for a total file size of 350 mb).

[StainlessS]

Dont know, but TmpGenc mpeg used to have a minimum bitrate that it would not go below (and would not tell you that it was going to ignore your settings).
(about 200kb/s I think, for mpeg)

[Selur]

@Katie Boundary: try setting overflow treatment settings all to 10% instead of the default 5%.

[Katie Boundary]

@Katie Boundary: try setting overflow treatment settings all to 10% instead of the default 5%.

That helped somewhat.

[Blue_MiSfit]

Any particular reason you're still encoding MPEG-4 ASP in 2017?

Mainly I'm just curious :)

[fantasmanegro]

Any particular reason you're still encoding MPEG-4 ASP in 2017?

Mainly I'm just curious :)

Old DVD player compatibility as I do?:D

[Katie Boundary]

Any particular reason you're still encoding MPEG-4 ASP in 2017?

Mainly I'm just curious :)

What is ASP?

[Midzuki]

What is ASP?

Advanced Simple Profile, of course.

https://en.wikipedia.org/wiki/MPEG-4_Part_2#Advanced_Simple_Profile_.28ASP.29

[Selur]

@Katie Boundary: also make sure qmin=1 and qmax = 51

[Midzuki]

@Katie Boundary: also make sure qmin=1 and qmax = 51

I think you meant qmin=2 and qmax=31 :)

[Katie Boundary]

Advanced Simple Profile, of course.

https://en.wikipedia.org/wiki/MPEG-4_Part_2#Advanced_Simple_Profile_.28ASP.29

Oh. No, I don't bother with profiles. I always set the profile to unrestricted.

[Groucho2004]

Oh. No, I don't bother with profiles. I always set the profile to unrestricted.
So why are you using such an inefficient encoder?

[Katie Boundary]

So why are you using such an inefficient encoder?

If I tell you, would that make it easier for you to figure out why XviD is misbehaving?

[Groucho2004]

If I tell you, would that make it easier for you to figure out why XviD is misbehaving?
What makes you think it's misbehaving? When you set all parameters the xvid library exposes correctly it will hit the desired bitrate unless you're trying something silly like encoding a series of black images into a 10Mbps video.

Besides, this question has been asked hundreds of times in the last ~15 years. A simple google search ("xvid undersized site:forum.doom9.org") would have provided the same answers you got in this thread and a lot more suggestions to solve the problem.

[fantasmanegro]

What gui or software are you using for xvid encoding?...

[hello_hello]

Old DVD player compatibility as I do?:D

Yeah, but don't you knock up a quick copy for DVD player viewing with Xvid single pass encoding, or 2 pass encoding if you're really keen, using the x264 encode you ran for archiving as the source? ;)

[fantasmanegro]

Yeah, but don't you knock up a quick copy for DVD player viewing with Xvid single pass encoding, or 2 pass encoding if you're really keen, using the x264 encode you ran for archiving as the source? ;)

i guess not... if you have an old 26" crt tv at home :rolleyes:

[Selur]

I think you meant qmin=2 and qmax=31
No, I meant qmin=1 and qmax=31. :)

[Katie Boundary]

What makes you think it's misbehaving?

The part where I tell it to aim for one bitrate and it gives me a completely different bitrate even though I'm not feeding it pathological content.

What gui or software are you using for xvid encoding?...

Virtualdub.

[Groucho2004]

The part where I tell it to aim for one bitrate and it gives me a completely different bitrate even though I'm not feeding it pathological content.Read the second paragraph of my post and try the various suggestions. This can't be solved by guessing what might be wrong with your setup, there are too many variables.
You could of course upload your source so one of us could try but that seems rather excessive for such a trivial problem.

p.s. What version of xvid are you using?

[Katie Boundary]

1.3.4

I'm currently converting season 2 of Modern Family to VCD-compliant Mpeg-1, but once that's done, I'll dust off some of ye olde Andromeda or Birds of Prey scripts and see if changing the minimum quantizer from 1 to 2 has much of an effect.

[Sharc]

...and see if changing the minimum quantizer from 1 to 2 has much of an effect.
It depends on the matrix. See for example the discussion here
http://forum.doom9.org/showthread.php?p=787775#post787775
Looking forward to seeing your findings.

[Katie Boundary]

Unfortunately, upping the minimum quantizer did not fix the problem.

[Sharc]

Same problem with single-pass and target bitrate?

[hello_hello]

Katie probably won't read this post, or pretend not to, because I think I'm one of the "other forum members" she refers to in her signature which means I'm on her ignore list so I can't learn from her reading my posts..... or however it works according to her signature...... but...
she probably should try what I did here (https://forum.doom9.org/showthread.php?p=1803680#post1803680) and run a CQ2.0 single pass encode to see what the resulting bitrate will be, and whether it's in the same ballpark as the bitrate she's using for 2 pass encoding. Or she could at least run a compression test. I'd explain how to run one but I probably haven't got it wrong enough to learn how.... or however it works according to her signature..... but it's just guessing without any further details.

I played around a little more with the sample I'd been using for testing a couple of days ago, and this time Xvid wasn't overshooting the requested bitrate. After hunting through log files to see WTF, it turned out I somehow reset the overflow percentage. I don't remember doing that, but with it increased to 10% again today the target bitrate was almost exact.

Highlights from yesterday's log file:

--[Information] [14/04/17 7:09:27 PM] Avisynth input script
--[Information] [14/04/17 7:09:27 PM] resolution: 960x396
--[Information] [14/04/17 7:09:27 PM] frame rate: 24000/1001
--[Information] [14/04/17 7:09:27 PM] aspect ratio: 80:33 (2.424)
--[Information] [14/04/17 7:09:27 PM] custom command line: -vbvsize 3145728 -vbvmax 4854000 -vbvpeak 2359296
--[Information] [14/04/17 7:09:27 PM] Job command line: "C:\Program Files\MeGUI\tools\xvid_encraw\xvid_encraw.exe" -i "D:\Test.avs" -o "D:\h263.m4v" -pass2 "D:\h263.stats" -bitrate 2000 -max_key_interval 240 -max_bframes 1 -bvhq -masking 2 -imin 1 -bmin 1 -pmin 1 -par 1 -vbvsize 3145728 -vbvmax 4854000 -vbvpeak 2359296
--[Information] [14/04/17 7:09:49 PM] Final statistics
---[Information] [14/04/17 7:09:49 PM] Video Bitrate Desired: 2000 kbit/s
---[Information] [14/04/17 7:09:49 PM] Video Bitrate Obtained (approximate): 2936 kbit/s

And today's:

--[Information] [17/04/17 10:01:35 PM] Avisynth input script
--[Information] [17/04/17 10:01:35 PM] resolution: 960x396
--[Information] [17/04/17 10:01:35 PM] frame rate: 24000/1001
--[Information] [17/04/17 10:01:35 PM] aspect ratio: 80:33 (2.424)
--[Information] [17/04/17 10:01:35 PM] custom command line: -vbvsize 3145728 -vbvmax 4854000 -vbvpeak 2359296
--[Information] [17/04/17 10:01:35 PM] Job command line: "C:\Program Files\MeGUI\tools\xvid_encraw\xvid_encraw.exe" -i "D:\Test.avs" -o "D:\Test megui 2050.m4v" -pass2 "D:\Test megui 2050.stats" -bitrate 2050 -max_key_interval 240 -ostrength 10 -oimprove 10 -odegrade 10 -max_bframes 1 -bvhq -masking 2 -imin 1 -bmin 1 -pmin 1 -par 1 -vbvsize 3145728 -vbvmax 4854000 -vbvpeak 2359296
--[Information] [17/04/17 10:01:51 PM] Final statistics
---[Information] [17/04/17 10:01:51 PM] Video Bitrate Desired: 2050 kbit/s
---[Information] [17/04/17 10:01:51 PM] Video Bitrate Obtained (approximate): 2058 kbit/s

That's Xvid 1.3.4.

Out of curiosity, I tried the same encode using AutoGK and Xvid 1.2.1. AutoGK wasn't happy about the bitrate.

[17/04/17 10:06:48 PM] Custom resolution settings: fixed width of 960 pixels
[17/04/17 10:06:48 PM] Standalone support enabled: ESS
[17/04/17 10:06:48 PM] Started encoding.
[17/04/17 10:06:48 PM] Source resolution: 960x396
[17/04/17 10:06:48 PM] Source fps: 23.976
[17/04/17 10:06:48 PM] Output will contain 1001 frames
[17/04/17 10:06:48 PM] Using VAQ in XviD
[17/04/17 10:06:48 PM] Overhead: 0 bytes (0.00 Mb)
[17/04/17 10:06:48 PM] Video size: 10,485,760 bytes (10.00 Mb)
[17/04/17 10:06:48 PM] Running compressibility test.
[17/04/17 10:06:48 PM] Writing the following script to D:\agk_tmp\test 2050_agk_comptest.avs
===========================================================
[17/04/17 10:07:10 PM] Duration was: 21 seconds
[17/04/17 10:07:10 PM] Speed was: 46.59 fps.
[17/04/17 10:07:10 PM] Compressibility percentage is: 191.70
[17/04/17 10:07:10 PM] Using sharper matrix
[17/04/17 10:07:10 PM] Switching b-frames off
[17/04/17 10:07:10 PM] Chosen resolution is: 960x400 ( AR: 2.40 )
[17/04/17 10:07:10 PM] Predicted comptest value is: 129.65%
[17/04/17 10:07:10 PM] Running first pass.
[17/04/17 10:07:10 PM] Writing the following script to D:\agk_tmp\test 2050_agk_movie.avs
===========================================================
[17/04/17 10:07:31 PM] Duration was: 20 seconds
[17/04/17 10:07:31 PM] Speed was: 47.70 fps.
[17/04/17 10:07:31 PM] Expected quality of first pass size: 133.54%
[17/04/17 10:07:31 PM] Trying to adjust settings.
[17/04/17 10:07:31 PM] No adjustment is possible
[17/04/17 10:07:31 PM] Warning: final AVI will likely be undersized.
[17/04/17 10:07:31 PM] Running second pass.
[17/04/17 10:07:52 PM] Duration was: 20 seconds
[17/04/17 10:07:52 PM] Speed was: 47.70 fps.
[17/04/17 10:07:52 PM] Job finished. Total time: 1 minute, 3 seconds

It lied about using the sharper (mpeg) matrix, probably because I have the most strict hardware compatibility setting enabled, but the resulting bitrate was 1438kbps, however visually it wasn't that far off the encode I ran at 2050kbps in respect to detail retention. Maybe about 95%, but the bitrate was about 600kbps less.

AutoGK doesn't set the min quantizer to 1. Instead, when the quality is high, it actually does this (for lower quality it might reduce the min and max to 2 and 4 respectively etc.)....
Would the following really force the same quantizer for all frames (assuming B-Frames were still enabled)?

https://s27.postimg.org/8qwd6he2r/xvid1.gif https://s27.postimg.org/tminhqa9v/xvid2.gif

[Katie Boundary]

Old thread, but I thought I'd let you all know how I ultimately fixed the problem:

Remember when upping a few key variables from 5% to 10% helped a little bit, but not enough? Well, I tried cranking them up to around 50% to see what would happen... and that worked! Now I'm playing around with restricting the range of permitted quantizers in order to minimize the "spoiling" of the quantizer distribution.

Special thanks to Selur for providing the one response that was actually helpful.