problem with rate adjustment?

[plugh]

In a couple of my recent encodes I've observed a behaviour that makes me suspicious I am bumping into an inherant limitation in the rate control / bit redistribution mechanism.

Specifically, these are two-pass encodes, where the second pass target size is fairly near the size of the first pass output file. Due to the application of vbv parameters and low weight end-credit zone, I suspect xvid is trying to redistribute some freed-up / extra bits.

What I am seeing is a general pattern of a q1 P-frame followed about half a dozen frames later by one or two q3/4 ones. I suspect the q1 frame overshot and the lower quant frames are an attempt to recover. In general, these q1 frames occur about every 2-3 seconds through a scene.

- If a scene would truly benefit from some lower quant frames, I would like to allow that to continue.
- I am much more concerned about oversize than undersize.

Given the above, what is the best approach?

Just ignore it (the 'easy way')

Iterate choosing a lower target size until there aren't enough spare bits for it to create these infrequent q1 frames. (yuck)

Force min p quant to 2? (this totally precludes improvement in 'needy' scenes?)

Play with i-frame boost params (try to give more of the extra bits to keyframes)?

Diddle with the overflow settings? Perhaps set 'max overflow improvement %' to zero? Not sure I fully understand the ramifications of doing this...

FYI - I'm using xvidvfw 1.1.0

[Didée]

Yes indeed. It has been mentioned and discussed a thousand times. Making the default quant restrictions q1-q31 was a decision in order to not get a hundred "why do I get undersizing" whining posts per week. Achieved overall quality won't raise from allowing q1, but might even degrade.
A better idea would be to use min.quant of q2, and use a matrix that produces bigger filesizes at given quantizers (so-called "high bitrate" matrices).

IIRC, one of the more recent mentionings of the q1 desaster was in this post (http://forum.doom9.org/showthread.php?p=866322#post866322).

A good reading would also be the Xvid presets (http://forum.doom9.org/showthread.php?t=107897) thread, wich is basically about finding a balanced mixture of Xvid settings for a source to be encoded.

[plugh]

I like your picture in that thread. Yup, that's it exactly.

Like I said, I don't care if it comes in smaller than my specified max size. It just seemed like the rate control was trying overly hard over a relatively short span of frames.

The fact it put in the q1 and then squeezed a nearby frame to compensate sounded like the descriptions associated with the 'overflow treatment' parameters. Like maybe the 5/5/5 settings were too aggressive in this case. The sticky thread suggests larger values for the oversize case, but nothing for this case.

Would 1/1/1 or 1/0/1 or even 1/0/0 or some such be better values than the default for this case - where 1st and 2nd pass sizes are near each other? Specifically, 1st pass was 8.26GB and I want final size less than 8GB after vbv and end-credit shrink (its an HD encode from a 22+GB TS file).

I'm a bit reluctant to force q2; that 'feels' like overkill to me. I'd like to allow it to boost keyframes and very low bitrate scenes, it's just this pattern of inserting a q1 followed shortly by a q4 in 'normal' scenes I'd like to kill, even if the file ends up undersized,,,

[jon.schaffer]

It's an issue I have been interested in, some tests lead me to that settings: http://forum.doom9.org/showthread.php?t=92366

[plugh]

Interesting reading; among other things it makes me think that zero in any of the three fields will yeild the "10" default. It also makes me start to doubt that the overflow parameters are the culprit or the solution.

I guess I'll have to dig into the code, but I think what I am seeing is an artifact of the mechanism that attempts to adjust Quants in response to the scaled frame sizes and the running total of the differance between them and the actual encoded frame sizes.

As I understand it, at the beginning of the second pass the 1st pass data is read in and processed - the frame sizes are scaled as per a number of differant algorithms, in stages. For example, the LAST stage is the vbv stuff, that takes bits away from 'evil scenes' (I liked that comment) and distributes them to others at the other end of the vbv spectrum.

Then, it actually starts the encoding. And as it goes along, it looks at the scaled length for a frame vs the actual encoded length and keeps a running total of the difference. If that differance goes sufficiently positive, it will lower quant for the next frame, to try to 'catch up'. If it is sufficiently negative, it raises the quant for the next frame(s). And I would guess there are thresholds for 'sufficiently'...

At least, that is how I think about it.

I've attached three snapshots of a running encode that illustrate my case. It is being to aggressive in 'catching up'. If it either (1) waited a while longer before doing the q1, then the accumulated delta would be sufficient that it wouldn't need to correct a few frames later or (2) waited longer after inserting the q1, then the deltas it was accumlating would eventually bring things back into line without needing the lower quant frame(s).

I think in my case, the deltas between the calculated-scaled and actual-encoded are fairly small, which is in effect throwing the rate control mechanism into oscillation, because it doesn't accumulate delta fast enough...

I'll have to dig into the code, but I'll guess that the 'overflow' parameters are used during the preliminary scaling phase, not during the encoding phase, so they would only indirectly affect this behaviour.

Comments, O Wise Ones? :)

edit - wow, those jpgs looked bad - replaced with png's

[plugh]

Didée, tried your suggestion, setting p and b min quants to 2.

Results are fascinating... Not only did it eliminate the spike / trough pattern, it somehow actually resulted in a switch to a 'shrink' pattern (removing bits from occasional frames) from the 'grow' pattern (adding bits to occasional frames) it had, and the file ended up 'exact' sized.

fyi - I'm also using your 2/1.62/0 setting on this encode, first time I've tried them. Results are great!

Thanks!

[plugh]

Was doing some more experimenting and code examination...

The troughs following the spikes WERE because the overflow mechanism was being to aggressive. Stumbled upon this chunk of conditionalized code, which is enabled in the 'normal' xvid builds.
#ifdef SMART_OVERFLOW_SETTING
if (scaler > 0.9) {
rc->param.max_overflow_degradation *= 5;
rc->param.max_overflow_improvement *= 5;
rc->param.overflow_control_strength *= 3;
} else if (scaler > 0.6) {
rc->param.max_overflow_degradation *= 2;
rc->param.max_overflow_improvement *= 2;
rc->param.overflow_control_strength *= 2;
} else {
rc->min_quant = 2;
}
#endif
In my case 'scaler' was ~1.0, which causes this code to turn the vfwgui default overflow settings of 5/5/5 into 15/25/25. I reencoded the clip with 2/1/1 settings (yeilding 6/5/5 internally) and the troughs after the q=1 frames went away.