How is quantizer for each frame calculated in xvid.

[OUTPinged_]

:/

It isnt written anywhere. Some time ago it was mentioned xvid doesnt use gauge-like rate control, so it would be nice if somebody posted here a brief explanation of how quantizer is calculated.

More understanding on codec's work = better rips, as always :-)

For an egzample, a good old post of Waldo himself on Nandub's drf selection(from an old forum):

----------------------------------------------------------------
vV nn
Moderator
Posts: 509
(6/30/01 12:42:38 am)
Re: gauge/br vs drf, pls, explain
--------------------------------------------------------------------------------
Well, let's try in simple words.

To achieve its bitrate control, the codec uses mainly 2 parameters :

* The Bits Reservoir (BR) is simply the number of bits that the codec may use for 5 seconds of video. Obviously, it is directly proportional to the bitrate (R). And a bitrate change is equivalent to a scaling of this variable and all those who are related to it, like the gauge.

* The Gauge (G) is the marker (from 0 up to BR) that the codec uses to 'know' how it is consuming bits. Since Nandub changes the bitrate while encoding, it is more practical to consider it as a %age of the BR.

G is updated after each encoded frame. Let's call F the amount of bits allocated for one single frame at a specific bitrate (divide the bitrate expressed in bits/s by the framerate expressed in frames/s). Now, let's call S the actual size of the encoded frame. Then G is updated like this :

G = G + F - S

Simple, uh? You can easily see that if S>F (if the encoded frame is bigger than what the bitrate prescribes), G gets lower.

How does all this relate to compression levels (CL) ? Well, there's a rather complex algorythm which looks approximately like this :
- look at the gauge
- look at the frame
- look at the bitrate
- look at the previous CL
- chew a bit
- spit a CL
- check it against to min/max
- encode
- update the gauge
(of course, Nandub adds a bit salt here, since it can change many things between frames)

There's a special 'level' of the gauge (35%) at which the codec starts to be nasty with CLs. If the gauge gets lower than this value, the codec starts seriously increasing CLs. And of course, the lower it gets, the greatest the CLs. And naturally, when the gauge is over this 35% mark, the codec tends to use more bits than prescribed by the bitrate.

[Foxer]

ATM the 2pass second pass mode in xvid is quite simple and goes like this.

- Read in 1st pass's framesize.
- Scale it to the movie's target quality.
- Apply curve mod (I've made a new one. It's ready for use and I'm waiting for -h to modify the GUI) and bias/proportional adjustment.
- Cap the curved framesize to a max(either using the current curved framesize or a smooth average). NOTE: This isn't implemented yet.
- Cap it to quant2 framesize so quant1 monsters aren't chosen that often.
- take 1/8 of the current overflow and scale it by framesize / avg framesize so too much/little bitrate won't be given to smaller frames and make sure the scaled overflow isn't too large (ie. framesize > 8x average)
- depending on how large the scaled overflow is, cap it to +- 60% of desired framesize or (only in underflow situations) 60% of the scaled overflow and apply it.
- use a formula to determine the quantizer that should be used to produce the desired framesize.
- if the calculated quant is < 31, if the accumulated decision error for the calculated quant is >= 1.0, use it + 1.

*whew* Winded explanation I know, but it covers all the steps.

[Teegedeck]

Do you think there could be any virtue in the curve-algo taking the 'dynamics' of the unscaled curve into consideration (i.e. a relatively 'flat' curve should need less compression)? ATM we have to figure this out ourselves and I guess not everyone does.

[OUTPinged_]

2Foxer, will there be some sort of motion based curve modulation implemented?


And does overflow equal to deviation in divx3 or is it calculated differently?

2Teegedeck, i remember there was a minor discussion on that kind of algo (or at least a very close one) back in nandub development days, but i dont have that thread backed and maybe the only person who has it is doom9. I do remember though it was considered "could prove usefull" but it didnt made into nandub.

[Foxer]

@Teegedeck
With curve aggression set to low, my curve algo will do that.

@OUTPinged_
MBCM.. If I knew exactly how 'motion' is calculated, I'd do it. heh

Overflow is just the current deviation from the video's current curved size.

[EDIT]
@Teegedeck
Oops. Low aggression does that. Corrected.
Skimmed and thought you meant for more variation in such videos.

[OUTPinged_]

2Foxer:

well i am not the right person to refer to about codec's internals, but in divx3 the calculation process goes like that: (a reply by more competent person than me)

http://forum.doom9.org/showthread.php?s=&threadid=7229&highlight=keyblocks

[Teegedeck]

Low aggression? Humm, I think it's time to download a new build... :)

Thanks, Foxer!

[Foxer]

Low aggression is a setting you can use with the curve algo I made but it's not in the cvs yet. I'm no good with GUI stuff so I'm waiting for -h to modify it.

There are 4 main params along with 3 curve aggression settings. Low distance, high distance, minimum relative quality (has an automatic mode w/ strength), and bonus bias percentage (has an automatic mode)

The distances are used to calculate the framesize thresholds at which the best (low) and worst (high) qualities are applied.

Of course if low distance is > 100% (negative threshold and unreachable), it scales the quality deviation and repositions the quality midpoint so that 100% to min relative quality is still attainable.

The aggression setting determines how quick it is to improve/degrade quality based upon how close the scaled framesize is to the low/high threshold.

@Teegedeck
I tried your max quant = min quant + # with high & low percentages of video content method but it didn't produce very good results with various target qualities & videos with various levels of action and took quite a long time to initialize, so I decided to expand upon curve compression and this is what I've come up with so far.

[OUTPinged_]

heh, more new CC algos, more reasons to toy with stuff.

Foxer, when you will set a final version of your CC options, please scribe a formulas of all the option's correlations somewhere or there will be lots of lame newbie posts (and i will come up first of course :P).

[insert evil laughter here]

I already imagine my fellow rippers to open a codec's tab and see there "curve aggression", "bonus bias percentage", etc =)



I still cant figure out the way to check a frame of xvid stream for its quantizer :/ Any ideas on that?

[Foxer]

Heheheh

The bonus bias percentage is the percentage of the bonus that is applied in a bias manner and the rest in a proportional manner.

The reason I was able to add this (I could add it for regular CC too but with regular CC there can be an overflow. ie 25/15 low/high and that would create a negative bonus and I don't think you'd like it if your low bitrate frames were like quant10 - 15 when the average is supposed to be quant4 - 5) was the way it works. 100% quality is 1.0 and min relative quality is min relative quality% unless low distance is > 100. With this, there is always an underflow and the results need to be scaled up to meet the desired filesize so I decided to let people decide how much of this bonus is to be applied in a bias manner and how much in a proportional manner.

You do need to be careful with this percentage though because even if your low & high distances are large, if the quality midpoint is low, the bonus would be large and if most of it is applied as bias, the low bitrate frames would look great and the high bitrate frames would look quite poor.

[edit]
@OUTPinged_
Hmm.. -h described how to find a frame's quantizer in an earlier thread. Forgot which one it was though or if it was on xvid.org's forum. I do remember that it wasn't a snap to do and that you'd have to look at the previous frame's header and such.

And don't worry.. I'll post exactly how everything works when it's encorporated into the codec or you can always download codec.c at that time and see 'exactly' how everything works :)

[-h]

Heh ok ok :)

I'll be home in about 9 hours or so, I'll upload an ugly hacked interface update with the new CC for people to test, until the *new* new interface is ready.

Then I will sleep.

-h

[Teegedeck]

How nice, everything keeps getting simpler and simpler...!

:) :) :)

It's hard to believe that it all comes down to the decision whether a frame will get compressed at quant 4 or quant 5, in the end...

You put a lot of work into this. Thanks! I'll have a look at it when it's in the CVS.