Sizes with crf n and crf n+1

[martinfrombern]

I know that theoreticaly bitrate of crf n should be 12.5% higher than the bitrate of crf n+1.

However, with the source I've just worked with (720p)
crf 18 -> 4603 kbps
crf 17 -> 5559 kbps
which is 20.76% increase.

Also, with the source I've worked few weeks ago (1080p)...
crf 19 -> 11256 kbps
crf 18 -> 15193 kbps
which equals to 34.97% increase.

So what's the explanation for increases such higher than theoretically they should be? Is this normal?

[akupenguin]

That theoretical calculation assumes that the distribution of dct coefficient magnitudes is Laplacian. If you divide each sample of a Laplacian by a constant (i.e. change the quantizer), then you get the same distribution as dividing the number of samples of each magnitude by a constant, which likewise divides the number of bits spent on nonzero dct coefs. However, the second constant is not necessarily the same as the first. 12% assumes that they are the same, which is a decent approximation for some videos and not for others.
If the distribution isn't Laplacian, then you can get not only other factors, but nonlinear scaling. If your content has, say, a bunch of noise at one particular amplitude, then the delta bitrate across whatever quantizer decimates that noise is rather larger than you would otherwise get. And then there's the bits spent on zero coefs, which don't always cluster by the same amount. And the macroblock mode bits, which scale by some other relation entirely, and affect the accuracy of inter/intra prediction (and thus distribution of coefs) as they do so.

[Snowknight26]

http://i29.tinypic.com/2va1f75.jpg
http://i29.tinypic.com/2vjzr40.jpg
You'd think the graph of the (red or green) line would be similar to that of a standard exponential density function, but throw in a non-integral value..

[akupenguin]

Wiggles in the mbtree=0,qcomp=1 line make sense. QP is an integer, so fractional CRF can only produce intermediate bitrate if there's already some reasonably-uniform distribution of fractional QP offsets for it to be added to. qcomp!=1 is one source of fractional target-QPs. AQ is another. MB-tree replaces qcomp, and I would have thought it would produce plenty of fractions, but maybe the distribution of inter prediction ratios is skewed; and it doesn't touch B-frames.

How we could reduce this effect:
* Separate lambda from QP. Isn't perfect, and only really helps if you use trellis quant. libavcodec does this.
* Dither QPs. Wastes bits on dquant if you do it within one frame. Has nonlinear effects: incrementing the QP of one block saves bits locally, but if the next block is lower QP then you just end up moving most of the bits there, not reducing overall bitrate much. Xvid does this.