How does AutoGK calculate compressibility percentage?

[orion44]

I would like to learn how to do compressibility calculation manually.

From what info I could find on Doom9, AutoGK performs a constant quality pass at Q2 quantizer on 5% of the video,
but I couldn't find the exact formulas that it uses.

How exactly does AutoGK calculate compressibility percentage?

[hello_hello]

If I remember correctly it's just a percentage, so if Q2 equals 100% and (based on the sections of video it tests) the total video size at Q2 would be 1000MB, then a file size of 750MB would be 75% quality, which is what it aims for, from memory (or it might be between 70% and 100%). That part would be easy enough to test if you ran a single pass encode at 100% quality and then again at 75% quality (same resolution each time).

What I don't think I've ever known is exactly how it calculates a new resolution after the compression test if need be, when it's in 2 pass auto mode, in order to achieve a 70% to 75% quality. It's possibly a simple formula but I don't know what it is. You could probably ask the same question in the StaxRip thread because StaxRip can (or at least it used to) run a compression test and I think it updates the expected quality when you change the resizing. It's been a while since I've used it.