Where can I read more about AQ?

[microchip8]

I'm not even sure what AQ modes above 1 are trying to "fix". If they all look subjectively slightly worse than AQ1 (I don't care for metrics, I use my eyes to see what looks good and what not to me), what's the point? I've used AQ3 in x264 for a while, but still had issues in dark scenes where it is supposed to "fix" them (eg, banding). Since banding is mostly a problem of precision, you don't need AQ3 when in 10-bit mode as there is nothing to fix/improve. x264 in 10-bit looks stunning and there are no issues in dark scenes. Sadly, 10-bit H.264 is very poorly supported so I use 8-bit when using x264 just to be compatible with as many as possible. Adding a bit of dither (eg, noise) to the encode works better than what AQ3 is supposed to fix.

[GeoffreyA]

If the problem is disproportionate loss between small and big coefficients, AQ mode 1 should solve it. What the auto-variance modes are doing is not clear. I also used AQ3 when encoding x264, and yes, 10-bit x264 is ill supported, making it safer to use 10-bit HEVC or AV1 from a compatibility point of view.

SVT-AV1's implementation of some these things seems more complete and elegant. Would be interesting if it could be ported into x265. They seem to be basing it off x264's mode 1, lending further credence to AQ1's being on the right track.

[Z2697]

There's clear comments in x264 code that says AQ2 is/was experimental and is created for (or coincidentally have?) better SSIM. I don't know very much of x264 history.

[Z2697]

Originally Posted by GeoffreyA

If the problem is disproportionate loss between small and big coefficients, AQ mode 1 should solve it. What the auto-variance modes are doing is not clear. I also used AQ3 when encoding x264, and yes, 10-bit x264 is ill supported, making it safer to use 10-bit HEVC or AV1 from a compatibility point of view.

SVT-AV1's implementation of some these things seems more complete and elegant. Would be interesting if it could be ported into x265. They seem to be basing it off x264's mode 1, lending further credence to AQ1's being on the right track.

I'm not very familiar with SVT-AV1 code, but this function seems to be selecting 3 subblocks (8x8) and calculating weighted average of their variance to represent a superblock(64x64)'s variance, while in x264 the AQ is performed on each marcoblock (16x16), and in x265 depending on the qgsize, each 8x8 (please don't) or 16x16 blocks, the qp adjustments gets averaged up for larger CUs/QGs.

So the SVT-AV1 algorithm seems to be operating at a coarser level. Not saying how it will end up performing like though.
To bad that they had the name "AQ" for a different thing