Where can I read more about AQ? - Page 3

microchip8 · 10th January 2026, 18:12

I'm not even sure what AQ modes above 1 are trying to "fix". If they all look subjectively slightly worse than AQ1 (I don't care for metrics, I use my eyes to see what looks good and what not to me), what's the point? I've used AQ3 in x264 for a while, but still had issues in dark scenes where it is supposed to "fix" them (eg, banding). Since banding is mostly a problem of precision, you don't need AQ3 when in 10-bit mode as there is nothing to fix/improve. x264 in 10-bit looks stunning and there are no issues in dark scenes. Sadly, 10-bit H.264 is very poorly supported so I use 8-bit when using x264 just to be compatible with as many as possible. Adding a bit of dither (eg, noise) to the encode works better than what AQ3 is supposed to fix.

GeoffreyA · 11th January 2026, 08:03

If the problem is disproportionate loss between small and big coefficients, AQ mode 1 should solve it. What the auto-variance modes are doing is not clear. I also used AQ3 when encoding x264, and yes, 10-bit x264 is ill supported, making it safer to use 10-bit HEVC or AV1 from a compatibility point of view.

SVT-AV1's implementation of some these things seems more complete and elegant. Would be interesting if it could be ported into x265. They seem to be basing it off x264's mode 1, lending further credence to AQ1's being on the right track.

Z2697 · 11th January 2026, 10:59

There's clear comments in x264 code that says AQ2 is/was experimental and is created for (or coincidentally have?) better SSIM. I don't know very much of x264 history.

Z2697 · 11th January 2026, 11:14

Quote:

Originally Posted by GeoffreyA

If the problem is disproportionate loss between small and big coefficients, AQ mode 1 should solve it. What the auto-variance modes are doing is not clear. I also used AQ3 when encoding x264, and yes, 10-bit x264 is ill supported, making it safer to use 10-bit HEVC or AV1 from a compatibility point of view.

SVT-AV1's implementation of some these things seems more complete and elegant. Would be interesting if it could be ported into x265. They seem to be basing it off x264's mode 1, lending further credence to AQ1's being on the right track.

I'm not very familiar with SVT-AV1 code, but this function seems to be selecting 3 subblocks (8x8) and calculating weighted average of their variance to represent a superblock(64x64)'s variance, while in x264 the AQ is performed on each marcoblock (16x16), and in x265 depending on the qgsize, each 8x8 (please don't) or 16x16 blocks, the qp adjustments gets averaged up for larger CUs/QGs.

So the SVT-AV1 algorithm seems to be operating at a coarser level. Not saying how it will end up performing like though.
It does have some info about all 64 subblocks in a superblock, because the variance values are first sorted.
Too bad that they had the name "AQ" for a different thing

Related, but different.

GeoffreyA · Today, 18:26

With AV1, I think it's hard to tell if an algorithm is better: it might be, but then the picture is confounded with other parts of the AV1 pipeline.

10th January 2026, 18:12	#41 \| Link
microchip8 ffx264/ffhevc author Join Date: May 2007 Location: /dev/video0 Posts: 2,017	I'm not even sure what AQ modes above 1 are trying to "fix". If they all look subjectively slightly worse than AQ1 (I don't care for metrics, I use my eyes to see what looks good and what not to me), what's the point? I've used AQ3 in x264 for a while, but still had issues in dark scenes where it is supposed to "fix" them (eg, banding). Since banding is mostly a problem of precision, you don't need AQ3 when in 10-bit mode as there is nothing to fix/improve. x264 in 10-bit looks stunning and there are no issues in dark scenes. Sadly, 10-bit H.264 is very poorly supported so I use 8-bit when using x264 just to be compatible with as many as possible. Adding a bit of dither (eg, noise) to the encode works better than what AQ3 is supposed to fix. __________________ The Cyberpunk Database \|\| NTP & NTS Time Server \|\| Mirror Services ffhevc \|\| ffx264 \|\| ffxvid

11th January 2026, 08:03	#42 \| Link
GeoffreyA Registered User Join Date: Jun 2024 Location: South Africa Posts: 606	If the problem is disproportionate loss between small and big coefficients, AQ mode 1 should solve it. What the auto-variance modes are doing is not clear. I also used AQ3 when encoding x264, and yes, 10-bit x264 is ill supported, making it safer to use 10-bit HEVC or AV1 from a compatibility point of view. SVT-AV1's implementation of some these things seems more complete and elegant. Would be interesting if it could be ported into x265. They seem to be basing it off x264's mode 1, lending further credence to AQ1's being on the right track. Last edited by GeoffreyA; 11th January 2026 at 08:29.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

11th January 2026, 10:59	#43 \| Link
Z2697 Registered User Join Date: Aug 2024 Location: Between my two ears Posts: 858	There's clear comments in x264 code that says AQ2 is/was experimental and is created for (or coincidentally have?) better SSIM. I don't know very much of x264 history.

Today, 18:26	#45 \| Link
GeoffreyA Registered User Join Date: Jun 2024 Location: South Africa Posts: 606	With AV1, I think it's hard to tell if an algorithm is better: it might be, but then the picture is confounded with other parts of the AV1 pipeline.