Where can I read more about AQ?

[Lan4]

Where can I read more about AQ and its modes in the X264 codec?

[microchip8]

http://www.chaneru.com/Roku/HLS/X264_Settings.htm

and for x265 - https://x265.readthedocs.io/en/master/cli.html

[Lan4]

Thanks for the reply, but the links don't provide a detailed description of AQ's functions. In fact, there's no description at all, just a list of functions.

[microchip8]

I could be wrong, but I don't think there's extensive description on how each AQ mode works. Read the code?

[Selur]

https://mailman.videolan.org/piperma...ly/009403.html
https://huyunf.github.io/blogs/2017/...daptive_quant/
might be interesting

[microchip8]

Good find, Selur! Especially second link!

[Lan4]

Selur: Thanks for the original data.
I still have doubts about how AQ works, because other descriptions have given different information. I'll share my thoughts. By default, the quantizer transfers data too strongly from monotonic regions to complex regions. AQ constrains this data transfer. Therefore, if there are few monotonic regions, weak AQ constraint is required, and therefore the AQ strength should be set to a small value less than 1. If there are many monotonic regions, strong AQ constraint is required, and therefore a large AQ strength greater than 1 should be set.
AQ mode=2 is two-step constraint, for more accurate work, in short.
Is this correct, or can you correct me?

[hello_hello]

The x264 Animation preset sets AQ strength to 0.6 by default.
For the Grain tuning it's 0.5.
For the Film tuning (or no tuning) it's 1.0.
For the Still Image tuning it's 1.2.

Based on that, and the explanation on the x264 wiki, I've always understood that when there's less variation in detail throughout the picture, the AQ strength can be lower than when there's a lot of variation.

http://www.chaneru.com/Roku/HLS/X264...tm#aq-strength
Adaptive Quantization Strength
Default: 1.0
Sets the strength of AQ bias towards low detail ('flat') macroblocks.

I vaguely remember playing around with the AQ strength for animation years ago and deciding to keep it at 1.0, because if I remember correctly, there was more likely to be banding in gradients when using lower AQ strengths. The background in this image is the sort of thing I'm referring to.

[GeoffreyA]

I stand to be corrected, but AQ does at the frame level what DCT and quantisation do within a macroblock. As DCT groups coefficients by low and high frequencies, quantisation reducing the latter more aggressively, AQ redistributes bits from more complex (HF) macroblocks to flatter (LF) ones by lowering quantisers, in inverse proportion to AQ strength. In other words, the frame-level quantiser varies according to each macroblock's detail. Since the frame's budget is constrained, increasing the quality of one region lowers that of others.

AQ mode 3 performs this bias towards dark as well as flatter blocks.

[Z2697]

Originally Posted by Lan4

I still have doubts about how AQ works, because other descriptions have given different information.

Let's not forget higher QP = lower quality

[Z2697]

variance is different than frequency though, better distinguish them better. (my bad english)

[GeoffreyA]

Originally Posted by Z2697

variance is different than frequency though, better distinguish them better. (my bad english)

Would variance be the right term for macroblock-to-macroblock quantiser variation? The terminology is a bit puzzling.

[Z2697]

Originally Posted by GeoffreyA

Would variance be the right term for macroblock-to-macroblock quantiser variation? The terminology is a bit puzzling.

The variance in this context is the "mathematical variance", not just the fact that there's variation.

[GeoffreyA]

Originally Posted by Z2697

The variance in this context is the "mathematical variance", not just the fact that there's variation.

I see. So it appears to be the deviation from the mean of a set of values.

[Z2697]

Sidenote, some time ago I said the "recent" SVT-AV1 variance-boost is similar to AQ1, just from the names and general idea in the showcase, but I haven't take a deeper look.
It does have more dials, so it could be more sophiscated. Though AQ1 (x264/5) is simple and elegant and works very well.

Anyways, whatever it is, we didn't have any equivalent of x264 AQ in SVT-AV1 before it, is what I wanted to say. I could be wrong, the "aq-mode" in SVT-AV1 and libaom does have "variance" in one of its options, but if it's the AQ we familiar with, why would variance-boost be needed? (and it's "mode 1", but "crf" requires "mode 2")
(me being lazy and don't want to read av1 code )

[GeoffreyA]

Simplicity tends to be best; and Dark Shikari was touched by genius, coming up with and implementing these techniques, not to mention mbtree, which takes a similar concept in the temporal direction.

It's a nightmare trying to decipher SVT-AV1's options. Certainly, we haven't seen this level of inspiration and ingenuity since x264, but it's too confusing; and if one encodes something, there's always doubt: were the settings right, would it handle the range of content well, or did we use the right fork?

Didn't variance-boost work analogously to AQ3, helping dark macroblocks?

[Z2697]

The essence of x264 AQ1 is just like, 3 or 4 lines of code (or even 2), depending on whether you think the strength multiplied with constant and the loop are lines of code or not.

strength = h->param.rc.f_aq_strength * 1.0397f;

for each macroblock {
uint32_t energy = ac_energy_mb( h, mb_x, mb_y, frame );
qp_adj = strength * (x264_log2( X264_MAX(energy, 1) ) - (14.427f + 2*(BIT_DEPTH-8)));
}

As you can see the more "ac energy" / variance / contrast a block has, the higher the QP is added to it, with a negative shift to balance the adjustments.

(ac_energy_mb function does not return the actual variance, it returns the value without the division by (n-1) at the end, if I read the code correctly)

[GeoffreyA]

It's a surprisingly straightforward method. So, the QP adjustment is directly proportional to the energy of each macroblock.

[Z2697]

x265 changed AQ2 "constant" from 14 to 11 and "exponent" from 0.125 to 0.1 for claimed "better bit distribution".
According to my undisclosed test this is the main cause of "grain smearing" (as microchip8 describes it).
But I still prefer AQ1 even if the change is reverted.

commit b72e2aef5a7fc445839e647a564970880333681d
Author: Santhoshini Sekar <santhoshini@multicorewareinc.com>
Date: Wed May 21 12:04:52 2014 +0530

aq: fine tune aq logic to distribute bits even better

[GeoffreyA]

Well, one gets the impression that lots of adjustments in x265 were not of x264's calibre. I think the lower constant should result in higher QPs.