Where can I read more about AQ?

[Lan4]

Where can I read more about AQ and its modes in the X264 codec?

[microchip8]

http://www.chaneru.com/Roku/HLS/X264_Settings.htm

and for x265 - https://x265.readthedocs.io/en/master/cli.html

[Lan4]

Thanks for the reply, but the links don't provide a detailed description of AQ's functions. In fact, there's no description at all, just a list of functions.

[microchip8]

I could be wrong, but I don't think there's extensive description on how each AQ mode works. Read the code?

[Selur]

https://mailman.videolan.org/pipermail/x264-devel/2012-July/009403.html
https://huyunf.github.io/blogs/2017/12/06/x264_adaptive_quant/
might be interesting

[microchip8]

Good find, Selur! Especially second link!

[Lan4]

Selur: Thanks for the original data.
I still have doubts about how AQ works, because other descriptions have given different information. I'll share my thoughts. By default, the quantizer transfers data too strongly from monotonic regions to complex regions. AQ constrains this data transfer. Therefore, if there are few monotonic regions, weak AQ constraint is required, and therefore the AQ strength should be set to a small value less than 1. If there are many monotonic regions, strong AQ constraint is required, and therefore a large AQ strength greater than 1 should be set.
AQ mode=2 is two-step constraint, for more accurate work, in short.
Is this correct, or can you correct me?

[hello_hello]

The x264 Animation preset sets AQ strength to 0.6 by default.
For the Grain tuning it's 0.5.
For the Film tuning (or no tuning) it's 1.0.
For the Still Image tuning it's 1.2.

Based on that, and the explanation on the x264 wiki, I've always understood that when there's less variation in detail throughout the picture, the AQ strength can be lower than when there's a lot of variation.

http://www.chaneru.com/Roku/HLS/X264_Settings.htm#aq-strength
Adaptive Quantization Strength
Default: 1.0
Sets the strength of AQ bias towards low detail ('flat') macroblocks.

I vaguely remember playing around with the AQ strength for animation years ago and deciding to keep it at 1.0, because if I remember correctly, there was more likely to be banding in gradients when using lower AQ strengths. The background in this image is the sort of thing I'm referring to.

https://static.wikia.nocookie.net/familyguy/images/0/0c/Joyce_and_Tom.png

[GeoffreyA]

I stand to be corrected, but AQ does at the frame level what DCT and quantisation do within a macroblock. As DCT groups coefficients by low and high frequencies, quantisation reducing the latter more aggressively, AQ redistributes bits from more complex (HF) macroblocks to flatter (LF) ones by lowering quantisers, in inverse proportion to AQ strength. In other words, the frame-level quantiser varies according to each macroblock's detail. Since the frame's budget is constrained, increasing the quality of one region lowers that of others.

AQ mode 3 performs this bias towards dark as well as flatter blocks.

[Z2697]

I still have doubts about how AQ works, because other descriptions have given different information.

Let's not forget higher QP = lower quality

[Z2697]

variance is different than frequency though, better distinguish them better. (my bad english)

[GeoffreyA]

variance is different than frequency though, better distinguish them better. (my bad english)

Would variance be the right term for macroblock-to-macroblock quantiser variation? The terminology is a bit puzzling.

[Z2697]

Would variance be the right term for macroblock-to-macroblock quantiser variation? The terminology is a bit puzzling.

The variance in this context is the "mathematical variance (https://en.wikipedia.org/wiki/Variance)", not just the fact that there's variation.

[GeoffreyA]

The variance in this context is the "mathematical variance (https://en.wikipedia.org/wiki/Variance)", not just the fact that there's variation.

I see. So it appears to be the deviation from the mean of a set of values.

[Z2697]

Sidenote, some time ago I said the "recent" SVT-AV1 variance-boost is similar to AQ1, just from the names and general idea in the showcase, but I haven't take a deeper look.
It does have more dials, so it could be more sophiscated. Though AQ1 (x264/5) is simple and elegant and works very well.

Anyways, whatever it is, we didn't have any equivalent of x264 AQ in SVT-AV1 before it, is what I wanted to say. I could be wrong, the "aq-mode" in SVT-AV1 and libaom does have "variance" in one of its options, but if it's the AQ we familiar with, why would variance-boost be needed? (and it's "mode 1", but "crf" requires "mode 2")
(me being lazy and don't want to read av1 code :o)

[GeoffreyA]

Simplicity tends to be best; and Dark Shikari was touched by genius, coming up with and implementing these techniques, not to mention mbtree, which takes a similar concept in the temporal direction.

It's a nightmare trying to decipher SVT-AV1's options. Certainly, we haven't seen this level of inspiration and ingenuity since x264, but it's too confusing; and if one encodes something, there's always doubt: were the settings right, would it handle the range of content well, or did we use the right fork?

Didn't variance-boost work analogously to AQ3, helping dark macroblocks?

[Z2697]

The essence of x264 AQ1 is just like, 3 or 4 lines of code (or even 2), depending on whether you think the strength multiplied with constant and the loop are lines of code or not.

strength = h->param.rc.f_aq_strength * 1.0397f;

for each macroblock {
uint32_t energy = ac_energy_mb( h, mb_x, mb_y, frame );
qp_adj = strength * (x264_log2( X264_MAX(energy, 1) ) - (14.427f + 2*(BIT_DEPTH-8)));
}


As you can see the more "ac energy" / variance / contrast a block has, the higher the QP is added to it, with a negative shift to balance the adjustments.

(ac_energy_mb function does not return the actual variance, it returns the value without the division by (n-1) at the end, if I read the code correctly)

[GeoffreyA]

It's a surprisingly straightforward method. So, the QP adjustment is directly proportional to the energy of each macroblock.

[Z2697]

x265 changed AQ2 "constant" from 14 to 11 and "exponent" from 0.125 to 0.1 for claimed "better bit distribution".
According to my undisclosed test this is the main cause of "grain smearing" (as microchip8 describes it).
But I still prefer AQ1 even if the change is reverted. :o


commit b72e2aef5a7fc445839e647a564970880333681d
Author: Santhoshini Sekar <santhoshini@multicorewareinc.com>
Date: Wed May 21 12:04:52 2014 +0530

aq: fine tune aq logic to distribute bits even better

[GeoffreyA]

Well, one gets the impression that lots of adjustments in x265 were not of x264's calibre. I think the lower constant should result in higher QPs.

[Z2697]

Well, one gets the impression that lots of adjustments in x265 were not of x264's calibre. I think the lower constant should result in higher QPs.

The qp_adj's get raised to a smaller exponent though.
(Is "raised" the right term for an exponent smaller than 1?)

And AQ2 is
1) calculating qp_adj differently ( qp_adj = pow(energy * bit_depth_correction + 1, 0.1); )
2) averaging whole frame qp_adj and qp_adj_pow2
3) doing some calculation based on avg_adj, avg_adj_pow2 and the constant, as the final "average" ( avg_adj = avg_adj - 0.5f * (avg_adj_pow2 - modeTwoConst) / avg_adj; )
4) subtracting "average" from per-block qp_adj
5) multiplying by strength

My brain is too rotten to comprehend what changing two values does to this level of complex operation.
(Nor do I feel I must know...)

[MasterNobody]

The qp_adj's get raised to a smaller exponent though.
(Is "raised" the right term for an exponent smaller than 1?)

And AQ2 is
1) calculating qp_adj differently ( qp_adj = pow(energy * bit_depth_correction + 1, 0.1); )
2) averaging whole frame qp_adj and qp_adj_pow2
3) doing some calculation based on avg_adj, avg_adj_pow2 and the constant, as the final "average" ( avg_adj = avg_adj - 0.5f * (avg_adj_pow2 - modeTwoConst) / avg_adj; )
4) subtracting "average" from per-block qp_adj
5) multiplying by strength

My brain is too rotten to comprehend what changing two values does to this level of complex operation.
(Nor do I feel I must know...)
Here are the graphs for different modes of x264 AQ 1/2/3 and modified x265 AQ 2
https://i.ibb.co/x81BHJqC/2025-11-21-02-18-17.png (https://www.wolframalpha.com/input?i=plot+%28log2%28x%29+-+14.427%29+*+1.0397%2C+x%5E0.125+*+%280.5+*+%28x%5E0.25+-+14%29+%2F+x%5E0.125%29%2C+x%5E0.125+*+%280.5+*+%28x%5E0.25+-+14%29+%2F+x%5E0.125%29+%2B+%281+-+14+%2F+x%5E0.25%29%2C+x%5E0.1+*+%280.5+*+%28x%5E0.2+-+11%29+%2F+x%5E0.1%29%2C+x%3D1..100000)

[GeoffreyA]

(Is "raised" the right term for an exponent smaller than 1?)

I would think so. There's a mathematical generalisation to that language. Just as we "add" a negative number.


And AQ2 is
1) calculating qp_adj differently ( qp_adj = pow(energy * bit_depth_correction + 1, 0.1); )
2) averaging whole frame qp_adj and qp_adj_pow2
3) doing some calculation based on avg_adj, avg_adj_pow2 and the constant, as the final "average" ( avg_adj = avg_adj - 0.5f * (avg_adj_pow2 - modeTwoConst) / avg_adj; )
4) subtracting "average" from per-block qp_adj
5) multiplying by strength

My brain is too rotten to comprehend what changing two values does to this level of complex operation.
(Nor do I feel I must know...)

Now, this is too much for the brain. I give up :)

[hello_hello]

I thought understanding the idea behind AQ was relatively straight forward... higher strengths increase the bias towards low detail macro blocks.... and I assumed for a fixed bitrate higher strengths must rob Peter to pay Paul... macro blocks with a lot of detail use less bits so low detail macro blocks can use more.... or something like that.

But what about CRF encoding? For standard "film" content, higher AQ strengths invariably increase the bit-rate, and the bitrate is higher than with AQ disabled, so does that mean low detail macro blocks are using more bits, but not at the expense of high detail macro blocks?

I'd also not compared different AQ strengths with AQ disabled before, so I'm not sure I understand why enabling AQ can decrease the bitrate for animation with large flat areas of a single color (such as the Simpsons or Family Guy), or why for x264 an AQ strength of 1.0 can result in a lower bitrate than an AQ strength of 0.5.

Each of the test encodes below used my usual settings (close to the slow preset) and 3000 1080p frames were encoded (1440x1080 for animation).

Somewhat grainy film, x265 CRF 20
AQ Mode / Strength / Bitrate (kb/s)

0 0 2981

1 0.5 4360
1 1.0 5977
1 1.5 7843

2 0.5 4130
2 1.0 5452
2 1.5 6192


Denoised film, x265 CRF 20
AQ Mode / Strength / Bitrate (kb/s)

0 0 1584

1 0.5 1934
1 1.0 2693
1 1.5 3957

2 0.5 1901
2 1.0 2330
2 1.5 2910


Animation, x265 CRF 20
AQ Mode / Strength / Bitrate (kb/s)

0 0 1303

1 0.5 1247
1 1.0 1272
1 1.5 1423

2 0.5 1330
2 1.0 1366
2 1.5 1416


Animation, x264 CRF 20 (tune film)
AQ Mode / Strength / Bitrate (kb/s)

0 0 1725

1 0.5 1545
1 1.0 1492
1 1.5 1639

2 0.5 1461
2 1.0 1279
2 1.5 1188

[GeoffreyA]

It's puzzling to answer because we have the conventional understanding that bits are moved from high- to low-detail blocks. But if we look at the code that Z2967 posted, the per-block quantiser is being adjusted based on energy and aq-strength, and a log function makes the calculation non-linear. How that maps to bitrate, as quantisers rise or drop, should vary. But yes, I would think that a fixed-bitrate scenario would rob Peter to pay Paul, and CRF would take the limits off.

Your animation data is a tough one. Could mbtree be the missing link to explain this, the way it interacts with AQ?

[Z2697]

Here are the graphs for different modes of x264 AQ 1/2/3 and modified x265 AQ 2
https://i.ibb.co/x81BHJqC/2025-11-21-02-18-17.png (https://www.wolframalpha.com/input?i=plot+%28log2%28x%29+-+14.427%29+*+1.0397%2C+x%5E0.125+*+%280.5+*+%28x%5E0.25+-+14%29+%2F+x%5E0.125%29%2C+x%5E0.125+*+%280.5+*+%28x%5E0.25+-+14%29+%2F+x%5E0.125%29+%2B+%281+-+14+%2F+x%5E0.25%29%2C+x%5E0.1+*+%280.5+*+%28x%5E0.2+-+11%29+%2F+x%5E0.1%29%2C+x%3D1..100000)

I don't think this is entirely correct, the plot suggests that x265 AQ2 should give anything a negative QP adjustment, which is certainly not the case...

[MasterNobody]

I don't think this is entirely correct, the plot suggests that x265 AQ2 should give anything a negative QP adjustment, which is certainly not the case...
It switches to a positive offset at a significantly higher energy value (around 161051). Most likely, they simply didn't properly normalize the energy for different CU/QG sizes relative to constant 16x16 in x264, and decided to simply optimize for larger blocks, say 32x32 (and neglected the other sizes). Or they always calculate the 10-bit version of energy instead of the 8-bit one (to which x264 adjusts the value, even for 10-bit).
https://i.ibb.co/tpLBXvt4/2025-11-24-01-47-02.png (https://www.wolframalpha.com/input?i=plot+%28log2%28x%29+-+14.427%29+*+1.0397%2C+x%5E0.125+*+%280.5+*+%28x%5E0.25+-+14%29+%2F+x%5E0.125%29%2C+x%5E0.125+*+%280.5+*+%28x%5E0.25+-+14%29+%2F+x%5E0.125%29+%2B+%281+-+14+%2F+x%5E0.25%29%2C+x%5E0.1+*+%280.5+*+%28x%5E0.2+-+11%29+%2F+x%5E0.1%29%2C+x%3D1..1000000)

They only handle the 8x8 case separately, but that doesn't involve energy normalization, they just change the offset constant:
https://i.ibb.co/7JMjC593/2025-11-24-02-02-28.png (https://www.wolframalpha.com/input?i=plot+%28log2%28x%29-14.427%29*1.0397%2C+0.5*%28x%5E0.25-14%29%2C+0.5*%28x%5E0.25-14%29%2B%281-14%2Fx%5E0.25%29%2C+0.5*%28x%5E0.2-11%29%2C+%28log2%28x%29-11.427%29*1.0397%2C+0.5*%28x%5E0.2-8%29%2C+x%3D1..100000)

Normalizing them to 16x16 they will look like this:
https://i.ibb.co/q3JKDg1Y/2025-11-24-02-46-54.png (https://www.wolframalpha.com/input?i=plot+%28log2%28x%29-14.427%29*1.0397%2C+0.5*%28x%5E0.25-14%29%2C+0.5*%28x%5E0.25-14%29%2B%281-14%2Fx%5E0.25%29%2C+0.5*%28%28x*4%29%5E0.2-11%29%2C+%28log2%28x%2F4%29-11.427%29*1.0397%2C+0.5*%28%28x%2F4%29%5E0.2-8%29%2C+x%3D1..100000)

[Z2697]

It switches to a positive offset at a significantly higher energy value (around 161051). Most likely, they simply didn't properly normalize the energy for different CU/QG sizes relative to constant 16x16 in x264, and decided to simply optimize for larger blocks, say 32x32 (and neglected the other sizes). Or they always calculate the 10-bit version of energy instead of the 8-bit one (to which x264 adjusts the value, even for 10-bit).

x265 AQ "treats" blocks like 16x16 when qgsize > 8.
Or I mean, they do it at 16x16 (and then average whatever the size up I guess).
I don't think average step can be ploted nicely in this way, and avg_adj_pow2 is not avg_adj^2, but is the average of qp_adj^2, there's a difference.
The AQ1 curve should be correct for its simplicity.

[MasterNobody]

x265 AQ "treats" blocks like 16x16 when qgsize > 8.
Or I mean, they do it at 16x16 (and then average whatever the size up I guess).
I don't think average step can be ploted nicely in this way, and avg_adj_pow2 is not avg_adj^2, but is the average of qp_adj^2, there's a difference.

Obviously, this is a simplification. The only way to draw AQ2 graph on a 2D plane is to assume that the energy of all 16x16 blocks is the same across the entire frame (or to assume that the entire frame consists of only one 16x16 block). Otherwise, there would be at least one more variable, and we'd have to draw the surface in 3D space.


x - energy of a 16x16 block
mb_avg_adj = x^0.125
mb_avg_adj_pow2 = x^0.125 * x^0.125
= x^0.25
strength = h->param.rc.f_aq_strength * mb_avg_adj
= 1 * x^0.125
= x^0.125
avg_adj = mb_avg_adj - 0.5 * (mb_avg_adj_pow2 - 14) / mb_avg_adj
= x^0.125 - 0.5 * (x^0.25 - 14) / x^0.125
qp_adj = strength * (mb_avg_adj - avg_adj)
= x^0.125 * (x^0.125 - (x^0.125 - 0.5 * (x^0.25 - 14) / x^0.125))
= x^0.125 * x^0.125 - x^0.125 * (x^0.125 - 0.5 * (x^0.25 - 14) / x^0.125)
= x^0.25 - (x^0.125 * x^0.125 - 0.5 * (x^0.25 - 14) / x^0.125 * x^0.125)
= x^0.25 - (x^0.25 - 0.5 * (x^0.25 - 14))
= x^0.25 - x^0.25 + 0.5 * (x^0.25 - 14)
= 0.5 * (x^0.25 - 14)


And as a result we get: qp_adj = 0.5 * (x^0.25 - 14)
And this is the correct AQ2 formula in x264 for comparison with AQ1.

Why they significantly shifted the zero-crossing point in x265 for AQ2 (but not for AQ1), I don't know; I can only speculate. Either it's a bug, or some unknown coefficient that quadrupled the input energy of a 16x16 block compared to x264, and they needed to compensate for this increase.

Or maybe it's just that the power reduction to 0.1 made qp_adj so weak (only -3.63, compared to -6.5 in the x264 version, or even -15 for AQ1) at low energies at the correct offset of 8.25852 (to preserve the zero crossing point) that they decided to shift the whole curve down to get -5.5 there.
https://i.ibb.co/WNFF6ctq/2025-11-25-00-25-55.png (https://www.wolframalpha.com/input?i=plot+log2%28x%29-14.427%29*1.0397%2C+0.5+*+%28x%5E0.25+-+14%29%2C+0.5+*+%28x%5E0.2+-+8.25852%29%2C+0.5+*+%28x%5E0.2+-+11%29%2C+x%3D1..100)
https://i.ibb.co/whwJgg70/2025-11-25-00-24-40.png (https://www.wolframalpha.com/input?i=plot+log2%28x%29-14.427%29*1.0397%2C+0.5+*+%28x%5E0.25+-+14%29%2C+0.5+*+%28x%5E0.2+-+8.25852%29%2C+0.5+*+%28x%5E0.2+-+11%29%2C+x%3D1..100000)

Essentially, these offsets (if it is constant) don't really affect anything; they merely shift everything by a fixed value, meaning they simply change the resulting file size / CRF ratio (which can be compensated for by changing the CRF). In x264, it was adjusted roughly so that the same CRF would yield the same final file size in different AQ modes.

[Z2697]

The commit message clearly says "to distribute bits even better".
So it's not like the "old" constants are not good or degrading...
It's just arbitrarily made decision, I guess.
(the qg-size 8 support was added later than this change)

[GeoffreyA]

What about x265's AQ4, hevc-aq, and mods that have auto-aq? Surely, the latter would be optimal, but the implementation more prone to problems, visible quality fluctuations, etc. Also, if AQ2 and above are problematic owing to their complexity, I imagine porting AQ3's low-luma bias to AQ1 would yield better results for BT.709 content.

[Z2697]

AQ3's dark bias is not easy to just pick out, it requires some AQ2 "pre-calculation", but I did it anyway. I mean, just frankensteined them.
https://github.com/Mr-Z-2697/x265-Yuuki-Asuna/commit/cb1a14a9e4388599e3b69fff2c1cf1127bee7001
https://github.com/Mr-Z-2697/x265-Yuuki-Asuna/commit/9907c908bb4a4adc6fe76967354ff247f5461550

[Z2697]

AQ4 is, AQ2 plus edge angle detection.

If it detects a block with an edge with its angle in 45 ± 15 or 135(45+90) ± 15 degrees range, and it's "final AQ2 qp adjustment" is positive, it adds 0.5 to the strength.
Which means it only lowers the quality (and bit), ever.

I don't know the methametical logic behind it but it seems weird to me.
What if there's a circualr shape across multiple blocks on screen?

[GeoffreyA]

I think the idea is that, if there are more edges in a region, our eyes are less sensitive to the complexity, so chop its bits a bit. (Edit: Or rather, that these "edgy" regions lose less information after quantisation, relative to low-frequency blocks.)

Actually, you made me think: I had been comparing AQ1 and AQ4, and it didn't occur to me that the bitrate reduction, or increase, was coming from AQ2 largely.

[microchip8]

I've tested hevc-aq a bit myself. The end result was it made the picture look horrible from up-close (no stills). I'm sticking to AQ1 and calling it a day.

[GeoffreyA]

I've tested hevc-aq a bit myself. The end result was it made the picture look horrible from up-close (no stills). I'm sticking to AQ1 and calling it a day.

After a detour with AQ4, I've fallen back to AQ1, but it still bugs me that the auto-variance modes get higher scores (I know AQ doesn't translate favourably in terms of metrics). The mode 1 algorithm makes sense: higher energy == more quantisation, and uniform across blocks. Mode 2 is harder to understand, and as averages are at play, I would guess that some blocks may do better and some worse.

[microchip8]

After a detour with AQ4, I've fallen back to AQ1, but it still bugs me that the auto-variance modes get higher scores (I know AQ doesn't translate favourably in terms of metrics). The mode 1 algorithm makes sense: higher energy == more quantisation, and uniform across blocks. Mode 2 is harder to understand, and as averages are at play, I would guess that some blocks may do better and some worse.

All AQ modes but AQ 1 are somewhat "broken" because they're all based on AQ 2 and this mode has been changed IMO for the worse by the x265 devs :(

[GeoffreyA]

All AQ modes but AQ 1 are somewhat "broken" because they're all based on AQ 2 and this mode has been changed IMO for the worse by the x265 devs :(

It's not clear if, or how, the AQ2 algorithm, as posted above, is better than the simpler mode 1, or even needed. According to the graph, it has a higher threshold for more-complex blocks.

Surely, there is some proper or textbook way for this to be done. It seems quite ad-hoc.

[microchip8]

It's not clear if, or how, the AQ2 algorithm, as posted above, is better than the simpler mode 1, or even needed. According to the graph, it has a higher threshold for more-complex blocks.

Surely, there is some proper or textbook way for this to be done. It seems quite ad-hoc.

I've said it in the past. Sometimes, the more simpler algorithm gives the better visual results subjectively.

Even in x264, where AQ2 and above are not "broken" and work as intended, I still prefer AQ1. All modes above 1 give slightly worse picture subjectively but blow up the bitrate, sometimes considerably.

[GeoffreyA]

I've said it in the past. Sometimes, the more simpler algorithm gives the better visual results subjectively.

Even in x264, where AQ2 and above are not "broken" and work as intended, I still prefer AQ1. All modes above 1 give slightly worse picture subjectively but blow up the bitrate, sometimes considerably.

Yes, the better algorithm, or theory, tends to be as complex as it needs to be and no more, in line with Occam's razor.

I've been looking at x264's version of AQ (https://code.videolan.org/videolan/x264/-/blob/master/encoder/ratecontrol.c?ref_type=heads#L349), trying to make sense of it.

/* constants chosen to result in approximately the same overall bitrate as without AQ.
* FIXME: while they're written in 5 significant digits, they're only tuned to 2. */
float strength;
float avg_adj = 0.f;
float bias_strength = 0.f;

if( h->param.rc.i_aq_mode == X264_AQ_AUTOVARIANCE || h->param.rc.i_aq_mode == X264_AQ_AUTOVARIANCE_BIASED )
{
float bit_depth_correction = 1.f / (1 << (2*(BIT_DEPTH-8)));
float avg_adj_pow2 = 0.f;
for( int mb_y = 0; mb_y < h->mb.i_mb_height; mb_y++ )
for( int mb_x = 0; mb_x < h->mb.i_mb_width; mb_x++ )
{
uint32_t energy = ac_energy_mb( h, mb_x, mb_y, frame );
float qp_adj = powf( energy * bit_depth_correction + 1, 0.125f );
frame->f_qp_offset[mb_x + mb_y*h->mb.i_mb_stride] = qp_adj;
avg_adj += qp_adj;
avg_adj_pow2 += qp_adj * qp_adj;
}
avg_adj /= h->mb.i_mb_count;
avg_adj_pow2 /= h->mb.i_mb_count;
strength = h->param.rc.f_aq_strength * avg_adj;
avg_adj = avg_adj - 0.5f * (avg_adj_pow2 - 14.f) / avg_adj;
bias_strength = h->param.rc.f_aq_strength;
}
else
strength = h->param.rc.f_aq_strength * 1.0397f;

for( int mb_y = 0; mb_y < h->mb.i_mb_height; mb_y++ )
for( int mb_x = 0; mb_x < h->mb.i_mb_width; mb_x++ )
{
float qp_adj;
int mb_xy = mb_x + mb_y*h->mb.i_mb_stride;
if( h->param.rc.i_aq_mode == X264_AQ_AUTOVARIANCE_BIASED )
{
qp_adj = frame->f_qp_offset[mb_xy];
qp_adj = strength * (qp_adj - avg_adj) + bias_strength * (1.f - 14.f / (qp_adj * qp_adj));
}
else if( h->param.rc.i_aq_mode == X264_AQ_AUTOVARIANCE )
{
qp_adj = frame->f_qp_offset[mb_xy];
qp_adj = strength * (qp_adj - avg_adj);
}
else
{
uint32_t energy = ac_energy_mb( h, mb_x, mb_y, frame );
qp_adj = strength * (x264_log2( X264_MAX(energy, 1) ) - (14.427f + 2*(BIT_DEPTH-8)));
}


I'm not clever enough but get the feeling that auto-variance could be rewritten with a simpler, more effective approach. From what I can tell, the QP adjustment depends on the average adjustment of the frame. In AQ1, each block is adjusted based on its energy without regard to a frame-level average. So, one takes a more "global" approach; the other, local.

Another thing. AQ3 is supposed to lower quantisers of darker blocks. But, as far as I can tell, the code for X264_AQ_AUTOVARIANCE_BIASED should increase the quality of flatter as well as darker blocks, if I'm not mistaken.

[microchip8]

I'm not even sure what AQ modes above 1 are trying to "fix". If they all look subjectively slightly worse than AQ1 (I don't care for metrics, I use my eyes to see what looks good and what not to me), what's the point? I've used AQ3 in x264 for a while, but still had issues in dark scenes where it is supposed to "fix" them (eg, banding). Since banding is mostly a problem of precision, you don't need AQ3 when in 10-bit mode as there is nothing to fix/improve. x264 in 10-bit looks stunning and there are no issues in dark scenes. Sadly, 10-bit H.264 is very poorly supported so I use 8-bit when using x264 just to be compatible with as many as possible. Adding a bit of dither (eg, noise) to the encode works better than what AQ3 is supposed to fix.

[GeoffreyA]

If the problem is disproportionate loss between small and big coefficients, AQ mode 1 should solve it. What the auto-variance modes are doing is not clear. I also used AQ3 when encoding x264, and yes, 10-bit x264 is ill supported, making it safer to use 10-bit HEVC or AV1 from a compatibility point of view.

SVT-AV1's implementation (https://github.com/juliobbv-p/svt-av1-hdr/blob/836d56b21db64b4d24642cc1d8056305dea56882/Source/Lib/Codec/rc_process.c#L1559) of some these things seems more complete and elegant. Would be interesting if it could be ported into x265. They seem to be basing it off x264's mode 1, lending further credence to AQ1's being on the right track.

[Z2697]

There's clear comments in x264 code that says AQ2 is/was experimental and is created for (or coincidentally have?) better SSIM. I don't know very much of x264 history.

edit: or is it in the commit message? It's hard to remember things these days 'cause the Memory price is too high. ;)

[Z2697]

If the problem is disproportionate loss between small and big coefficients, AQ mode 1 should solve it. What the auto-variance modes are doing is not clear. I also used AQ3 when encoding x264, and yes, 10-bit x264 is ill supported, making it safer to use 10-bit HEVC or AV1 from a compatibility point of view.

SVT-AV1's implementation (https://github.com/juliobbv-p/svt-av1-hdr/blob/836d56b21db64b4d24642cc1d8056305dea56882/Source/Lib/Codec/rc_process.c#L1559) of some these things seems more complete and elegant. Would be interesting if it could be ported into x265. They seem to be basing it off x264's mode 1, lending further credence to AQ1's being on the right track.

I'm not very familiar with SVT-AV1 code, but this function seems to be selecting 3 subblocks (8x8) and calculating weighted average of their variance to represent a superblock(64x64)'s variance, while in x264 the AQ is performed on each marcoblock (16x16), and in x265 depending on the qgsize, each 8x8 (please don't) or 16x16 blocks, the qp adjustments gets averaged up for larger CUs/QGs.

So the SVT-AV1 algorithm seems to be operating at a coarser level. Not saying how it will end up performing like though.
It does have some info about all 64 subblocks in a superblock, because the variance values are first sorted.
Too bad that they had the name "AQ" for a different thing :rolleyes: Related, but different.

[GeoffreyA]

With AV1, I think it's hard to tell if an algorithm is better: it might be, but then the picture is confounded with other parts of the AV1 pipeline.