vs_AQVisualizer - A Tool to Visualize x264 AQ results

[Chrysoberyl]

Hi everyone! It's my first time posting in the forum. Glad to see there are still lots of ppl discussing vs and encoders here.

As we know, adaptive quantization (AQ) is a very important feature for modern video encoders. However, the difference between different modes and the effect of aq-strength are only described qualitatively if not vaguely, and different sources state contradictorily. For example, some sources suggested that there is temporal calculation in AQ process, but actually AQ is completely spatial in the source code.

In order to demystify this process, I implemented the AQ algorithm from x264 source code in python, and integrated it with Vapoursynth to develop vs_AQVisualizer. The purpose of this tool - vs_AQVisualizer, is to provide user a straightforward results of all aspects of AQ, and possibly comparison between the modes as well as aq-strengths.

Please check out the early view version of the script here: https://github.com/Khrysoberyl/vs_AQVisualizer

Currently the doc is very primitive and the script only supports YUV420P8 and YUV444P8. But I'm still actively working on the script to optimize its speed, expand support for 10-bit clips and write a better documentation. Stay tuned! Feel free to send me your feelings and suggestions!

=============================
UPDATED On 02/08/2026
=============================
The new release optimized the speed, now it should run at 30-40 fps for 1080p clips!
8-16 bit formats are supported now, but only for YUV420 and YUV444 formats.
Reconstructed code to simplify the functions, removing sluggish python loops and use more vectorized numpy calculations. Less straightforward though.
A faster version v0.2.0 is also released apart from the stable v1.0.0 version. The matplotlin dependency is removed and a custom colormap rendering function is used. It's ~10% faster but cannot reproduce bit-identical results. Some pixel values would deviate 1-2.


A few examples:
https://raw.githubusercontent.com/Khrysoberyl/vs_AQVisualizer/main/docs/src2.png
https://raw.githubusercontent.com/Khrysoberyl/vs_AQVisualizer/main/docs/aqmode1_2.png

https://raw.githubusercontent.com/Khrysoberyl/vs_AQVisualizer/main/docs/src3.png
https://raw.githubusercontent.com/Khrysoberyl/vs_AQVisualizer/main/docs/aqmode2_3.png

[GeoffreyA]

Welcome to Doom9, and good work on the script. It could indeed be useful when trying to visualise exactly what AQ is doing, particularly modes two and above where it's not clear what the code is attempting (at least to me and apart from AQ3's low-luminance bias). Recently, there was discussion (https://forum.doom9.org/showthread.php?t=186641) on this topic.

I gave your script a go. If I understand correctly, colour represents the amount of energy in that macroblock. For the difference map, does colour represent energy or the quantiser difference?

Also, x265 has doubtful changes to AQ modes two and above. Would it be feasible to implement some of them?

[Chrysoberyl]

Welcome to Doom9, and good work on the script. It could indeed be useful when trying to visualise exactly what AQ is doing, particularly modes two and above where it's not clear what the code is attempting (at least to me and apart from AQ3's low-luminance bias). Recently, there was discussion (https://forum.doom9.org/showthread.php?t=186641) on this topic.

I gave your script a go. If I understand correctly, colour represents the amount of energy in that macroblock. For the difference map, does colour represent energy or the quantiser difference?

Also, x265 has doubtful changes to AQ modes two and above. Would it be feasible to implement some of them?

Thank your for the reply! Only in aq_mode = 0 mode, the color represents AC energy (basically variance of the macroblock mutiplies sample number, i.e. 256 or 64) . In all the other modes, the color represents the adjustment of the quantization parameters (Δqp), or difference of Δqp between different aq-modes in x264. Δqp has a complicated non-linear relationship towards AC energy.

The spatial part of x265 AQ is essentially the same as x264, apart from a few constants. However, it is realized on the coding unit (CU) level, which is variable and can be smaller/bigger than the fixed 16×16 macroblocks. Therefore the exact duplication would require getting the CU partition from x265. Maybe I'll try to read it from the analysis file of x265. The temporal part of x265 AQ is basically CU-Tree, which in my understanding is x265 version of MB-Tree.

Besides, x265 also has experimental motion-adaptive AQ, though disabled by default. This feature apparently needs information from motion estimation and will be too complicated to implement with python. Hevc-aq is also experimental and I'm not very familiar with it.

Given the similarities of x264 and x265 spatial AQ algorithm, they should give very close results qualitatively, if we ignore the macroblock/CU difference. There is an aq-mode 4 for x265, which seems to lower the quality for strong edges even more compared to aq-mode 2. I'll consider adding it to the script in the future too.

[GeoffreyA]

Thanks for the explanation. It was a bit confused about the colours.

Yes, the AQ algorithms of x264 and x265 aren't that different qualitatively, so it won't be worth development time, except for AQ4, which is somewhat different.

[MasterNobody]

Please check out the early view version of the script here: https://github.com/Khrysoberyl/vs_AQVisualizer


Hi. `bias_strength` variable should only be used in bias part of aq-mode=3, all other calculations of `strength_1` / `strength_2` / `strength_3` should depend only on raw `aq_strength`. That is because `bias_strength` is independent from `aq_strength` and can be changed (in theory) independently. It was made to be equal `aq_strength` only to not create one more option.

[Chrysoberyl]

Hi. `bias_strength` variable should only be used in bias part of aq-mode=3, all other calculations of `strength_1` / `strength_2` / `strength_3` should depend only on raw `aq_strength`. That is because `bias_strength` is independent from `aq_strength` and can be changed (in theory) independently. It was made to be equal `aq_strength` only to not create one more option.

Hi, thanks for your reply! However, I cannot agree with your statement. I admit that this is a bit confusing, but if we look at the source code ratecontrol.c, we can see how strength and bias-strength are defined and related to each other.

Apart from the initialization of both variables, they are first assigned in line 370 and line 372:
strength = h->param.rc.f_aq_strength * avg_adj;
bias_strength = h->param.rc.f_aq_strength;

if aq-mode = 1, then strength is adjusted slightly differently in line 375:
strength = h->param.rc.f_aq_strength * 1.0397f;

Apparently, if we treath->param.rc.f_aq_strength as user input aq-strength (which should be), then bias_strength is this original value, while strength is scaled to the average qp adjustment of all the MBs in the frame. They all originate from the user-defined aq-strength parameter though.

Later in line 385 (aq-mode=3), 390 (aq-mode=2) and 395 (aq-mode=1), these two variables are used as they are:
qp_adj = strength * (qp_adj - avg_adj) + bias_strength * (1.f - 14.f / (qp_adj * qp_adj));
qp_adj = strength * (qp_adj - avg_adj);
qp_adj = strength * (x264_log2( X264_MAX(energy, 1) ) - (14.427f + 2*(BIT_DEPTH-8)));

Indeed, bias_strength is only used in aq-mode=3. But according to our previous definitions, here strength is the "biased" aq-strength and bias_strength is the original, user-defined aq-strength.

I have no idea why x264 would name these variables in such a misleading way, but it is what it is. Besides, at least in the official x264 encoder, I don't think you can change bias_strength as a parameter directly because it is not listed as a CLI parameter. It is possible that they may intend to use a separate parameter for it though.

[MasterNobody]

Hi, thanks for your reply! However, I cannot agree with your statement. I admit that this is a bit confusing, but if we look at the source code ratecontrol.c, we can see how strength and bias-strength are defined and related to each other.

Apart from the initialization of both variables, they are first assigned in line 370 and line 372:
strength = h->param.rc.f_aq_strength * avg_adj;
bias_strength = h->param.rc.f_aq_strength;

if aq-mode = 1, then strength is adjusted slightly differently in line 375:
strength = h->param.rc.f_aq_strength * 1.0397f;

Apparently, if we treath->param.rc.f_aq_strength as user input aq-strength (which should be), then bias_strength is this original value, while strength is scaled to the average qp adjustment of all the MBs in the frame. They all originate from the user-defined aq-strength parameter though.

Later in line 385 (aq-mode=3), 390 (aq-mode=2) and 395 (aq-mode=1), these two variables are used as they are:
qp_adj = strength * (qp_adj - avg_adj) + bias_strength * (1.f - 14.f / (qp_adj * qp_adj));
qp_adj = strength * (qp_adj - avg_adj);
qp_adj = strength * (x264_log2( X264_MAX(energy, 1) ) - (14.427f + 2*(BIT_DEPTH-8)));

Indeed, bias_strength is only used in aq-mode=3. But according to our previous definitions, here strength is the "biased" aq-strength and bias_strength is the original, user-defined aq-strength.

I have no idea why x264 would name these variables in such a misleading way, but it is what it is. Besides, at least in the official x264 encoder, I don't think you can change bias_strength as a parameter directly because it is not listed as a CLI parameter. It is possible that they may intend to use a separate parameter for it though.

I just meant that `strength` (lines 370 (https://code.videolan.org/videolan/x264/-/blob/master/encoder/ratecontrol.c?ref_type=heads#L370) and 375 (https://code.videolan.org/videolan/x264/-/blob/master/encoder/ratecontrol.c?ref_type=heads#L375)) and `bias_strength` (line 372 (https://code.videolan.org/videolan/x264/-/blob/master/encoder/ratecontrol.c?ref_type=heads#L372)) are different variables.
And `bias_strength` is only used in the aq-mode=3 part (line 385 (https://code.videolan.org/videolan/x264/-/blob/master/encoder/ratecontrol.c?ref_type=heads#L385)). In all other cases, `strength` is used (lines 390 (https://code.videolan.org/videolan/x264/-/blob/master/encoder/ratecontrol.c?ref_type=heads#L390) and 395 (https://code.videolan.org/videolan/x264/-/blob/master/encoder/ratecontrol.c?ref_type=heads#L395)).
The fact that `strength` is the product of `h->param.rc.f_aq_strength * avg_adj` doesn't make it `bias_strength` — it's still just `strength`.
If `bias_strength` is zero, then aq-mode=3 should produce the same result as aq-mode=2.
As for why there wasn't a separate parameter for `bias_strength` in x264 (and it's set equal to `bias_strength = h->param.rc.f_aq_strength;`), I wrote about it in the previous post.

All I wanted to correct was your calculation of strength_1 / strength_2 / strength_3 (lines 106 (https://github.com/Khrysoberyl/vs_AQVisualizer/blob/main/AQAnalyzer.py#L106), 112 (https://github.com/Khrysoberyl/vs_AQVisualizer/blob/main/AQAnalyzer.py#L112), 117 (https://github.com/Khrysoberyl/vs_AQVisualizer/blob/main/AQAnalyzer.py#L117)). They should be:

strength_1 = aq_strength * 1.0397
strength_2 = aq_strength * avg_adj
strength_3 = aq_strength * avg_adj

[Z2697]

The result is the same though, unless the bias_strength is indeed controlled separately. (that option does exist in some mods)