Questions about Haali's AQ

[TheBashar]

I'm having some trouble with using AQ and how I go about trying to mitigate the problems depends on how AQ works.

My first question is: Does AQ only look for "flat" areas or does it look for "flat dark" areas?

My second question is: In an automated 2-pass mode (in MeGUI-x264 0.2.2.7) do the extra bits allocated to "flat" areas come at the expense of non-flat areas in that frame or from non-flat areas throught the clip ?

My speculation about the second question is that AQ adjustments happen after normal ratecontrol. So, the first pass will use unexpectedly more bits for frames where AQ kicks in. I think the ratecontrol for the 2nd pass will up the Qps (to compensate) across the entire clip. Am I on track?

In case you're interested, the problem I'm seeing is that with AQ enabled with default values (0.5 and 15 I think) very similar blocking problems which AQ attacks so well in dark areas are appearing in caucasian skin-tones.

[unmei]

I'd say it looks for flat areas in general. For example it kicks in daytime sky areas.

For the second one, from the fact that in QC mode it gives lower quantisers to the flat areas but not higher quantisers (than it would without AQ), i derive that in two-pass it will compeantate across the entire clip. Unless the AQ itself works different in the two-pass mode.

[akupenguin]

AQ activates on blocks that are (flat && (dark || blue)).
Yes, any raising of QPs is done by ratecontrol, and is spread evenly across the entire video.

[bond]

afaik the aq gives more bits to the blue areas, will it also give more bits to dark areas?
shouldnt the dark areas normally need less bits?

[akupenguin]

See, many HVS algorithms have this weird idea that dark areas are worthy of less bits than average. They're wrong, and this is why lumimask utterly fails in XviD, DivX, and libavcodec.
It may be true that in a frame containing both dark and light areas, you're less likely to be looking at the dark. But a given numerical magnitude of artifact is much more noticeable on a dark background.
</IMNSHO>

[bond]

someone should checkout atemes aq bit distribution. they are at least very proud of their psy stuff :D

[bond]

ok i now did some tests on AQ regarding how it is able to reduce the blocks in flat areas like walls

i tried three settings:
1) strength 1.0 and sensitivity 10 (the default)
2) strength 1.0 and sensitivity 18
3) strength 1.1 and sensitivity 22 (max settings)

1) shows no effect at all for the walls in the clip i tested (matrix). 2) shows the funny effect that the darker parts of the walls show clearly less blocks than the brighter parts (which look like without aq) 3) shows pretty much the same results as 2) so aq doesnt seem to cover bright flat areas at all

the big downside of aq in any mode tough is that it decreases the level of details in other parts of many frames too, which makes it for me pretty unuseable :(


on the other hand the aq of the ateme encoder makes walls pretty blockfree (similar to x264 aq with higher settings on darker areas), tough it additionally manages to keep the details in the rest of the picture. without aq the ateme encoder also shows blocks in the walls
the in-frame quants shown by ffdshow show that ateme uses lower quants on the walls than the average quant of the frame. ateme uses "zone-like" seperated slightly lower quants on the darker parts of the walls and lower ones on the brighter parts (x264 aq on the other hand shows more of a "quant mix" on the walls)

i therefore assume that x264's aq needs more tuning