What happened to --auto-aq ?

[ShortKatz]

In February there was a nice patch about a new parameter --auto-aq, which automatically decides the AQ Mode (2, 3, 4 or 5) for each frame using its scene statistics.
https://mailman.videolan.org/pipermail/x265-devel/2020-February/012882.html
But this patch never made it into master. Does anybody know more why they did not pursue on that idea?
I quite liked that idea, so I made myself a customized version of HandBrake, that includes this patch. And it gives me nice results, good quality and much smaller files size. So, I don't really understand why this patch never made it into master.

[benwaggoner]

In February there was a nice patch about a new parameter --auto-aq, which automatically decides the AQ Mode (2, 3, 4 or 5) for each frame using its scene statistics.
https://mailman.videolan.org/pipermail/x265-devel/2020-February/012882.html
But this patch never made it into master. Does anybody know more why they did not pursue on that idea?
I quite liked that idea, so I made myself a customized version of HandBrake, that includes this patch. And it gives me nice results, good quality and much smaller files size. So, I don't really understand why this patch never made it into master.
Glad to hear you got some more testing in.

I can imagine it is a difficult feature to avoid thresholding issues with. For example, if the aq-mode changes mid-shot, that could be quite jarring. It likely wouldn't happen much with most content, or with long GOPs such that there aren't many mid-shot IDRs. But with content that varies a lot in style (like mixed CGI/cel/live action) there could be some challenges. And that's exactly the content where changing aq-mode would be most valuable.

With an analysis pass or long lookahead these could be addressed, but it strikes me as a finicky feature.

Plus, does it auto vary --aq-strength too? Because the same value might not be optimal in different modes. That likely would need some auto-adaption as well.

And there simply aren't any objective metrics that do a good job of comparing adaptive quantization modes anyway. Even VMAF gives very similar scores for different approaches, and sometimes penalizes the subjectively preferable mode. I suspect Netflix didn't include a lot of aq-mode variation in their subjective rating database, so VMAF didn't get trained on those variations.

Machine Learning/AI is only as good as the training set, and isn't good at finding types of differences it hasn't been exposed to.

[ShortKatz]

Thanks for your detailed answer. This explains it to me very well. And yes, it does auto vary --aq-strength too. Hope they finalize the patch soon then.

[ShortKatz]

I thought, maybe more people want to test --auto-aq and experiment with it. Therefore I made some versions of HandBrake that include a patched version of x265 to enable --auto-aq. The executables are for Windows and macOS.
You can find them on this very basic (didn't put much effort in it, I know) download page: https://polysom.verilite.de/handbrake/handbrake.html

There must be a reason why they didn't follow up on this patch, so use --auto-aq only for testing purposes.

I need to say, that I don't own a Windows machine, so the Windows executables are completely untested and I can't build a GUI version for Windows. I only tested the macOS executables.
There is also one issue, I found with --auto-aq. Even if --auto-aq is enabled and --auto-aq does show up in MediaInfo, it does say --aq-mode=2. But I think it should be --aq-mode=auto, shouldn't it?

[foxyshadis]

x265 has been entirely in a maintenance mode for at least a couple of years. The only big features that get in are sponsored by streaming companies now, there's no budget or will to review and tweak something as wide-reaching as this patch. Maybe if there's a significant groundswell it might help, so your builds are a big deal!

[Greenhorn]

I thought, maybe more people want to test --auto-aq and experiment with it. Therefore I made some versions of HandBrake that include a patched version of x265 to enable --auto-aq. The executables are for Windows and macOS.
You can find them on this very basic (didn't put much effort in it, I know) download page: https://polysom.verilite.de/handbrake/handbrake.html

There must be a reason why they didn't follow up on this patch, so use --auto-aq only for testing purposes.

I need to say, that I don't own a Windows machine, so the Windows executables are completely untested and I can't build a GUI version for Windows. I only tested the macOS executables.
There is also one issue, I found with --auto-aq. Even if --auto-aq is enabled and --auto-aq does show up in MediaInfo, it does say --aq-mode=2. But I think it should be --aq-mode=auto, shouldn't it?

How did you manage to apply the patch? I downloaded it from x265's patchwork at https://patches.videolan.org/patch/26544/, but git just freaks out about corrupt lines when I try to apply it.

Edit: d'oh, this was from before the HG to git migration wasn't it?

[ShortKatz]

How did you manage to apply the patch? I downloaded it from x265's patchwork at https://patches.videolan.org/patch/26544/, but git just freaks out about corrupt lines when I try to apply it.

Edit: d'oh, this was from before the HG to git migration wasn't it?

Yes, this patch is very old. It was before the migration. But I also had to fix some lines by hand. I can upload the patch that I have unbitrotted to current x265-master, if there is interest.
Edit: I've uploaded the unbitrotted combined patch I've used here: https://github.com/Nomis101/x265/commit/aa354f707043fdf27af89cf836b1e3bb147fa0ff

[benwaggoner]

Is there any reason to think this older algorithm would be beneficial in current builds? Other changes around AQ stuff likely would require auto-aq changes. I believe it predates --hevc-aq, among other things.

[Greenhorn]

Is there any reason to think this older algorithm would be beneficial in current builds? Other changes around AQ stuff likely would require auto-aq changes. I believe it predates --hevc-aq, among other things.

I don't know how useful --auto-aq itself will be (it's just using simplistic statistics to determine which aq mode to use), but the patch itself is from this year (hevc-aq was added in ... 2018 I think?), and additionally includes another aq-mode (5) which is mode 4 with a dark-scene bias. IMHO that's the really interesting part of it.

[benwaggoner]

I don't know how useful --auto-aq itself will be (it's just using simplistic statistics to determine which aq mode to use), but the patch itself is from this year (hevc-aq was added in ... 2018 I think?), and additionally includes another aq-mode (5) which is mode 4 with a dark-scene bias. IMHO that's the really interesting part of it.
There really should be separate --aq-method and --aq-luma-bias parameters, so we can have dark optimization in all AQ modes.