Variance AQ Megathread (AQ v0.48 update--defaults changed)

[Dark Shikari]

Originally Posted by Yoshiyuki Blade

I have a n00b question. What does lowerering the AQ strength do besides create a smaller file size at a given sensitivity? lol. Say from 1.0 to the (now suggested) 0.5-0.6. There's not much I know beyond that.

Because the new VAQ redistributes quantizers; unlike the old AQ, it doesn't just lower them. On many sources, this means a decrease in file size.

[DeathTheSheep]

Strength 0.5 to 0.6 is now recommended for 0.47? Looks like I missed something here.

[Yoshiyuki Blade]

Originally Posted by DeathTheSheep

Strength 0.5 to 0.6 is now recommended for 0.47? Looks like I missed something here.

I wouldn't say recommended, but DS has suggested it a few posts ago, as well as setting qcomp to 1 for the time being. Unfortunately, it's gonna be hours till I get some results O_O!

[Sagekilla]

Originally Posted by Dark Shikari

Because the new VAQ redistributes quantizers; unlike the old AQ, it doesn't just lower them. On many sources, this means a decrease in file size.

See, in my eyes AQ was an algorithm that rearranged quants in a frame to get the best quality, not one that would just "lower quants" like Haali's AQ does.


On a side note, if qcomp is set to 1 and aq-strength is varied, isn't that just making AQ the rate control algorithm?

[akupenguin]

Yes. Static-mode AQ is very similar algorithm-wise to CRF, it just uses a different complexity metric and applies to MBs.

[fields_g]

Originally Posted by Yoshiyuki Blade

I have a n00b question. What does lowerering the AQ strength do besides create a smaller file size at a given sensitivity? lol. Say from 1.0 to the (now suggested) 0.5-0.6. There's not much I know beyond that.

I want anyone to feel free to correct me.

FIRST think of Sensitivity.... What level of flatness should be considered necessary adjustment. THEN.. knowing what needs to be changed, Strength determines how radical the adjustment is.

Higher Sensitivity, more should be fixed.
Higher Strength, more radically the attempt to fix it.
This is why while lowering one of these and raising the other, you can get a file with the same bitrate.

[Yoshiyuki Blade]

From what I've tested so far, AQ on anime look much better with strong settings (such as --aq-strength 1.0 --aq-sensitivity 20). Even though it makes file sizes large at a given crf, it still looks nice at smaller file sizes too, though doing that will compromise overall image quality.

Setting qcomp 1, aq str 0.5, and sensitivity 20 looks nice, and the file size is small, but dark areas don't get improved much in comparison. I'm going to re-run the same settings but change strength to 1.0 and see how it turns out. Sorry that I can't post numbers, I'm unorganized and everything's a mess . I'll try and get a consistent order going.

[Morte66]

Slight diversion from crf: any thoughts about qcomp and sensitivity for 2-pass encode to a target size with 0.47?

[Dark Shikari]

Originally Posted by Morte66

Slight diversion from crf: any thoughts about qcomp and sensitivity for 2-pass encode to a target size with 0.47?

qcomp = 1, sensitivity = 10-15? Sensitivity is much less meaningful with 2pass.

[burfadel]

I did some tests on 'A knight's tale' and 'the 6th day', on both occasions I found the best aq strength and sensitivity, in regards to these two clips as strength 0.6 and senstivity 17.6 to maintain the same file size (very slightly larger, I mean by about 0.5 percent) and be of good quality. I didn't set qcomp.

Again, setting the sensitivity lower than that, even with qcomp being 1, the filesize was way undersized. The quality was also noticeable lower, but was about right in terms of its file size.

Setting aq strength to 1 (auto sens.) with the same clips ended up with doubling the file size.

I keep ending up with a sensitivity of around 17 for strengths of say, 0.6, and slightly lower at around 15 or 16 for strength 1

[Dark Shikari]

Originally Posted by Morte66

Before I head out for the day, let me throw up a concept to be shot down (in the full knowledge that I've no idea what I'm talking about). This describes functionality, not necessarily algorithm:

The encoder chugs along, frame after frame. Within each individual frame, it calculates a spatial bitrate distribution according to the new Variance AQ's definition of quality. Then at the whole frame level, it measures the quality according to the vanilla CRF metric, the one that's not PSNR or SSIM but "implicit in the algorithm", and varies the whole-frame bit budget to hit a quality level measured that way. So VAQ does spatial bitrate distribution within frames, and non-AQ does temporally varying bitrate allocation across frames.

I wonder if that would give the excellent smooth/dark area handling by using VAQ spatially within frames, then make sure the frames are crisp enough by using the current CRF metric (which seems more dependable re crispness than VAQCRF 0.47).

Using CRF for frame QP distribution and VAQ for within-frame bit distribution can be done using --aq-sensitivity 0 and adjusting qcomp to however strong you want CRF to be (lower = stronger)

[Morte66]

Originally Posted by Dark Shikari

Using CRF for frame QP distribution and VAQ for within-frame bit distribution can be done using --aq-sensitivity 0 and adjusting qcomp to however strong you want CRF to be (lower = stronger)

I thought that approach had a problem on frames with a small amount of sharp detail that gets clobbered by variance AQ to feed the large amount of smooth stuff?

I was thinking particularly of CRF measuring quality after VAQ has done spatial bitrate distribution, so it would feed extra bitrate in from the top to resharpen the sharp bits of frames like that. Or is that what it does anyhow?

{I seem to have deleted my original post by accident (!), but DS quoted it whole.}

[Jawed]

Originally Posted by akupenguin

Yes. Static-mode AQ is very similar algorithm-wise to CRF, it just uses a different complexity metric and applies to MBs.

Should VAQ constitute an x264 mode alongside crf, qp, bitrate, pass ?

Jawed

[Dark Shikari]

Originally Posted by Jawed

Should VAQ constitute an x264 mode alongside crf, qp, bitrate, pass ?

Jawed

Well, VAQ works with all of those modes (though, technically, qcomp=1 and CRF is equivalent to QP mode)... so not really, I think.

[Jawed]

A few months back I experimented with qcomp in conjunction with HAQ (pre-processed to PC Levels + MVDegrain2). It certainly produced interesting results:

                 AQ 0.3 5    qcomp 0    qcomp 1  source bitrate
24                 1827.9     2272.1     1621.0     4762.7
Black Pearl        2270.7     1671.9     2924.5     3639.8
Carnivale          2485.2     2115.3     2942.0     6623.8
Hana & Alice 1     2102.7     2557.1     1958.2     4015.5
Monk               3207.5     2132.0     4358.8     8354.6
War of the Worlds  6829.3     3047.2    11542.4     7059.3
X-Files            1951.0     2275.5     1693.9     4821.1
Average            2953.5     2295.9     3863.0     5611.0

Note all three test encodes are with the same HAQ setting, --aq-strength 0.3 --aq-sensitivity 5. The other x264 settings were:

--crf 16 --ref 5 --mixed-refs --no-fast-pskip --bframes 8 --b-pyramid --b-rdo --bime --weightb --subme 6 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 32 --no-dct-decimate

qcomp is a bit surprising. Maybe you guys are used to it.

I've attached the spreadsheet containing a bit more detail.

Jawed

[bob0r]

AQ patch 0.47: http://akuvian.org/src/x264/x264_aq_var.47.diff

x264.735.dark.aq.0.47.exe (pthreads/mp4 = yes, made with make fprofiled)

Should work i guess, patch did apply.

[nurbs]

I did some encodes with qcomp = 1. With aq-strenght 0.5 and sensitivity 10 it gives me similar filesizes to my previous encodes with normal qcomp and without aq. The videos look very good.

[Yoshiyuki Blade]

After some more testing, older anime with lots of noise and details look better (in my opinion) with high CRF and high AQ strength vs low CRF and low AQ strength (at a fixed sensitivity). Both results have their ups and downs.

-Low CRF and Low AQ strength gives a cleaner overall picture as expected, AQ but hardly seems to address ugly blocking in dark areas.

-High CRF and High AQ strength results in a worse overall image quality (as expected), but dark areas look much better.

This trend happens at lower bitrates. At high bitrates, the results look very similar to each other (very good quality across the board).

I'm going to edit this post in a couple hours with some examples. In the meantime, this is what I have so far:

--crf 28.0 --qcomp 1 aq-strength 0.5 aq-sensitivity 20: 188 MB (Sample 1) (Sample 2)
--crf 33.0 --qcomp 1 aq-strength 1.0 aq-sensitivity 20: 199 MB (Sample 1) (Sample 2)
--crf 34.0 --qcomp 1 aq-strength 1.0 aq-sensitivity 20: 161 MB

Update: Uploaded the samples. To be honest, none of them are satisfactory lol, but I expect them to look much better in a 2-pass encode. I did a 2-pass encode a while ago (before patch .47) at lower bitrates than those, and it still looks better than both.

I'm currently re-running tests with 2-pass at a higher bitrate not only to keep the final filesizes very close to each other (there's an 11 MB bias in the tests above), but to narrow the gap on the differences between high CRF/high AQ strength vs low CRF/low AQ strength. The next set of results will look much easier on the eyes.

[Raere]

Are low AQ values recommended as a general psychovisual improvement at this point, or is it still just a tradeoff for lower overall quality for better dark areas?

[Dark Shikari]

Originally Posted by Raere

Are low AQ values recommended as a general psychovisual improvement at this point, or is it still just a tradeoff for lower overall quality for better dark areas?

VAQ is supposed to be a general psychovisual improvement, period. Too high AQ strengths may backfire though.