Variance AQ Megathread (AQ v0.48 update--defaults changed)

[Southstorm]

I'm sold!

[Razorholt]

I never mentioned 200kbps and I never encode anime stuff.

[Dark Shikari]

Originally Posted by Razorholt

I never mentioned 200kbps and I never encode anime stuff.

I was referring to the general commentary that AQ wasn't good at low bitrate anime (including such comments made by myself).

[karasu]

I'm sorry for the confusion, the different settings in deblocking come form an error of copy/pasting. (I'm trying to vary each settings at once to see what's happens, I think it's the best way to learn about x264)

My mistake was to copy the wrong line on my "huge settings log file"...

Of course the deblock settings was the same for my two tests.

Perhaps a deblocking of -2-1 is stupid, but to my eyes it look more like the original than deblock 0 0 .

However, I'm still using an outdated version of AQ, I'll retry with one of the builds of this thread, or maybe VAQ2

[Dark Shikari]

Originally Posted by karasu

Perhaps a deblocking of -2-1 is stupid, but to my eyes it look more like the original than deblock 0 0 .

AQ in particular suffers from overly low deblocking strengths because they result in deblocking not being applied at all to background blocks.

[karasu]

Originally Posted by Dark Shikari

AQ in particular suffers from overly low deblocking strengths because they result in deblocking not being applied at all to background blocks.

Ok, thank you for these explications.
I have made some tests with default deblocking. The SSIM boost is important.
High motions scenes are greatly improved,
high motion AQ
high motion no AQ

but the the stills or very slow motions scenes are less sharper than without AQ.
low motion AQ
low motion no AQ

Perhaps I'm completely wrong (I'm here to learn) but I think that we are more sensible to quality in slow motion scenes. So for the human eye, the gain in perceptual quality isn't as good as the SSIM gain.
Do you think that a way to adapt the AQ strength in function of the speed of the motion can be useful in that case?

[Inventive Software]

For the record, if you're encoding SD Elephants Dream at 500 Kbits or around that mark, the MINIMUM deblocking you can get away with is -1:-1 (not tried it, but might help in the telephone wires scene), highly recommended is the default (0:0). I did a comparison between VC-1 and H.264 a while back, I did one encode with all the bells and whistles, long before VAQ was around, and Haali's AQ was disabled. It looked very good!

[Lele-brz]

I had the same impression of Karasu.
Sometimes AQ seems to decrease the perceived quality on some videos.
After some test I would use it on high motion (especially soccer games where the grass with AQ looks much better), but not always.
So, I'm wondering why it's turned on by default.

[MfA]

Originally Posted by karasu

Do you think that a way to adapt the AQ strength in function of the speed of the motion can be useful in that case?

Maybe someone can step up to do something like this after Aki Jäntti's SoC project, once his temporal search and per MB record keeping code is in there (hopefully) you could add future motion on top of future coding relevance (which he is working on). Just naively weighting based on current frame motion is a bad idea, a large foreground object which moves predictably can be in perfect focus on your retina ... only for extreme velocities would this kind of naive weighting make sense, you need more data and better metrics.

[Ranguvar]

By far the worst "problem", in my opinion, with VAQ, is the mosquito noise that inevitably surrounds hardcoded subtitles. For example, in The Lord of the Rings, when there are hardcoded subs for the Elvish parts. I can make a 2CD backup that looks pretty perfect in my opinion, except for said noise.

Are there any plans to alleviate this, or can you recommend settings that will mitigate that without impacting the rest of the video much?

[Dark Shikari]

Originally Posted by Ranguvar

By far the worst "problem", in my opinion, with VAQ, is the mosquito noise that inevitably surrounds hardcoded subtitles. For example, in The Lord of the Rings, when there are hardcoded subs for the Elvish parts. I can make a 2CD backup that looks pretty perfect in my opinion, except for said noise.

Are there any plans to alleviate this, or can you recommend settings that will mitigate that without impacting the rest of the video much?

On the one hand, without VAQ, hardcoded subtitles end up using up an absurd proportion of the number of bits in a video because of their sharp edges. VAQ's algorithm makes the assumption--a valid one, I think--that you probably don't want the subtitles wasting so many bits.

On the other hand, you could just try weakening AQ a bit; IMO subtitles can go quite a bit up quantizer-wise before there's a serious visual problem.

[Ranguvar]

Alright, thanks. I guess I'm just screwed because my eyes seem to hone in on subtitles... too much anime xD

Didn't know they took up so many bits, however, so I'm at least somewhat put at ease

[Ranguvar]

Hey, me again with little annoying questions

This thread says:

Version history:

0.48: AQ strength 0.5, sensitivity 13 made the defaults. Updated to r736. Qcomp is now scaled based on AQ strength automatically.

However, --longhelp on r899 reports that a strength of 1.0 is the default.

Which is it, please? I poked around the source a little, but I am still an über-noob at code...

[Dark Shikari]

Originally Posted by Ranguvar

Hey, me again with little annoying questions

This thread says:


However, --longhelp on r899 reports that a strength of 1.0 is the default.

Which is it, please? I poked around the source a little, but I am still an über-noob at code...

The strengths were all doubled so that 1.0 would be the default, for interface reasons.

Old 0.5 == New 1.0

[egrimisu]

What is the recomended variance aq setting to achive highest quality in megui: strengh 1.0 or 2.0 ??? i' using 967-1 skytrife release

[Atak_Snajpera]

default strengh 1.0 should be optimal for most cases

[egrimisu]

Thanks, i used 2.0 for an encode and the results where NOT that greate, retrying using 1.0. THANKS again

Originally Posted by Atak_Snajpera

default strengh 1.0 should be optimal for most cases

[Zwitterion]

Now that we have PsyRD, I find the AQ default value of 1.0 a little too high.
When AQ was tuned, trading off some edge definition for the vast improvement of flat areas was acceptable. But now PsyRD improves those flat areas a lot, so AQ doesn't need to be as high.

I recommend 0.5 as a sensible default value when using PsyRD.

[Dark Shikari]

Originally Posted by Zwitterion

Now that we have PsyRD, I find the AQ default value of 1.0 a little too high.
When AQ was tuned, trading off some edge definition for the vast improvement of flat areas was acceptable. But now PsyRD improves those flat areas a lot, so AQ doesn't need to be as high.

I recommend 0.5 as a sensible default value when using PsyRD.

Quantizer-field smoothing is coming soon, and should resolve most of these issues.

[CruNcher]

1.0 can be very agressive in some low bitrate situations but hopefully quantizer field smoothing will really bring the visual fix for that, also trying to compensate some of these edge problems currently by useing higher chroma quants seems to sometimes work wonders currently, though it will use a little more bitrate. I guess what happens is it compensates AQ a little so it acts like a fine tuneing mechanism takeing efficiency of AQ and on the other side i guess as the chroma information most of the times is everywhere near edges it helps here by allocateing more bits to these so they dont (the chroma near the edges of a object) fuzz out to fast in motion (but all this only becomes a real visual problem at very low bitrates most of the times, and @ these you dont should care so much anymore about detail preservation but motion stability).