Variance AQ Megathread (AQ v0.48 update--defaults changed)

[ToS_Maverick]

all current VAQ builds have VAQ enabled by defaul, IIRC.

[Umamio]

I have quite a lot of football (soccer) caps (in huffy). In the past I've tried to get around the loss of detail in the grass by raising qcomp and lowering deblocking to -4, -5 but I think that was just giving the illusion of more detail in the form of fast changing blockiness, but I don't really know what I'm talking about, I just know my football encodes make me sad when I watch them back, whatever settings I try.

Anyway, I am very interested in the developments of this and would like to be of some use however I am able.

So if anyone is interested in specific tests on football material (standard def, PAL) in particular then send me the command line args you'd like me to use and I could encode a 1 minute clip into whatever different specs for comparison.

USE ME.

VAQ2.0 alpha testing Football Comparisons Here (http://forum.doom9.org/showthread.php?p=1120492#post1120492)

[professor_desty_nova]

I have quite a lot of football (soccer) caps (in huffy). In the past I've tried to get around the loss of detail in the grass by raising qcomp and lowering deblocking to -4, -5 but I think that was just giving the illusion of more detail in the form of fast changing blockiness, but I don't really know what I'm talking about, I just know my football encodes make me sad when I watch them back, whatever settings I try.

Anyway, I am very interested in the developments of this and would like to be of some use however I am able.

So if anyone is interested in specific tests on football material (standard def, PAL) in particular then send me the command line args you'd like me to use and I could encode a 1 minute clip into whatever different specs for comparison.

USE ME.

Maybe you should go to the VAQ 2.0 Alpha Testing (http://forum.doom9.org/showthread.php?t=136445) thread, since VAQ 1.0 is finished and already in 805+ builds. I'm sure Dark Shikari will like to ear your tests with the new VAQ 2.0 ;)

[gav1577]

Hi guys i have lost track of this thread lately and was wondering are the aq strength & sensitivity parameters still the same for v805 0.5 and 13 by default
or have things changed? the reason i asked i read somewhere default is now 1.0 for strength or is that just for 2.0 alpha i am a bit confused could someone please explain thanks

[DarkZell666]

"x264 --longhelp" says default strength is 1.0 and doesn't say anything about --aq-sensitivity (which was to be expected, DarkShikari said at some point that this parameter would be removed) ;)

[microchip8]

"x264 --longhelp" says default strength is 1.0 and doesn't say anything about --aq-sensitivity (which was to be expected, DarkShikari said at some point that this parameter would be removed) ;)

it is not removed, it has been replaced by --aq-mode

[gav1577]

So where as before i used --aq-strength 1.0 does this now mean to achieve the same affect i would have to use --aq-strength 1.5 with this new version v805 ? :)

[Dark Shikari]

So where as before i used --aq-strength 1.0 does this now mean to achieve the same affect i would have to use --aq-strength 1.5 with this new version v805 ? :)No, you'd use 2.0.

Every value has been doubled.

[gav1577]

No, you'd use 2.0.

Every value has been doubled.

Ok thanks Dark Shikari great work btw :D

[ToS_Maverick]

is it just me, or did the filesize decrease a bit with 805+ and standard AQ settings?

[Dark Shikari]

is it just me, or did the filesize decrease a bit with 805+ and standard AQ settings?Yes, pengvado changed AQ so as to (hopefully) better correlate with CRF filesizes. This doesn't change twopass.

[Chabb]

In 805+ builds VAQ1 was implemented officially,
but with previous builds I was able
to fine-tune one pass QP ratecontrol using --aq-sensitivity
(fractional changes)
and usage of fractional CRF values for that purpose was
(and still is) impossible, because of fixed value qcomp=1.00
Is there any workaround in this case (except 2-pass)?

[Dark Shikari]

In 805+ builds VAQ1 was implemented officially,
but with previous builds I was able
to fine-tune one pass QP ratecontrol using --aq-sensitivity
(fractional changes)
and usage of fractional CRF values for that purpose was
(and still is) impossible, because of fixed value qcomp=1.00
Is there any workaround in this case (except 2-pass)?I will fix that some time this week.

[Chabb]

Waiting impatiently :)

[karasu]

I recently discovered AQ but I'm a little confused with my own tests.
In short, I cant get visible better quality with AQ on.

My source Is Elephant Dream in sd resolution, and here's my settings :

With AQ disabled :

--pass 2 --bitrate 700 --stats ".stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 64 --threads auto --thread-input --progress --no-psnr --output "output" "input" --aq-strength 0.0
SSIM Mean Y:0.9785036
sample 1 (http://gloomydream.net/bazar/AQ/1-noaq.png)
sample 2 (http://gloomydream.net/bazar/AQ/2-noaq.png)

With default AQ settings :

--pass 2 --bitrate 700 --stats ".stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 64 --threads auto --thread-input --progress --no-psnr --output "output" "input"
SSIM Mean Y:0.9791959
sample 1 (http://gloomydream.net/bazar/AQ/1-aq.png)
sample 2 (http://gloomydream.net/bazar/AQ/2-aq.png)

Even if the SSIM is higher for the AQ, the image is sharper on the non-AQ (look at the antenna in the background of the first sample, or the textures and the elephant head on the second sample)
The only frames where I found slightly better details on the AQ version are on scenes with high motion, so it's hard to see the improvement while watching the movie. The sample above are from low-motion scenes, where it's easier to see the improvement.

I also try with different settings for AQ ( such as --aq-strength 1.0 --aq-sensitivity 20) but my results are worse than with the default settings.

Is there something I'm doing wrong? Is the bitrate too low to see the benefits of AQ?

(I'm using megui with x264 jarod's patched build 798)

[Razorholt]

I recently discovered AQ but I'm a little confused with my own tests.
In short, I cant get visible better quality with AQ on.

My source Is Elephant Dream in sd resolution, and here's my settings :

With AQ disabled :

--pass 2 --bitrate 700 --stats ".stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 64 --threads auto --thread-input --progress --no-psnr --output "output" "input" --aq-strength 0.0
SSIM Mean Y:0.9785036
sample 1 (http://gloomydream.net/bazar/AQ/1-noaq.png)
sample 2 (http://gloomydream.net/bazar/AQ/2-noaq.png)

With default AQ settings :
--pass 2 --bitrate 700 --stats ".stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,-2 --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 64 --threads auto --thread-input --progress --no-psnr --output "output" "input"
SSIM Mean Y:0.9791959
sample 1 (http://gloomydream.net/bazar/AQ/1-aq.png)
sample 2 (http://gloomydream.net/bazar/AQ/2-aq.png)

Even if the SSIM is higher for the AQ, the image is sharper on the non-AQ (look at the antenna in the background of the first sample, or the textures and the elephant head on the second sample)
The only frames where I found slightly better details on the AQ version are on scenes with high motion, so it's hard to see the improvement while watching the movie. The sample above are from low-motion scenes, where it's easier to see the improvement.

I also try with different settings for AQ ( such as --aq-strength 1.0 --aq-sensitivity 20) but my results are worse than with the default settings.

Is there something I'm doing wrong? Is the bitrate too low to see the benefits of AQ?

(I'm using megui with x264 jarod's patched build 798)

I second that statement. Try the 0.46 version and see what you get. I've already asked DS regarding the minimum birates requirement for VAQ to be efficient.

x264 with VAQ 0.46: x264.772.modified.exe (http://70.86.69.186/~matrix2/x264.772.modified.exe) (use --aq-strength 0.5 --aq-sensitivity 11)

Now, I must say that VAQ2 is quite good @ 700kb/s - You should try it.

- Dan

[Dark Shikari]

Is there something I'm doing wrong? Is the bitrate too low to see the benefits of AQ?AQ is a great way for x264 to teach people about the disadvantages of using stupid deblocking settings, like you're using ;)I second that statement. Try the 0.46 version and see what you get.Can people seriously stop spreading bugged code with the bizarre claim that because it randomly makes mistakes, its "better," even though nobody has ever been able to prove it in the well over two months since the bug was fixed? This is ridiculous, and this sort of conduct is exactly why I dislike releasing patches.

[DarkZell666]

@karasu : One thing you did do wrong was to use different deblocking settings on each sample in the first place. The No AQ one is -2,-1, and the AQ one is -3,-2. That voids your comparison completely :)

[Razorholt]

Can people seriously stop spreading bugged code with the bizarre claim that because it randomly makes mistakes, its "better," even though nobody has ever been able to prove it in the well over two months since the bug was fixed? This is ridiculous, and this sort of conduct is exactly why I dislike releasing patches.

I don't want to argue with you on that subject because someone else is already doing it...

The point is: you said that the best way to judge the settings is by using your eyes (over SSIM). Well, I think it's no secret that people have different eyes and not the same definition of quality.

My experience with your VAQ releases is that at very low bitrates 0.46 was giving ME better results and MY EYES found it quite satisfactory (yes, I may have shitty eyes...). I know you find ridiculous to encode at low birates but sometimes that's the only option. Have you ever tried encoding at 600 or 700kb/s ?

Anyway, I'm quite happy with VAQ2, even at low birates :)

Keep up the great work, DS.

- Dan

[Dark Shikari]

I don't want to argue with you on that subject because someone else is already doing it...

The point is: you said that the best way to judge the settings is by using your eyes (over SSIM). Well, I think it's no secret that people have different eyes and not the same definition of quality.

My experience with your VAQ releases is that at very low bitrates 0.46 was giving ME better results and MY EYES found it quite satisfactory (yes, I may have shitty eyes...). I know you find ridiculous to encode at low birates but sometimes that's the only option. Have you ever tried encoding at 600 or 700kb/s ?Yes, I often test my AQ at bitrates as low as 200kbps or less. It generally has no effect on its usefulness.

Example: "blah blah blah AQ is useless on low-bitrate anime blah blah blah"

http://i31.tinypic.com/2ilg3ud.gif

[Southstorm]

I'm sold!

[Razorholt]

I never mentioned 200kbps and I never encode anime stuff.

[Dark Shikari]

I never mentioned 200kbps and I never encode anime stuff.I was referring to the general commentary that AQ wasn't good at low bitrate anime (including such comments made by myself).

[karasu]

I'm sorry for the confusion, the different settings in deblocking come form an error of copy/pasting. (I'm trying to vary each settings at once to see what's happens, I think it's the best way to learn about x264)

My mistake was to copy the wrong line on my "huge settings log file"...

Of course the deblock settings was the same for my two tests.

Perhaps a deblocking of -2-1 is stupid, but to my eyes it look more like the original than deblock 0 0 .

However, I'm still using an outdated version of AQ, I'll retry with one of the builds of this thread, or maybe VAQ2

[Dark Shikari]

Perhaps a deblocking of -2-1 is stupid, but to my eyes it look more like the original than deblock 0 0 .AQ in particular suffers from overly low deblocking strengths because they result in deblocking not being applied at all to background blocks.

[karasu]

AQ in particular suffers from overly low deblocking strengths because they result in deblocking not being applied at all to background blocks.

Ok, thank you for these explications.
I have made some tests with default deblocking. The SSIM boost is important.
High motions scenes are greatly improved,
high motion AQ (http://gloomydream.net/bazar/AQ/hm-aq.png)
high motion no AQ (http://gloomydream.net/bazar/AQ/hm-noaq.png)

but the the stills or very slow motions scenes are less sharper than without AQ.
low motion AQ (http://gloomydream.net/bazar/AQ/lm-aq.png)
low motion no AQ (http://gloomydream.net/bazar/AQ/lm-noaq.png)

Perhaps I'm completely wrong (I'm here to learn) but I think that we are more sensible to quality in slow motion scenes. So for the human eye, the gain in perceptual quality isn't as good as the SSIM gain.
Do you think that a way to adapt the AQ strength in function of the speed of the motion can be useful in that case?

[Inventive Software]

For the record, if you're encoding SD Elephants Dream at 500 Kbits or around that mark, the MINIMUM deblocking you can get away with is -1:-1 (not tried it, but might help in the telephone wires scene), highly recommended is the default (0:0). I did a comparison between VC-1 and H.264 a while back, I did one encode with all the bells and whistles, long before VAQ was around, and Haali's AQ was disabled. It looked very good! :)

[Lele-brz]

I had the same impression of Karasu.
Sometimes AQ seems to decrease the perceived quality on some videos.
After some test I would use it on high motion (especially soccer games where the grass with AQ looks much better), but not always.
So, I'm wondering why it's turned on by default.

[MfA]

Do you think that a way to adapt the AQ strength in function of the speed of the motion can be useful in that case?
Maybe someone can step up to do something like this after Aki Jäntti's SoC project, once his temporal search and per MB record keeping code is in there (hopefully) you could add future motion on top of future coding relevance (which he is working on). Just naively weighting based on current frame motion is a bad idea, a large foreground object which moves predictably can be in perfect focus on your retina ... only for extreme velocities would this kind of naive weighting make sense, you need more data and better metrics.

[Ranguvar]

By far the worst "problem", in my opinion, with VAQ, is the mosquito noise that inevitably surrounds hardcoded subtitles. For example, in The Lord of the Rings, when there are hardcoded subs for the Elvish parts. I can make a 2CD backup that looks pretty perfect in my opinion, except for said noise.

Are there any plans to alleviate this, or can you recommend settings that will mitigate that without impacting the rest of the video much?

[Dark Shikari]

By far the worst "problem", in my opinion, with VAQ, is the mosquito noise that inevitably surrounds hardcoded subtitles. For example, in The Lord of the Rings, when there are hardcoded subs for the Elvish parts. I can make a 2CD backup that looks pretty perfect in my opinion, except for said noise.

Are there any plans to alleviate this, or can you recommend settings that will mitigate that without impacting the rest of the video much?On the one hand, without VAQ, hardcoded subtitles end up using up an absurd proportion of the number of bits in a video because of their sharp edges. VAQ's algorithm makes the assumption--a valid one, I think--that you probably don't want the subtitles wasting so many bits.

On the other hand, you could just try weakening AQ a bit; IMO subtitles can go quite a bit up quantizer-wise before there's a serious visual problem.

[Ranguvar]

Alright, thanks. I guess I'm just screwed because my eyes seem to hone in on subtitles... too much anime xD

Didn't know they took up so many bits, however, so I'm at least somewhat put at ease :p

[Ranguvar]

Hey, me again with little annoying questions :)

This thread says:
Version history:

0.48: AQ strength 0.5, sensitivity 13 made the defaults. Updated to r736. Qcomp is now scaled based on AQ strength automatically.

However, --longhelp on r899 reports that a strength of 1.0 is the default.

Which is it, please? I poked around the source a little, but I am still an über-noob at code...

[Dark Shikari]

Hey, me again with little annoying questions :)

This thread says:


However, --longhelp on r899 reports that a strength of 1.0 is the default.

Which is it, please? I poked around the source a little, but I am still an über-noob at code...The strengths were all doubled so that 1.0 would be the default, for interface reasons.

Old 0.5 == New 1.0

[egrimisu]

What is the recomended variance aq setting to achive highest quality in megui: strengh 1.0 or 2.0 ??? i' using 967-1 skytrife release

[Atak_Snajpera]

default strengh 1.0 should be optimal for most cases

[egrimisu]

Thanks, i used 2.0 for an encode and the results where NOT that greate, retrying using 1.0. THANKS again

default strengh 1.0 should be optimal for most cases

[Zwitterion]

Now that we have PsyRD, I find the AQ default value of 1.0 a little too high.
When AQ was tuned, trading off some edge definition for the vast improvement of flat areas was acceptable. But now PsyRD improves those flat areas a lot, so AQ doesn't need to be as high.

I recommend 0.5 as a sensible default value when using PsyRD.

[Dark Shikari]

Now that we have PsyRD, I find the AQ default value of 1.0 a little too high.
When AQ was tuned, trading off some edge definition for the vast improvement of flat areas was acceptable. But now PsyRD improves those flat areas a lot, so AQ doesn't need to be as high.

I recommend 0.5 as a sensible default value when using PsyRD.Quantizer-field smoothing is coming soon, and should resolve most of these issues.

[CruNcher]

1.0 can be very agressive in some low bitrate situations but hopefully quantizer field smoothing will really bring the visual fix for that, also trying to compensate some of these edge problems currently by useing higher chroma quants seems to sometimes work wonders currently, though it will use a little more bitrate. I guess what happens is it compensates AQ a little so it acts like a fine tuneing mechanism takeing efficiency of AQ and on the other side i guess as the chroma information most of the times is everywhere near edges it helps here by allocateing more bits to these so they dont (the chroma near the edges of a object) fuzz out to fast in motion :) (but all this only becomes a real visual problem at very low bitrates most of the times, and @ these you dont should care so much anymore about detail preservation but motion stability).

[salehin]

DarkShikari: is r736 the latest x264 build with your FGO? If not, can you please suggest any available build for a very grainy/noisy source. I see 998 (by skystrife) with MeGUI- iirc, that doesn't contain your FGO. The source is a lovely DVD that I own, mastered from a handheld camera contents, at least 20 yrs old (the programme)!

Cheers :)

[LoRd_MuldeR]

FGO has been superseeded by Psy-RDO and Psy-Trellis ;)

(I read that some people still claim that FGO is better for very grainy sources, but Psy RDO/Trellis is a more general approach and helpful for almost any source)

[burfadel]

Just remember Psy-trellis (or is it psy-rdo, can't check at the moment)! is disabled by default. The reason for the disable is some people claim in 'clean' (as in picture noise) sources, particularly some animation, it looks better without. At some stage it will be enabled by default, maybe after some slight tweaking. In terms of the source that you have described enabling it would be a good idea. It can be enabled by the command line option:
--psy-rd 1.0:1.0

[Sagekilla]

there's psy-rdo and psy-trellis: the first param in --psy-rd is the psy-rdo proper, and the second activates a psy-trellis mode.

[LoRd_MuldeR]

Just remember Psy-trellis (or is it psy-rdo, can't check at the moment)! is disabled by default. The reason for the disable is some people claim in 'clean' (as in picture noise) sources, particularly some animation, it looks better without. At some stage it will be enabled by default, maybe after some slight tweaking. In terms of the source that you have described enabling it would be a good idea. It can be enabled by the command line option:
--psy-rd 1.0:1.0

The current default is "--psy-rd 1.0:0.0", which means Psy-RDO is enabled and Psy-Trellis is disabled.

Some comparision can be found here:
* http://forum.doom9.org/showthread.php?t=141188
* http://forum.doom9.org/showthread.php?t=141249