Variance AQ Megathread (AQ v0.48 update--defaults changed)

[au]

Is there a way to completely eliminate banding in dark areas (smooth gradients etc.)?
I've tried different combinations of aq-strength(0-1), aq-sensitivity(0-20) and various bitrate(6-20Mbps), but it came to nothing.

The settings I use:
x264aq048.exe --pass 1 --bitrate 9000 --level 4.1 --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -3:-3 --subme 7 --partitions all --8x8dct --me umh --threads auto --thread-input --cqmfile "matrix/prestige.cfg" --progress --deadzone-inter 6 --deadzone-intra 4 --aq-strength 0.5 --qcomp 1.0 --aq-sensitivity 15 --no-dct-decimate --output NUL "xxx"
x264aq048.exe --pass 2 --bitrate 9000 --level 4.1 --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -3:-3 --subme 7 --partitions all --8x8dct --me umh --threads auto --thread-input --cqmfile "matrix/prestige.cfg" --progress --deadzone-inter 6 --deadzone-intra 4 --aq-strength 0.5 --qcomp 1.0 --aq-sensitivity 15 --no-dct-decimate --output "yyy" "xxx"

[Dark Shikari]

Is there a way to completely eliminate banding in dark areas (smooth gradients etc.)?
I've tried different combinations of aq-strength(0-1), aq-sensitivity(0-20) and various bitrate(6-20Mbps), but it came to nothing.That basically isn't possible in the YV12 color space.

Try running gradfun2db first?

[Sagittaire]

Is there a way to completely eliminate banding in dark areas (smooth gradients etc.)?
I've tried different combinations of aq-strength(0-1), aq-sensitivity(0-20) and various bitrate(6-20Mbps), but it came to nothing.


make dithering in pre-process ...

[Jawed]

Simplest sanity check for an x264 encode is --qp 0 :)

Jawed

[Inventive Software]

Cue the terabyte hard drive...

[nm]

No need to encode a complete movie. Just pick a scene that shows banding in the previous tries and encode that with --qp 0 to see if the problem goes away. If not, it could also be due to bad rendering.

[LoRd_MuldeR]

Yes, this is because AQ + CRF = AQ + CQP, because qcomp gets disabled.
If VAQ+CRF = VAQ+CQP and if VAQ is enabled by default, will CRF still be there as a separate mode when VAQ hits the SVN?

[Dark Shikari]

If VAQ+CRF = VAQ+CQP and if VAQ is enabled by default, will CRF still be there as a separate mode when VAQ hits the SVN?Sure, because you can still disable AQ.

[LoRd_MuldeR]

Sure, because you can still disable AQ.

Yeah, but wouldn't it make more sens to make three modes then?
* CQP (classic CQP, no VAQ)
* CRF (classic CRF, no VAQ)
* VAQ (CQP/CRF with VAQ on)

At the moment we have four combination, where two are exactly the same...

[hkazemi]

In the interest of backwards compatibility with the command lines rate control mode parameters, I suggest the following implementation (if this suggestion should also be made in another place, someone please let me know). (Backwards compatibility is important for easily using new x264 builds in existing scripts that assume what each mode does) :

VAQ=variance AQ
CRF=constant rate factor
CQP=constant quantization

1.) use VAQ if rate control mode is not specified (default)
2.) use VAQ if -vaq=xx (e.g. --vaq=20) is specified (this is the new CRF mode aka VAQ)
3.) use CRF if -crf=xx (e.g. --crf=20) is specified (this is the classical CRF mode)
4.) use CQP if -qp=xx (e.g. --qp=20) is specified (this is the classical CQP mode)

According to akupenguin (http://forum.doom9.org/showthread.php?t=132760&page=24#post1091386):
Current CRF is constant quality as measured by one particular metric (not psnr or ssim or any of the normal metrics, but an ad-hoc one implicit in the CRF algorithm). It's similar bitrate on average as the same value of CQP, because I multiplied the CRF scale by a magic number to make that true, not by anything inherent in the algorithm.

CRF+HaaliAQ is constant quality as measured by another metric. It's always at least a little higher bitrate per CRF value than plain CRF because HaaliAQ has a negative average QP bias, but that's just a UI tuning issue and could be fixed if desired.

CRF+VAQ(static) is constant quality as measured by another metric. If you pick some random value of aq-sensitivity, it won't be equal on average, but will have some bias. We're trying to find a sensitivity value such that that bias averages to 0, thus it will hopefully be about the same bitrate on average as a given value of plain CRF if we do the tuning right (unless we choose to keep psy quality the same instead of bitrate).

CRF+VAQ(auto) does the frame-wise bit allocation using the same metric as plain CRF, but reallocates bits within the frame using the VAQ metric.

Was it ever decided whether to have two kinds of VAQ (auto and static)? I hope that the final VAQ implementation will have the parameter value tuned like the CRF parameter was tuned...so that the average bitrate of the output files are similar.

[Dark Shikari]

In the interest of backwards compatibility with the command lines rate control mode parameters, I suggest the following implementation (if this suggestion should also be made in another place, someone please let me know):

VAQ=variance AQ
CRF=constant rate factor
CQP=constant quantization

1.) use VAQ if rate control mode is not specified (default)
2.) use VAQ if -vaq=xx (e.g. --vaq=20) is specified
3.) use CRF if -crf=xx (e.g. --crf=20) is specified
4.) use CQP if -qp=xx (e.g. --qp=20) is specified

According to akupengiun (http://forum.doom9.org/showthread.php?t=132760&page=24#post1091386):

Was it ever decided to have types VAQ? (auto and static)? I hope that the final VAQ implementation will have the parameter value tuned like the CRF parameter was tuned...so that the average bitrate of the output files are similar.But how do you intend to implement AQ with 1pass and two pass bitrate modes?

[hkazemi]

But how do you intend to implement AQ with 1pass and two pass bitrate modes?

For completely compatible behavior with current builds, I think the bitrate modes should have a new parameter that enables VAQ. (I know this goes against the idea of VAQ being enabled by default, however I think maintaining compatibility, and maintaining the expected output is important.)

Current:
x264 --bitrate 3000
x264 --pass 1 --bitrate 3000
x264 --pass 2 --bitrate 3000

Proposed (preferred):
x264 --bitrate 3000 (VAQ not used)
x264 --pass 1 --bitrate 3000 (VAQ not used)
x264 --pass 2 --bitrate 3000 (VAQ not used)
x264 --bitrate 3000 --vaqon (VAQ used)
x264 --pass 1 --bitrate 3000 --vaqon (VAQ used)
x264 --pass 2 --bitrate 3000 --vaqon (VAQ used)

The other way, if we don't mind breaking user expectations on the codec output, is to have a new parameter that brings back the 'old' style output. This new parameter would force VAQ off. It could be something like:
--vaqoff or --novaq

Alternate proposal:
x264 --bitrate 3000 (VAQ used)
x264 --pass 1 --bitrate 3000 (VAQ used)
x264 --pass 2 --bitrate 3000 (VAQ used)
x264 --bitrate 3000 --vaqoff (VAQ not used)
x264 --pass 1 --bitrate 3000 --vaqoff (VAQ not used)
x264 --pass 2 --bitrate 3000 --vaqoff (VAQ not used)

I do not know if it makes sense to include a value with the --vaqon parameter, like --vaqon=20, when using a fixed average bitrate. If it doesn't make sense, then don't use the number. If it does make sense, the proposed --vaq=20 parameter could be overloaded to serve two similar functions, one by itself when it is the main rate control function, or two, when it is subserviant to the average bitrate rate control function.

[hkazemi]

Here's my revised recommendation for full backwards compatibility, so as not to change the expected output as obtained by older command line parameters as used in existing x264 GUIs and scripts:
ABR=average bitrate
CBR=constant bitrate
CQP=constant quantization
CRF=constant rate factor
VAQ=variance adaptive quantizer

ABR and CBR are similar in that the bitrates should be equal over the whole encode, but differ in that ABR may have areas in the video above the specified bitrate, and other areas below the specified bitrate. CBR uses the same bitrate everywhere (whether or not that means too much or too little for particular scene). CBR has traditionally been used for streams where the bitrate must not exceed a certain value, such as on cpu-constrained playback devices that cannot handle the peaks in ABR. However, if you use ABR with --vbv-maxrate, you can put a ceiling on the output stream to not exceed a particular bitrate, thereby letting the x264 encoder use fewer bits in places they are not needed and also saving space. (Using the --vbv-maxrate parameter also requires you to use the --vbv-bufsize parameter.)

The command line should specify one of the following rate control factor mechanisms:

Quality Priority/Quality First Modes:
The average bitrates generated by the same parameter values will ideally be similar.
1.) --qp=xx means use CQP (e.g. --qp=20) (this is the classical CQP/constant quantization parameter mode) (the original constant quality mode)
2.) --crf=xx means use CRF (e.g. --crf=20) (this is the classical CRF/constant rate factor mode) (the second constant quality mode)
3.) --vaq=xx means use VAQ (e.g. --vaq=20) (this is the new mode using VAQ/variance adaptive quantizer) (the newest constant quality mode, meant to reduce banding and artifacts in dark areas)

Bitrate Priority/Bitrate First Modes:
4.) --bitrate 3000 means use ABR (this is the classical single pass ABR/average bitrate mode)
5.) --pass 1 --bitrate 3000 means run the 1st pass of a 2-pass ABR (this is the classical 1st pass ABR/average bitrate mode)
6.) --pass 2 --bitrate 3000 means run the 2nd pass of a 2-pass ABR (this is the classical 2nd pass ABR/average bitrate mode)
7.) the above three modes with the addition of the --vbv-maxrate and --vbv-bufsize parameters to constrain the maximum bitrate
8.) --bitrate 3000 --qcomp 0 means use CBR (this is the classical single pass CBR/constant bitrate mode)

Bitrate Priority/Bitrate First Modes with VAQ:
9.) --bitrate 3000 --vaq means use ABR+VAQ (this is the single pass ABR/average bitrate mode with VAQ enabled)
10.) --pass 1 --bitrate 3000 --vaq means run the 1st pass of a 2-pass ABR+VAQ (this is the 1st pass ABR/average bitrate mode with VAQ enabled)
11.) --pass 2 --bitrate 3000 --vaq means run the 2nd pass of a 2-pass ABR+VAQ (this is the 2nd pass ABR/average bitrate mode with VAQ enabled)
12.) the above three modes with the addition of the --vbv-maxrate and --vbv-bufsize parameters to constrain the maximum bitrate
13.) --bitrate 3000 --qcomp 0 --vaq means use CBR+VAQ (this is the single pass CBR/constant bitrate mode with VAQ enabled)

If it makes any sense to include a value with the --vaq parameter when using ABR, like --bitrate 3000 --vaq=20, then simply add that parameter. I am suggesting overloading the --vaq parameter to serve two similar functions, first when used by itself as the main rate control function, and second when it is subserviant to the average bitrate rate control function.

Do you know if the VAQ(auto) and the VAQ(static) options still exist as described by akupenguin (quoted in my previous post above)?

Also, I suggest renaming --aq-sensitivity=xx to --vaq-sensitivity=xx (--aq-strength is going away/being turned into --vaq=xx, right?).

Finally, for a default rate control mode if none is specified on the command line, something like this default may make sense:
--vaq=20 (bitrate equivalent to --crf=20)

P.S. CBR, ABR notes, particularly on using vbv-maxrate:
http://forum.doom9.org/printthread.php?t=130125&pp=40
http://forum.doom9.org/showthread.php?t=124560
http://forum.doom9.org/showthread.php?t=102893
http://forum.doom9.org/showthread.php?p=873073
http://forum.doom9.org/showthread.php?p=699479#post699479
http://forum.handbrake.fr/viewtopic.php?f=7&t=4099&start=20&st=0&sk=t&sd=a
http://forum.handbrake.fr/viewtopic.php?f=7&t=1732&st=0&sk=t&sd=a&start=40

[Dark Shikari]

Do you know if the VAQ(auto) and the VAQ(static) options still exist as described by akupenguin (quoted in my previous post above)?Yes, they still exist, and will still exist, especially since auto is very useful for CBR mode.

[hkazemi]

Yes, they still exist, and will still exist, especially since auto is very useful for CBR mode.

So how does one choose between VAQ (auto) and VAQ (static) on the command line?

Is VAQ (static) where a 'value' is provided, as in when people currently use '--crf 20' with the 20 being the value with the AQ-enabled-by-default builds?

Is VAQ (auto) only useful for the ABR/CBR mode? (Assuming I'm safe assuming ABR=CBR (average bitrate=constant bitrate) in x264.)

[Dark Shikari]

So how does one choose between VAQ (auto) and VAQ (static) on the command line?

Is VAQ (static) where a 'value' is provided, as in when people currently use '--crf 20' with the 20 being the value with the AQ-enabled-by-default builds?

Is VAQ (auto) only useful for the ABR/CBR mode? (Assuming I'm safe assuming ABR=CBR (average bitrate=constant bitrate) in x264.)Static is useful for unrestricted VBR of any sort (CRF/QP or bitrate). Auto is useful for CBR. There might be some sort of median for restricted VBR, but I haven't done much research into that.

One idea would be to have a scale between "auto" and "static" that is set automatically based on the ratio of VBV maxrate to bitrate.

[CruNcher]

make dithering in pre-process ...

fine tuning VAQ and useing a custom matrix can do it, see my last examples for some heavy (even luma) ROI banding problems and how i (almost) fixed them (at least they become very hard perceptable) Visually under low bitrate conditions :)

[cheshire2k]

does anyone know if i can use this with fairuse wizard ?

[vpupkind]

Constant Bitrate Modes with VAQ:
7.) --bitrate 3000 --vaq means use ABR+VAQ (this is the single pass ABR/average bitrate mode with VAQ enabled)
8.) --pass 1 --bitrate 3000 --vaq means run the 1st pass of a 2-pass ABR+VAQ (this is the 1st pass ABR/average bitrate mode with VAQ enabled)
9.) --pass 2 --bitrate 3000 --vaq means run the 2nd pass of a 2-pass ABR+VAQ (this is the 2nd pass ABR/average bitrate mode with VAQ enabled)


How would you actually make AQ work with ABR?
From my tests with 0.48, I have seen relatively non-constant behavior of the bitrate.

[Sharktooth]

no. FUW is old and it uses an interface x264 does no longer support.
use a proper program wich correctly support x264 as atak_snjpera suggested in the other thread.
also have a look here: http://forum.doom9.org/showthread.php?t=129748

[Dark Shikari]

How would you actually make AQ work with ABR?
From my tests with 0.48, I have seen relatively non-constant behavior of the bitrate.I've never had a problem with AQ and ABR as long as the same settings are used on both passes.

[cheshire2k]

theres so many to choose from any idea on what to use other than automkv coz that craps out on me now :( never use to.

[LoRd_MuldeR]

Well, recent VfW builds with VAQ included can be found here:
http://sf.net/project/showfiles.php?group_id=213809

You'll have to use "x264vfw experimental", if you want to use VAQ ;)


[EDIT]

Also you might want to give Avidemux a try:
http://forum.doom9.org/showthread.php?t=126164

There is an Avidemux build with x264+VAQ available now and it works fine :)

[Adub]

theres so many to choose from any idea on what to use other than automkv coz that craps out on me now :( never use to.

You should bring it up with the author then, and try to get the problem resolved.

[cheshire2k]

already have mate with no joy. Even updated tried the beta. Ive just looked at that avidemux but I cant put my dvd iso or folder thru it im after a dvd to x264 with this vaq built in but gui so i can crop the black bars.

[LoRd_MuldeR]

Ive just looked at that avidemux but I cant put my dvd iso or folder thru it

You have already posted the same question in the "Avidemux FAQ" thread :rolleyes:
See my answer there and avoid double posting in future...

[foxyshadis]

You have already posted the same question in the "Avidemux FAQ" thread :rolleyes:
See my answer there and avoid double posting in future...

Better yet, make your own thread if you suspect you'll be asking for help on a lot of scattered topics. That way everyone's on the same page. And don't crosspost it, just pick a likely-looking forum.

[au]

That basically isn't possible in the YV12 color space.


Yes, it's a limitation of YV12. But I don't actually understand how it works in AVISynth. As I remember, YV12 uses 3x 8 bit color components (Y, Cr and Cb) per pixel, so the color depth must not be reduced.

I've started the new thread in AVISynth development (http://forum.doom9.org/showthread.php?p=1102819#post1102819)

[Wilbert]

YV12 uses 3x 8 bit color components (Y, Cr and Cb) per pixel, so the color depth must not be reduced.
YV12 uses 12 bits per pixel: 8 bits for Y and 4 bits for CbCr (the chroma is shared between four pixels - 2*8bit/4pix = 4 bits per pixel). That's why it's called YV12 :)

[G_M_C]

A slight offtopic question:
Is the VAQ option allready implemented as option in apps/GUI's like MeGUI ?

[Dark Shikari]

A slight offtopic question:
Is the VAQ option allready implemented as option in apps/GUI's like MeGUI ?You can use it through the custom commandline option in MeGUI. Hopefully they'll add an actual GUI option soon.

Its enabled by default.

[Sagittaire]

YV12 uses 12 bits per pixel: 8 bits for Y and 4 bits for CbCr (the chroma is shared between four pixels - 2*8bit/4pix = 4 bits per pixel). That's why it's called YV12 :)

well in fact chroma chanel use 8 bits but with half resolution for chroma -> imply in practice 2 bits per pixel in full resolution.

YV12 is definitively the best compromise for quality/size in losseless/lossy mode.

[MasterNobody]

Dark Shikari
may be replace
float qp_adj = 3 * (logf(energy) - h->rc->aq_threshold);
with
float qp_adj = 3 * (logf(energy) / h->rc->aq_threshold - 1);
I don't test would it be good but QP would change more gradually.

[Atak_Snajpera]

Is the VAQ option allready implemented as option in apps/GUI's like MeGUI ?

RipBot264 has AQ support from very begining.

[Dark Shikari]

RipBot264 has AQ support from very begining.I think his question is whether Ripbot has the VAQ build or a Haali's AQ (HAQ?) build.

[Atak_Snajpera]

Before VAQ it had Haali's but know VAQ is used with default settings

[chipzoller]

Might VAQ make it into the SVN for x264 anytime soon? I notice it's already up to rev. 745 with no mention in the changelog of VAQ.

[Schrade]

Is there an ffmpeg build somewhere with the VAQ patch included?

This build doesn't appear to have it:

http://oss.netfarm.it/mplayer-win32.php

[netsnake]

Might VAQ make it into the SVN for x264 anytime soon? I notice it's already up to rev. 745 with no mention in the changelog of VAQ.

I'm waiting for this too

[vpupkind]

I am seeing that with v0.48 colors look washed out.
I tried aq-strength of [0...0.5] with the default sensitivity, and it looks like the colors become more washed-out as the aq-strength grows.
Has anyone encountered this?

[Dark Shikari]

I am seeing that with v0.48 colors look washed out.
I tried aq-strength of [0...0.5] with the default sensitivity, and it looks like the colors become more washed-out as the aq-strength grows.
Has anyone encountered this?Are you sure you're not just opening up two videos at once in two media players, and so one's luma isn't being converted properly by the renderer?

AQ does not affect color or brightness.

[bob0r]

Might VAQ make it into the SVN for x264 anytime soon? I notice it's already up to rev. 745 with no mention in the changelog of VAQ.

VAQ is still expermimental.

http://x264.nl is updated with GIT (original source) revision 748 + a link to 748 patched, the lack of sleep and some other issues caused me to make it unclear.
12 hours sleep made me look clear again :D ( yeah yeah pengvado, sleep is for pussies )

Anyways here is 748 patched:



Thanks to akupenguin (pengvado) x264.nl is auto updating again!!

So here is 748 (based on GIT updates count) + fixed HRD interlacing patch

x264.748.modified.exe (http://files.x264.nl/x264.748.modified.exe)

General thread:
http://forum.doom9.org/showthread.php?t=130364

x264_aq_var.48.diff
http://forum.doom9.org/showthread.php?t=132760
x264.gaussian.cplxblur.01.diff
Dark Shikari: - gaussian cplxblur: gives a tiny improvement in 2pass ratecontrol
x264_me-prepass_DeathTheSheep.01.diff
http://forum.doom9.org/showthread.php?p=1093523
x264_2pass_vbv.6.diff
http://thread.gmane.org/gmane.comp.video.x264.devel/3093/focus=3550
x264_hrd_pulldown.04_interlace.diff
- HRD and pulldown for HD compatibility, updated patch for interlacing
http://forum.doom9.org/showthread.php?p=1047919#post1047919

Link to x264 patches collected: http://files.x264.nl/x264_patches/

[vpupkind]

Brightness is not influenced, but chroma seems to be.
I tried displaying things in VLC and Nero, running 2-3 instances at the same time, side by side, on several different LCD displays; using both v0.48 / r747 and r736/v0.47.

[Dark Shikari]

Brightness is not influenced, but chroma seems to be.
I tried displaying things in VLC and Nero, running 2-3 instances at the same time, side by side, on several different LCD displays; using both v0.48 / r747 and r736/v0.47.Have you tried *not* displaying them at the same time to avoid renderer issues?

(I highly doubt you're getting a chroma shift from AQ).

[CruNcher]

Are you sure you're not just opening up two videos at once in two media players, and so one's luma isn't being converted properly by the renderer?

AQ does not affect color or brightness.

remember remember the 5th of december ;) *insider* (just ignore this)

[aking80]

Sorry to ask if this has been already but too many pages to read ~_~

Is this patches version the one that MeGUI automatically updates to (763 - Jared's Patched Build)? Also since there's no GUI option for these settings in MeGUI and yet it doesn't show in the command line... I thought this was "on" by default?

[Dark Shikari]

Sorry to ask if this has been already but too many pages to read ~_~

Is this patches version the one that MeGUI automatically updates to (763 - Jared's Patched Build)? Also since there's no GUI option for these settings in MeGUI and yet it doesn't show in the command line... I thought this was "on" by default?Yes, its on by default.

[Dark Shikari]

VAQ 1.0 has been released to Akupenguin for integrating with the official x264.

And here's a preview of VAQ 2.0 Pre-Alpha (links go to videos). The difference is... wow.
SSIM with VAQ 2.0 strength 1.0 (http://www.mediafire.com/?nmzhuturdm2): 0.9092416 (48.1% improvement over no AQ)
SSIM with VAQ 2.0 (http://www.mediafire.com/?vdimcbmzdiz): 0.9016686 (36.7% improvement over no AQ)
SSIM with VAQ 0.48/1.0 (http://www.mediafire.com/?dhastytmz5w): 0.8881315 (13.8% improvement over no AQ)
SSIM with no VAQ (http://www.mediafire.com/?d1byyzumied): 0.8655435

Settings: --bframes 3 --no-b-adapt --bime --weightb --subme 7 --keyint 300 --ref 16 --trellis 2 --mixed-refs --8x8dct --partitions all --b-rdo --direct auto --b-pyramid --pass 2 --bitrate 3700 --aq-strength 0.65 --no-fast-pskip --me tesa
Source: First 200 non-gray frames of parkrun.yuv

Thanks to Alex W for the basic idea behind VAQ 2.0.

API changes in the upcoming VAQ 1.0:

--aq-strength 1.0 is equivalent to the old --aq-strength 0.5. Akupenguin insisted on this. Therefore, 1.0 is now default, and everything else scales accordingly.

--aq-sensitivity no longer exists.

--aq-mode chooses between off (0), auto (1), and static sensitivity (2 [default] ).

Possible API changes in VAQ 2.0 could include:

--aq-metric, which would choose between the following (subject to change on a whim by me based on further developments):
0: Current metric from VAQ 0.48/1.0 (estimated speed: ~500 clocks per MB)
1: Block-based 8x8 window search of 1/4 of the pixels in each block (estimated speed: ~6,000 clocks per MB)
2: 7x7 Gaussian window search of 1/4 of the pixels in each block (estimated speed: ~15,000 clocks per MB)
3: 7x7 Gaussian window search of all the pixels in each block (estimated speed: ~60,000 clocks per MB)

[DeathTheSheep]

If it's as good as 0.46 was, I'm more than happy. :) Well, I'm happy anyway, just saying. :cool:

[J_Darnley]

Very nice work Dark Shikari. I do have a few questions though. With VAQ 1.0 can you turn it off with both --aq-strength 0 and --aq-mode 0? Is the current static mode equivalent/similar to the former static sensitivity? Also what is featured in 2.0 that makes such a large improvement?

Oh yes, any links to 1.0 or 2.0 or are we not allowed give them a test?
[EDIT] Ah I just saw your posts here: http://forum.doom9.org/showthread.php?p=1115512#1115512