Variance AQ Megathread (AQ v0.48 update--defaults changed)

[Sagekilla]

My algorithm will most definitely consider dithered blocks to be very very close to flat, and as a result will decrease their quantizer (though not necessarily by enough to keep the dither accurately).

Oho, that's interesting. Is this something unique to your latest version?

[Dark Shikari]

Oho, that's interesting. Is this something unique to your latest version?No, its inherent in the concept--dithered blocks will have extremely low variance, and so will get the strongest AQ applied.

[burfadel]

Will the latest changes made to x264 with version 717 affect the patch? is it possible for a new build :)?!

[Dark Shikari]

Will the latest changes made to x264 with version 717 affect the patch? is it possible for a new build :)?!717 just looks like an ESA improvement.

[burfadel]

ah ok! I see that now :) wouldn't a 1.3x increase (30 percent) in ESA speed bring it reasonably close to the speed of Multi hex?

[akupenguin]

ah ok! I see that now :) wouldn't a 1.3x increase (30 percent) in ESA speed bring it reasonably close to the speed of Multi hex?
No. Maybe if you also pull in UMH's early termination and range adaption, so that only the multi-hexagon part is replaced by ESA.

[Dark Shikari]

No. Maybe if you also pull in UMH's early termination and range adaption, so that only the multi-hexagon part is replaced by ESA.While we're at it, how much does this speed up ESA SATD?

[Sagekilla]

While we're at it, how much does this speed up ESA SATD?

Pardon my slightly off topic question but isn't the current esa actually a completely different algo? I remember you referring to it as "SEA," what exactly does that stand for?


On a side note, nice to hear that esa got a 30% boost in speed. I've been using esa (along with a number of other insane settings) on relatively short clips and I've been pleased with the (slight, but noticeable) improvement over umh.

[Dark Shikari]

Pardon my slightly off topic question but isn't the current esa actually a completely different algo? I remember you referring to it as "SEA," what exactly does that stand for?

On a side note, nice to hear that esa got a 30% boost in speed. I've been using esa (along with a number of other insane settings) on relatively short clips and I've been pleased with the (slight, but noticeable) improvement over umh.ESA SATD, at least Aku's version, uses SEA.

SEA is Sequential Elimination, and it uses a layered image representation to losslessly eliminate candidates (IIRC).

[Sagekilla]

Also, how compatible are the various speed and quality patches (fast ref, new aq) with rev 715? I'm looking to compile my own build with fast ref search, the older AQ algo (new one is dodgy with crf), and get the nice esa speed boost from the latest patch.

[ToS_Maverick]

just for the record

the new AQ at
--aq-strength 1.0 --aq-sensitivity 17

hast the same size as the old AQ @
--aq-strength 0.6 --aq-sensitivity 10 or --aq-strength 0.9

with all CRFs (tested on 18, 20 and 22)
with a higher visual quality.

[JvA_]

Dark Shikari, could you please make a diff against the latest SVN checkout, if possible?

Except knowledge of C/C++, how much mathematical knowledge is required to start hacking on x264? I have no previous knowledge on video compression, but I've had some courses about fourier-transform and similar math. I've seen JPEG2000 uses a lot of the math I've studied so far, so if MPEG4 has similarities I suppose I would understand a lot ;)

So, what do you recommend? Start digging the source code right away, or are there documents that I should read first that will give me a better "hands on"?

[Dark Shikari]

Dark Shikari, could you please make a diff against the latest SVN checkout, if possible?I don't think there should be any incompatibilities between my old patch and the current SVN. If there are, I'll fix them.
Except knowledge of C/C++, how much mathematical knowledge is required to start hacking on x264? I have no previous knowledge on video compression, but I've had some courses about fourier-transform and similar math. I've seen JPEG2000 uses a lot of the math I've studied so far, so if MPEG4 has similarities I suppose I would understand a lot ;)You don't need much math knowledge. There are dozens of places in x264 that you can start hacking at without even knowing more than a small amount about video compression, since one can easily treat every other part of the code as a "black box" and ignore how it works. This is why I started on me.c and was able to do a few useful things even as a clueless newbie back in the day.

So, what do you recommend? Start digging the source code right away, or are there documents that I should read first that will give me a better "hands on"?Dig through the source code while asking every question you can think of in #x264dev on Freenode. You'll learn faster than you thought you ever could.

[JvA_]

Thanks for your reply!

I tried to patch the source code with your web-published diff using patch -Np0 -i ../path/to/the/file.txt standing in the checked out x264 directory. Got a reject on every line it tried to add. If you got time, please change the diff so it matches. Can't wait to try this AQ :)

[Dark Shikari]

Thanks for your reply!

I tried to patch the source code with your web-published diff using patch -Np0 -i ../path/to/the/file.txt standing in the checked out x264 directory. Got a reject on every line it tried to add. If you got time, please change the diff so it matches. Can't wait to try this AQ :)Are you patching the SVN code, or the web code? The code on mirror05 is already patched with the other AQ, which would result in the errors.

[JvA_]

I'm patching the code I've fetched from:
svn co svn://svn.videolan.org/x264/trunk x264

[Dark Shikari]

New AQ is out. Massive changes.

1. Totally rewritten AQ. Same basic concept, but now uses a logarithmic scale instead of a hackneyed exponential one.
2. For B-frames, uses a tricky bit of lambda-changing instead of QP changing; this requires absolutely no bits for QP-deltas!
3. Totally rewritten, far faster automatic sensitivity. Respects bitrate in CRF mode better also.

Will post it in the original post soon.

[Sagekilla]

Very nice, I'll go try this out and see how well it behaves on my encoding. I've been holding out on a number of movies because I couldn't get decent flat detailed areas with good quality.

[akupenguin]

Pardon my slightly off topic question but isn't the current esa actually a completely different algo? I remember you referring to it as "SEA," what exactly does that stand for?
Technically ESA and SEA are different algorithms. But since they produce the same result and differ only in implementation details, I saw no reason to rename the commandline option when I switched algorithm in r388.

[mitsubishi]

The link for 0.4 doesn't seem to work: "Invalid Quickkey. This error has been forwarded to MediaFire's development team."

[Dark Shikari]

The link for 0.4 doesn't seem to work:Fixed.

[Atak_Snajpera]

Thanks Dark Shikari !
Finally I'm fully convinced to new AQ algorithm :)
No more dancing blocks on blue sky :)

BTW strength 1.0 gives me the best result. I've noticed also that sometimes file size is even lower than 0.5 in CQ mode.

[MasterNobody]

Dark Shikari
May be you will also upload source diff with current x264 trunk so people can compile own's builds (for example, I want to make experimental x264vfw version with this new AQ patch)

[Dark Shikari]

Dark Shikari
May be you will also upload source diff with current x264 trunk so people can compile own's builds (for example, I want to make experimental x264vfw version with this new AQ patch)
Patch version 0.41 (http://pastebin.com/f7fb3770d). 0.41 is just mainly documentation/code cleanup/code comments.

[ToS_Maverick]

Dark Shikari, i got bad news for you:

just did a quick test with these settings:

--crf 20.0 --level 3 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 1835 --vbv-maxrate 10000 --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "output" "input" --aq-strength 1.0
and got the following result
http://img444.imageshack.us/img444/2190/newaqpy2.th.png (http://img444.imageshack.us/my.php?image=newaqpy2.png)

for comparison your 0.3 algo:
http://img256.imageshack.us/img256/7029/aqck8.th.png (http://img256.imageshack.us/my.php?image=aqck8.png)

otherwise, your new AQ looks very promising, but i got the feeling that the grain "stutters" a little bit? like it has only 1/2 or 1/4th of the fps.

[Dark Shikari]

Dark Shikari, i got bad news for you:

just did a quick test with these settings:

--crf 20.0 --level 3 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 1835 --vbv-maxrate 10000 --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "output" "input" --aq-strength 1.0
and got the following result
http://img444.imageshack.us/img444/2190/newaqpy2.th.png (http://img444.imageshack.us/my.php?image=newaqpy2.png)

for comparison your 0.3 algo:
http://img256.imageshack.us/img256/7029/aqck8.th.png (http://img256.imageshack.us/my.php?image=aqck8.png)

otherwise, your new AQ looks very promising, but i got the feeling that the grain "stutters" a little bit? like it has only 1/2 or 1/4th of the fps.Whoa... something went wrong there... :p

Its likely the "stuttering" problem is due to the lambda trick in B-frames, since that really doesn't retain grain very well. I can add a commandline to either do AQ on all frames, AQ on non-B-frames and lambda on B-frames, or AQ on just P-frames.

Edit: Tested your command on the exact same source and didn't get the problem you got...

[DeathTheSheep]

Is this AQ patch applicable on top of hadamard, me-prepass, etc for r720?

[Dark Shikari]

Is this AQ patch applicable on top of hadamard, me-prepass, etc for r720?fpel-cmp seems to be incompatible as of r717, so you'll have to ask Pengvado to fix that.

ME-prepass should continue to be fine. Both should have no problem with AQ, and AQ should be fine with RDRC also.

[DeathTheSheep]

RDRC? Rate distortion rate control? I must have missed this...!

Anyway, I guess I'm backtracking to r680 until the patches gain some steam again. I'm guessing it was the ESA speedup that caused the incompatibilities, and r680 is pretty much the same speed/quality-wise for me without it.

Will this AQ work with 680 on top of the other patches?

PS: I'm not very compelling when it comes to asking people to do things, so a new fpel-cmp feels like a pipe dream. :)

[Dark Shikari]

RDRC? Rate distortion rate control? I must have missed this...!

Anyway, I guess I'm backtracking to r680 until the patches gain some steam again. I'm guessing it was the ESA speedup that caused the incompatibilities, and r680 is pretty much the same speed/quality-wise for me without it.

Will this AQ work with 680 on top of the other patches?

PS: I'm not very compelling when it comes to asking people to do things, so a new fpel-cmp feels like a pipe dream. :)AQ should work fine on r680 as far as I can think.

fpel-cmp just has to be updated--it doesn't have to be rewritten.

RDRC... come on #x264dev, and learn all about the potential 1db+ PSNR gain :p

[ToS_Maverick]

ok, now i reencoded the sample and got 3!!! different file-sizes?!

please reencode the sample and look at the file sizes, i hope you get the same result...

obviously there is something strange happening here

[Dark Shikari]

ok, now i reencoded the sample and got 3!!! different file-sizes?!

please reencode the sample and look at the file sizes, i hope you get the same result...

obviously there is something strange happening here
I just ran it three times... bitwise equivalent result. Something must be wrong with your machine.

[Atak_Snajpera]

I have no problems neither. Core2Duo 1.86GHz overclocked to 2.8GHz :)

[DeathTheSheep]

Overclock? Strange results? Data processing corruption? Hmm, something smells...like burning CPU!! Is your system stable, ToS_Maverick?

Oh and DS, hadamard doesn't work with r716 either. Compiles but crashes (and burns). :)

I'm wondering if r681 is a good choice, now that I look carefully, since it seems to integrate your improved subme7, correct? (As you can see, I want to test this with some insane options. Adaptive quantization goes best with static insanity, does it not?).

[Atak_Snajpera]

Download Go-orthos and check if your cpu is stable (run at least few hours)

[G_M_C]

Whoa... something went wrong there... :p

Its likely the "stuttering" problem is due to the lambda trick in B-frames, since that really doesn't retain grain very well. I can add a commandline to either do AQ on all frames, AQ on non-B-frames and lambda on B-frames, or AQ on just P-frames.

Edit: Tested your command on the exact same source and didn't get the problem you got...

The grainpulsing might be less when you preprocess the source with grainoptimizer (http://forum.doom9.org/showthread.php?p=1052870#post1052870).

[ToS_Maverick]

thank you all for your tips!

if my pc would be instable, many other programs would crash, i should get freezes, BSODs and whatever, but my machine is rock solid. i even underclocked it now to test.

i did a lot of tests now, but could not always get bit identical results.

what i tested:
my machine (C2D 6600@2.33GHz)
AQ 0.4 build - 4 of 6 identical
AQ 0.4 build no AQ - none
rev 720 std - 2 of 3 identical

father's machine (Athlon X2 3800 all std)
AQ 0.4 build - none
rev 720 std - none

the identical files seem to be related with the current system load. while doing nothing during the encoding process, the files have a higher chance to get identical. it's very strange that the athlon doesn't produce identical files. shouldn't a program deliver correct results, no matter what runs nearby?

maybe someone else has a better explanation to this? and sry for being a bit off topic.

[akupenguin]

Before trying to debug nondeterministic r680 (if that's what you're doing), read r713 (http://trac.videolan.org/x264/changeset/713). That wouldn't cause colored snow like ToS_Maverick saw, but it was a bug.

[Dark Shikari]

Two bugs have been discovered.

1. The modified lambda system doesn't handle chroma QPs properly, which causes breakage at high QPs. This has been fixed in my latest internal build. It will be uploaded soon. In the meantime, here's the updated patch (http://pastebin.com/f499214d3).

2. The deadzone lambda changing is somewhat broken. Using trellis is recommended until this is fixed.

[Razorholt]

trellis 1 or 2? Doesn't matter?

Thanks
- Dan

[Dark Shikari]

trellis 1 or 2? Doesn't matter?

Thanks
- Dan2 will completely eliminate the use of deadzone, ensuring no problems ever. 1 is probably sufficient.

[DeathTheSheep]

Aku: Then could you update your SATD patch for 720? It would make testing this more beneficial in the extreme scenarios.

Also, Dark Shikari: How about a refresh of that me-prepass? Last I checked it was thrown all over the place (b0rked diffs, missing lines, updates, etc)?

'Twould truly be much appreciated on my part. :)

[desta]

So using this AQ + deadzones is a no go (for now)?

[Dark Shikari]

Use this new patch:
]Index: encoder/encoder.c
===================================================================
--- encoder/encoder.c (revision 720)
+++ encoder/encoder.c (working copy)
@@ -374,7 +374,7 @@
h->param.analyse.i_direct_mv_pred = X264_DIRECT_PRED_SPATIAL;
}
}
-
+
if( h->param.rc.i_rc_method < 0 || h->param.rc.i_rc_method > 2 )
{
x264_log( h, X264_LOG_ERROR, "no ratecontrol method specified\n" );
@@ -472,6 +472,8 @@
if( !h->param.b_cabac )
h->param.analyse.i_trellis = 0;
h->param.analyse.i_trellis = x264_clip3( h->param.analyse.i_trellis, 0, 2 );
+ if( h->param.analyse.b_aq && h->param.analyse.f_aq_strength <= 0 )
+ h->param.analyse.b_aq = 0;
h->param.analyse.i_noise_reduction = x264_clip3( h->param.analyse.i_noise_reduction, 0, 1<<16 );

{
@@ -1020,6 +1022,32 @@
x264_macroblock_slice_init( h );
}

+//Finds the total AC energy of the block in all planes.
+static int ac_energy_mb(x264_t *h)
+{
+ DECLARE_ALIGNED( static uint8_t, zero[16], 16 );
+ int sad = h->pixf.sad[PIXEL_16x16](zero,0,h->mb.pic.p_fenc[0],FENC_STRIDE) >> 4;
+ int ssd = h->pixf.ssd[PIXEL_16x16](zero,0,h->mb.pic.p_fenc[0],FENC_STRIDE);
+ int totalSSD = ssd - (sad * sad);
+ sad = h->pixf.sad[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[1],FENC_STRIDE) >> 3;
+ ssd = h->pixf.ssd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[1],FENC_STRIDE);
+ totalSSD += ssd - (sad * sad);
+ sad = h->pixf.sad[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[2],FENC_STRIDE) >> 3;
+ ssd = h->pixf.ssd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[2],FENC_STRIDE);
+ totalSSD += ssd - (sad * sad);
+ return totalSSD;
+}
+
+//Find the total SATD score of a block. Represents the block's overall complexity (bit cost) for intra encoding.
+static int satd_mb(x264_t *h)
+{
+ DECLARE_ALIGNED( static uint8_t, zero[16], 16 );
+ int totalSATD = h->pixf.satd[PIXEL_16x16](zero,0,h->mb.pic.p_fenc[0],FENC_STRIDE);
+ totalSATD += h->pixf.satd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[1],FENC_STRIDE);
+ totalSATD += h->pixf.satd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[2],FENC_STRIDE);
+ return totalSATD;
+}
+
static void x264_slice_write( x264_t *h )
{
int i_skip;
@@ -1045,7 +1073,51 @@
}
h->mb.i_last_qp = h->sh.i_qp;
h->mb.i_last_dqp = 0;
-
+ x264_cpu_restore(h->param.cpu);
+ /* Adaptive AQ sensitivity algorithm. */
+ if(h->param.analyse.f_aq_sensitivity == 0 && h->param.analyse.f_aq_strength != 0)
+ {
+ double total = 0;
+ double n = 0;
+ /* FIXME: Easier way to iterate over MBs? Do we need to do the full cache_load? */
+ /* FIXME: Some of the SATDs might be already calculated elsewhere (ratecontrol?). Can we reuse them? */
+ /* FIXME: Store the data, then do the logs after, to avoid the cpu_restores every single cycle? */
+ /* FIXME: Is chroma SATD necessary? */
+ for( mb_xy = h->sh.i_first_mb; mb_xy < h->sh.i_last_mb; )
+ {
+ const int i_mb_y = mb_xy / h->sps->i_mb_width;
+ const int i_mb_x = mb_xy % h->sps->i_mb_width;
+ x264_macroblock_cache_load( h, i_mb_x, i_mb_y );
+ int energy = ac_energy_mb(h);
+ x264_cpu_restore(h->param.cpu);
+ /* Weight the energy value by the SATD value of the MB. This represents the fact that
+ the more complex blocks in a frame should be weighted more when calculating the optimal sensitivity.
+ This also helps diminish the negative effect of large numbers of simple blocks in a frame, such as in the case
+ of a letterboxed film. */
+ if(energy != 0)
+ {
+ int satd = satd_mb(h);
+ x264_cpu_restore(h->param.cpu);
+ total += log(energy) * satd;
+ n += satd;
+ }
+ if( h->sh.b_mbaff )
+ {
+ if( (i_mb_y&1) && i_mb_x == h->sps->i_mb_width - 1 )
+ mb_xy++;
+ else if( i_mb_y&1 )
+ mb_xy += 1 - h->sps->i_mb_width;
+ else
+ mb_xy += h->sps->i_mb_width;
+ }
+ else
+ mb_xy++;
+ }
+ x264_cpu_restore(h->param.cpu);
+ /* Calculate and store the threshold. */
+ if(n == 0) h->aq_threshold = 100000;
+ else h->aq_threshold = expf(total / n);
+ }
for( mb_xy = h->sh.i_first_mb, i_skip = 0; mb_xy < h->sh.i_last_mb; )
{
const int i_mb_y = mb_xy / h->sps->i_mb_width;
Index: encoder/analyse.c
===================================================================
--- encoder/analyse.c (revision 720)
+++ encoder/analyse.c (working copy)
@@ -29,6 +29,7 @@
#endif

#include "common/common.h"
+#include "common/cpu.h"
#include "macroblock.h"
#include "me.h"
#include "ratecontrol.h"
@@ -2037,8 +2038,68 @@
}
}

+//Finds the total AC energy of the macroblock in all planes.
+static int ac_energy_mb(x264_t *h)
+{
+ DECLARE_ALIGNED( static uint8_t, zero[16], 16 );
+ int sad = h->pixf.sad[PIXEL_16x16](zero,0,h->mb.pic.p_fenc[0],FENC_STRIDE) >> 4;
+ int ssd = h->pixf.ssd[PIXEL_16x16](zero,0,h->mb.pic.p_fenc[0],FENC_STRIDE);
+ int totalSSD = ssd - (sad * sad);
+ sad = h->pixf.sad[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[1],FENC_STRIDE) >> 3;
+ ssd = h->pixf.ssd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[1],FENC_STRIDE);
+ totalSSD += ssd - (sad * sad);
+ sad = h->pixf.sad[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[2],FENC_STRIDE) >> 3;
+ ssd = h->pixf.ssd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[2],FENC_STRIDE);
+ totalSSD += ssd - (sad * sad);
+ return totalSSD;
+}

/*****************************************************************************
+* x264_adaptive_quant:
+ * adjust macroblock QP based on variance (AC energy) of the MB.
+ * high variance = higher QP
+ * low variance = lower QP
+ * This generally increases SSIM and lowers PSNR.
+ * To save bits in B-frames, adaptive lambda is used instead of adaptive quantization.
+*****************************************************************************/
+void x264_adaptive_quant( x264_t *h, x264_mb_analysis_t *a )
+{
+ int qp = h->mb.i_qp;
+ int ac_energy = ac_energy_mb(h);
+ x264_cpu_restore(h->param.cpu);
+ float result = ac_energy;
+ float threshold;
+ /* In the case of adaptive AQ sensitivity, grab the value from the frame pre-process. Otherwise, calculate
+ the AQ sensitivity value for the current frame. */
+ if(h->param.analyse.f_aq_sensitivity == 0)
+ threshold = h->aq_threshold;
+ else threshold = powf(h->param.analyse.f_aq_sensitivity,4)/2;
+ /* Adjust the QP based on the AC energy of the macroblock. */
+ int qp_adj = -3.0 * h->param.analyse.f_aq_strength * log(result / threshold);
+ qp_adj = x264_clip3(qp_adj,-5*h->param.analyse.f_aq_strength,5*h->param.analyse.f_aq_strength);
+ int new_qp = x264_clip3(qp - qp_adj,h->param.rc.i_qp_min,h->param.rc.i_qp_max);
+ /* Change the lambda values. */
+ /* If the QP of this MB is within 1 of the previous MB, code the same QP as the previous MB, to lower the bit
+ cost of the qp_delta. */
+ if(abs(new_qp - h->mb.i_last_qp) == 1) new_qp = h->mb.i_last_qp;
+ a->i_lambda = i_qp0_cost_table[new_qp];
+ a->i_lambda2 = i_qp0_cost_table[new_qp];
+ //h->i_mod_qp = new_qp;
+ //h->i_mod_chroma_qp = i_chroma_qp_table[x264_clip3( new_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
+ /* Adaptive quantization only applies to I/P frames. Applying it to B-frames generally results in a lot of
+ unnecessary bits spent on delta_qp. Instead, the lambdas are changed.
+ FIXME: Choose whether to use adaptive lambda or adaptive quantization on a per-block basis?
+ FIXME: Choose which to use based on bit cost?
+ FIXME: Is this optimal? */
+ //if(h->sh.i_type != SLICE_TYPE_B )
+ {
+ /* Save the final QP and update the chroma QP. */
+ h->mb.i_qp = a->i_qp = new_qp;
+ h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
+ }
+}
+
+/*****************************************************************************
* x264_macroblock_analyse:
*****************************************************************************/
void x264_macroblock_analyse( x264_t *h )
@@ -2046,9 +2107,19 @@
x264_mb_analysis_t analysis;
int i_cost = COST_MAX;
int i;
+
+ h->mb.i_qp = x264_ratecontrol_qp( h );
+
+ if( h->param.analyse.b_aq )
+ x264_adaptive_quant( h, &analysis );
+ //else
+ //{
+ // h->i_mod_qp = h->mb.i_qp;
+ // h->i_mod_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
+ //}

/* init analysis */
- x264_mb_analyse_init( h, &analysis, x264_ratecontrol_qp( h ) );
+ x264_mb_analyse_init( h, &analysis, h->mb.i_qp );

/*--------------------------- Do the analysis ---------------------------*/
if( h->sh.i_type == SLICE_TYPE_I )
Index: x264.c
===================================================================
--- x264.c (revision 720)
+++ x264.c (working copy)
@@ -243,6 +243,14 @@
" - 2: enabled on all mode decisions\n", defaults->analyse.i_trellis );
H0( " --no-fast-pskip Disables early SKIP detection on P-frames\n" );
H0( " --no-dct-decimate Disables coefficient thresholding on P-frames\n" );
+ H0( " --aq-strength <float> Amount to adjust QP/lambda per MB [%.1f]\n"
+ " 0.0: no AQ\n"
+ " 0.7: medium AQ\n"
+ " 1.4: strong AQ\n", defaults->analyse.f_aq_strength );
+ H0( " --aq-sensitivity <float> \"Center\" of AQ curve. [%.1f]\n"
+ " 0: automatic sensitivity (recommended)\n"
+ " 5: almost all QPs are raised\n"
+ " 35: almost all QPs are lowered\n", defaults->analyse.f_aq_sensitivity );
H0( " --nr <integer> Noise reduction [%d]\n", defaults->analyse.i_noise_reduction );
H1( "\n" );
H1( " --deadzone-inter <int> Set the size of the inter luma quantization deadzone [%d]\n", defaults->analyse.i_luma_deadzone[0] );
@@ -406,6 +414,8 @@
{ "trellis", required_argument, NULL, 't' },
{ "no-fast-pskip", no_argument, NULL, 0 },
{ "no-dct-decimate", no_argument, NULL, 0 },
+ { "aq-strength", required_argument, NULL, 0 },
+ { "aq-sensitivity", required_argument, NULL, 0 },
{ "deadzone-inter", required_argument, NULL, '0' },
{ "deadzone-intra", required_argument, NULL, '0' },
{ "level", required_argument, NULL, 0 },
Index: common/common.c
===================================================================
--- common/common.c (revision 720)
+++ common/common.c (working copy)
@@ -123,6 +123,9 @@
param->analyse.i_chroma_qp_offset = 0;
param->analyse.b_fast_pskip = 1;
param->analyse.b_dct_decimate = 1;
+ param->analyse.b_aq = 0;
+ param->analyse.f_aq_strength = 0.0;
+ param->analyse.f_aq_sensitivity = 0;
param->analyse.i_luma_deadzone[0] = 21;
param->analyse.i_luma_deadzone[1] = 11;
param->analyse.b_psnr = 1;
@@ -455,6 +458,13 @@
p->analyse.b_fast_pskip = atobool(value);
OPT("dct-decimate")
p->analyse.b_dct_decimate = atobool(value);
+ OPT("aq-strength")
+ {
+ p->analyse.f_aq_strength = atof(value);
+ p->analyse.b_aq = (p->analyse.f_aq_strength > 0.0);
+ }
+ OPT("aq-sensitivity")
+ p->analyse.f_aq_sensitivity = atof(value);
OPT("deadzone-inter")
p->analyse.i_luma_deadzone[0] = atoi(value);
OPT("deadzone-intra")
@@ -939,6 +949,9 @@
s += sprintf( s, " zones" );
}

+ if( p->analyse.b_aq )
+ s += sprintf( s, " aq=1:%.1f:%.1f", p->analyse.f_aq_strength, p->analyse.f_aq_sensitivity );
+
return buf;
}

Index: common/common.h
===================================================================
--- common/common.h (revision 720)
+++ common/common.h (working copy)
@@ -232,6 +232,10 @@

struct x264_t
{
+ float aq_threshold;
+ //int i_mod_qp;
+ //int i_mod_chroma_qp;
+
/* encoder parameters */
x264_param_t param;

Index: x264.h
===================================================================
--- x264.h (revision 720)
+++ x264.h (working copy)
@@ -230,6 +230,9 @@
int i_trellis; /* trellis RD quantization */
int b_fast_pskip; /* early SKIP detection on P-frames */
int b_dct_decimate; /* transform coefficient thresholding on P-frames */
+ int b_aq; /* psy adaptive QP */
+ float f_aq_strength;
+ float f_aq_sensitivity;
int i_noise_reduction; /* adaptive pseudo-deadzone */

/* the deadzone size that will be used in luma quantization */
This should compile (I haven't tried it) and should have absolutely no potential for bugs/breakage. It just entirely disables the lambda-based AQ for the meantime, since its only on B-frames and not really a big deal.

[Dark Shikari]

New executable uploaded. Removing the lambda AQ didn't seem to have much of a negative or positive effect for now, but will fix most of the bugs with this version. I may add it back in later.

[Dark Shikari]

I have posted results in the OP of a test of test of the new AQ. Yes, that's right, roughly 21.5% bitrate-adjusted SSIM improvement. :eek:

[acrespo]

I didn't use version 0.4 but v0.42 crashes in Vista x64. When I execute I receive a message from windows that the pthreadGC2.dll is missing. I returned to version 0.3 and don't have problems.

[Dark Shikari]

I didn't use version 0.4 but v0.42 crashes in Vista x64. When I execute I receive a message from windows that the pthreadGC2.dll is missing. I returned to version 0.3 and don't have problems.That's because... you need pthreadGC2.dll?

You can get it here (http://mirror05.x264.nl/Dark/force.php?file=./pthreadGC2.dll).

[ToS_Maverick]

you somehow forgot to replace the link, it still points to 0.4
0.42 is there but not labeled as .exe:
http://mirror05.x264.nl/Dark/force.php?file=./x264_Experimental_AQ_0.42

[CruNcher]

@Dark Shikari
Great work, you really solved parts of the Banding Problem with this, im amazed by the results :)

Without Dark Shikaris Magic AQ
http://rapidshare.com/files/84197179/testseq-pearl-nodarkaq.mkv

With Dark Shikaris Magic AQ (HVS Quality is greatly improved)
http://rapidshare.com/files/84197485/testseq-pearl-darkaq.mkv

Get it now, it makes your "darkest" dreams come true ;D

i tested an older version i think of this tough (found the patch in your folder) with --aq-strength 0.9, bellow that it wouldn't look good enough and --aq-strength 1.0 coused a problem (gonna retest with the new patch) in this test cut :)

and yes im as crazy as you where with Vendeta 3 Mbit for 1080p ;)

Here is the Bug with --aq-strength 1.0 (also happens for --aq-strength 0.9 if --no-dct-decimate is used), excuse me if this problem allready has been encountered before i didn't read the whole thread yet (and maybe it's not even happening with the new patch anymore)
http://rapidshare.com/files/84200572/testseq-pearl-darkaq-bug.mkv (look @ the top left when the calendar pages are turned)