MeGUI and Adaptive Q parms: Any chance ?

[MarcioAB]

Is there any chance to see adaptive Q parms (--aq-strength and --aq-sensitivity) on MeGUI ?

Thank you.

Obs: there must be dozens of such request here, but I was not able to found any direct answer using search by "aq-strength".

[RaynQuist]

you can add it to the command line on the second tab of video config

[Atak_Snajpera]

Marcio send your request here
http://forum.doom9.org/showthread.php?p=1043540#post1043540

[Sharktooth]

megui wont "officially" support unofficial encoders features

[Atak_Snajpera]

In terms of dark and blue areas x264 is too agressive! Before AQ patch I was surprised when the same sample encoded with XviD looked better. You guys in MeGUI world are making big mistake not including AQ. Don't be surprised if people will keep you asking "Why my movie is so blocky even if I use high bitrate?" I use --aq-strength 0.5 --aq-sensitivity 5 and I have always constant quality in all areas.

[Sharktooth]

the AQ patch is simply a workaround that reduces image quality if favour of less blocking. if the AQ patch was a good solution it would have been included in x264...
however the x264 build that comes with megui supports it and there is a custom commandline textbox made for that purpouse...

[Atak_Snajpera]

So explain me Why XviD looks better (less blocky) despite fact it does not have deblocking? Compression was made to fool human senses and I don't care about number like PSNR or SSIM. I only care if I see artifacts or dont. Even my settings do not increase bitrate that much.

however the x264 build that comes with megui supports it and there is a custom commandline textbox made for that purpouse...

yeah! And of course every user knows that... ;)

[check]

Post the x264 and xvid settings you are using, as well as the log output from both encodes. Are you running any postprocessing on the video?

[Atak_Snajpera]

Ok I will post two samples encoded from HD cam (1440x1080) resampled to 1280x720 @6MBits . I need some time... I will be back soon...

[Sharktooth]

So explain me Why XviD looks better (less blocky) despite fact it does not have deblocking? Compression was made to fool human senses and I don't care about number like PSNR or SSIM. I only care if I see artifacts or dont. Even my settings do not increase bitrate that much.



yeah! And of course every user knows that... ;)
the users that know about AQ also know the commandline to set it. however the "blockiness" only happens when you use inappropriate settings. cranking everything up to the max is NOT the best quality solution...

[Atak_Snajpera]

Script
#MT
SetMTmode(2,0)

#Source
LoadPlugin("C:\Users\Dawidos\Documents\Delphi_Projects\RipBot264\Tools\dgindex\DGDecode.dll")
video=MPEG2Source("c:\Users\Dawidos\AppData\Local\Temp\RipBot264temp\job1\job1.d2v")

#Deinterlace
LoadCplugin("C:\Users\Dawidos\Documents\Delphi_Projects\RipBot264\Tools\AviSynth plugins\Yadif\Yadif.dll")
video=yadif(video,mode=0,order=1)

#Resize
video=Spline36Resize(video,1280,720).Sharpen(0.2)

Return video

Xvid
Quant - h.263
Motion Search Precision - Ultra High
VHQ - Mode Decision
Chroma Motion
http://www.mediafire.com/?6h2byl0dhgy

x264
--pass 2 --bitrate 6144 --stats ".stats" --level 5.1 --sar 1:1 --filter 0,0 --ref 5 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --subme 6 --trellis 1 --analyse all --8x8dct --me umh --threads auto --thread-input --progress --no-psnr --no-ssim --output "video.264"
http://www.mediafire.com/?5mmjml3lwcd

x264 --aq-strength 0.5 --aq-sensitivity 5
http://www.mediafire.com/?33nbedzcenn

[Terranigma]

*sigh*, I'm unfortunately unable to download the x264 aq version since mediafire's telling me to try back later, so I was only able to grab the first 2. Call me blind, but I don't see any blocking with the normal x264 encode. All I see is it not being detailed in certain areas, like by the trees. This can easily be remedied by modifying --ratetol

[Atak_Snajpera]

Xvid
http://img299.imageshack.us/img299/3985/xvidpj7.th.png (http://img299.imageshack.us/my.php?image=xvidpj7.png)

x264 AQ
http://img503.imageshack.us/img503/8449/x264aqpb0.th.png (http://img503.imageshack.us/my.php?image=x264aqpb0.png)

x264 NO AQ
http://img410.imageshack.us/img410/1031/x264noaquf5.th.png (http://img410.imageshack.us/my.php?image=x264noaquf5.png)

Still nothing?

[Terranigma]

Nope, there's no signs of blocks. Maybe you can highlight just what you're referring to as blocking?
I think the command you're looking for is --ratetol, not --aq-strength or --aq-sensitivity

[Sharktooth]

you can always lower the settings to remove the blocks (expecially B-RDO in conjuncton with direct auto) and still get better quality than using AQ...
however i cant see any blocks or any other artifacts in your screenshots except some blurring in the AQ pic...
maybe your monitor has WAY TO MUCH contrast or is not well calibrated.

[Atak_Snajpera]

I have 19'' LG widescreen monitor connected via DVI. It's also well calibrated. White is not too bright and black is black. Besides I don't see blocks in Xvid sample and AQ sample but I do see in NOAQ. BTW I use HQRGB32 in FFDShow.

Terranigma look at the sky.

[Terranigma]

Terranigma look at sky.
Nope, still nothing. Can anyone else confirm that there's blocks visible from the no-aq screenshot?

[Sharktooth]

There are indeed some blocks but i can spot them only by torturing the image with 200% contrast and -130% brightness... this way:

no AQ
http://img516.imageshack.us/img516/5300/x264noaquf5jn0.th.png (http://img516.imageshack.us/my.php?image=x264noaquf5jn0.png)

AQ
http://img516.imageshack.us/img516/7352/x264aqpb0dp2.th.png (http://img516.imageshack.us/my.php?image=x264aqpb0dp2.png)
but xvid shows even more blocks...

xvid
http://img61.imageshack.us/img61/2407/xvidpj7gf2.th.png (http://img61.imageshack.us/my.php?image=xvidpj7gf2.png)

[Atak_Snajpera]

Hey You selected wrong screen shot with AQ :)

Xvid with gamma 0.4
http://img401.imageshack.us/img401/6223/xvidgamma04ps9.th.png (http://img401.imageshack.us/my.php?image=xvidgamma04ps9.png)

x264 AQ
http://img401.imageshack.us/img401/3941/x264aqgamma04bq7.th.png (http://img401.imageshack.us/my.php?image=x264aqgamma04bq7.png)

x264 NOAQ
http://img401.imageshack.us/img401/9393/x264noaqgamma04cn9.th.png (http://img401.imageshack.us/my.php?image=x264noaqgamma04cn9.png)

Xvid and x264 AQ are very similar (blocks have the same size) but NOAQ is a disaster

[Sharktooth]

fixed...
maybe brightness is not the same for all the images but the amount of blocking is more or less the same.

[Atak_Snajpera]

Can't you see difference in size of blocks? We are not talking about brightness but about size of blocks. Xvid as you can see is less agressive in this case. Try to fix that without AQ :)

[Terranigma]

Can't you see difference in size of blocks?
I can see it now, but is it really all that bad?
Can you now do another test without adaptive quantization, using b-rdo and direct spatial instead of auto? Is this what you were saying sharktooth, or should he disable b-rdo as well?

[Sharktooth]

there are cases where b-rdo and direct auto may block.
try with direct spatial (or temporal) + brdo and with direct auto without brdo

[Atak_Snajpera]

NO AQ and Subme 5 (gamma 0.4)
http://img210.imageshack.us/img210/6482/x264noaqsubme5zm5.th.png (http://img210.imageshack.us/my.php?image=x264noaqsubme5zm5.png)

[Atak_Snajpera]

I can see it now, but is it really all that bad?
If I see difference without playing with Gamma It means that I have better monitor or it's better set than yours.
Like I said before. I see blocks only in NOAQ sample during playback! I don't see them in Xvid and in AQ.

[MarcioAB]

megui wont "officially" support unofficial encoders features

x264 is very good. It was necessary just 1 day of test to move from DivX to x264, but my #1 (and #2) issue with x264 is blockness on dark areas - that I discovered is an old and much known issue with x264.

And I learned in this forum that adaptive Q is the solution for blockness. In parallel I was glad to see that in a recent MeGUI auto-update, x264 became "667b -Cef's build", the one that supports adaptive Q. Is Cef's build "official" ? If MeGUI download "non-official" x264, why not fully interface with ? If Cef's build is "official", then adaptive Q is there.

In just a few tests with adaptive Q I (almost) can confirm: It is the way to go for me (but still testing).

I hope more people embrace MeGUI development ! (and I hope to see adaptive Q parms in it).

[MarcioAB]

Marcio send your request here
http://forum.doom9.org/showthread.php?p=1043540#post1043540

Thank you: already there.

[Sharktooth]

If I see difference without playing with Gamma It means that I have better monitor or it's better set than yours.not at all. having too much contrast and low brightness does not mean it's better set or that you have a better monitor. you have just something that looks as it should not look. also the AQ screen shows blocking as well and it's more blurry. i cant really see the reasons to use AQ at all for that source.

@marcio: your request has been already rejected. Also MeGUI always came with AQ patched x264 builds (usually mines) but AQ is not an official thing, so megui wont officially support it (but you also know you can specify custom command line options ant that they will be also saved in profiles).
FYI megui once had AQ support... it has just been removed.

[Atak_Snajpera]

also the AQ screen shows blocking as well and it's more blurry

So what? Have I ever said that AQ path removes blocks? No! AQ patch by increasing bitrate reduces size of the blocks! Smaller blocks are harder to spot. My setting produces almost the same size of blocks as Xvid so I don't see them.

I am almost sure that your monitor does not output full colors. I checked NOAQ screenshot on my 2 years old laptop and indeed blocks are not visible but they appear on my brand new LG monitor and on my friend's 40'' Samsung LCD TV. And believe me LG gives me a lot better colors than old laptop. In both cases I've calibrated colors the best as I could so stop saying about to much contrast and so on. Everything is set perfectly!

Try to compare DVD movie or DVDRips (without deblocking) on CRT TV and then on Good LCD. Sudently all blocks are visible along with noise and ringing. The same story applies to LCDs. Not every LCD has the same quality. (PVA vs TN)

[foxyshadis]

I suspect this is exacerbated if you have a 6-bit monitor. Calibration doesn't help if it just can't show everything and creates banding (this is why Sharktooth pulled the eqm_ulr matrix, after all). That one dark block is more noticeable in the non-aq, but you have to have an old/cheap lcd to see all the sky blocking.

Since MeGUI packages x264 it's not like most people will ever externally update it, except during long breaks. MeGUI should have support for a recommended feature enabled in all of its recommended builds, at least in certain profiles. (Scan for "unrecognized option `--aq-strength`" when you start it, if found, don't log and remake the command-line with AQ disabled.) I accept that the devs might not do this without a patch to support it though.

[Sharktooth]

we ensured that megui works with x264 SVN builds so if the auto-update server doesnt get... well... updated, any user can replace the x264 executable with a standard one build using x264 SVN sources.
when and if AQ will be commited to x264 SVN then we will support it.

[MarcioAB]

AQ patch by increasing bitrate reduces size of the blocks!

"reduce blocks" => reduce the size of the blocks -- in contrast with reduce the number of blocks: Fair. (EDIT: IMO, reduce the size of the blocks is better than reduce the number of the blocks).

I tried to "reduce blocks" by reasonable increase of bitrate without going into AQ and was not very successful.

With AQ, the bitrate increased but also the blocks became "reduced" (smaller in size, less visible).

obs: x264 blocks in slow motion dark-area is cruel: beside exist, they "dance", calling even more the attention.

I see the blocks in 1:1 resolution (720x480) but specially on full screen resolution. LCD monitor is Acer 20" (1680x1050) with GeForce 8600 GTS. The same when signal goes to "tv" that is Sony LCD 40" (1366x768).

[deets]

from my utterly untrained eye, if i load up the images in tabs then flick between, the sky blocks do become clear.

if anyone really wants it they can use ripbot264 ;) part of the reason i now use that over megui

[Atak_Snajpera]

Let's assume we have 2 blocks. One has R:205 G:218 B:220 color and other next to him has 204 216 221. Should I see small difference between or not? On my monitor I do see. It's not big but I'm able to see edge between those two. It's even more visible when block is moving. Since those blocks have different color values you should be able to see it too. If you can't It means that monitor is not able to produce full color range.

[Sharktooth]

in mpeg codecs colors are not expressed with RGB and the color range is really restricted. so in most cases even the average monitor has more colors than the mpeg colorspace has...
however, AQ gives bits to dark/blue areas, so, in your case, the bright sky has LESS bits...

[Atak_Snajpera]

AQ gives bits to dark/blue areas, so, in your case, the bright sky has LESS bits...

Less bits you say...Hmmm intresting... That's not true in my case. If blocks on the sky are smaller it means that they got more bits not less!

[Sharktooth]

if there isnt enough blue it will simply "return"... so that's it, the bright sky should have less or at max the same bits.
also AQ wont affect the picture if QP <= 10.

if( h->pixf.sad[PIXEL_16x16](h->mb.pic.p_fenc[0], FENC_STRIDE, zero, 16) > 64*16*16 )
{ /* light places */
if( h->pixf.count_8x8(h->mb.pic.p_fenc[1], FENC_STRIDE, 0x81818181) < 40 )
/* not enough "blue" pixels */
return;

if( h->pixf.count_8x8(h->mb.pic.p_fenc[2], FENC_STRIDE, 0x87878787) > 24 )
/* too many "red" pixels */
return;
}

[Atak_Snajpera]

That's funny but my common sense tells me that bright sky got more bits therefore blocks are smaller. According to x264 cli help sensitivity 0.5 means that AQ patch is applied to almost all blocks.

[Dark Shikari]

AQ gives more bits to flat areas in general, IIRC, not just dark/blue.

[Sharktooth]

@Dark Shikari: check the code snippet i attached above...

[Atak_Snajpera]

Finally somebody who knows what he says :)

Dark Shikari have also noticed big blocks in my NOAQ screenshot? I mean with normal gamma.

[Sharktooth]

Dark Shikari probably didnt look at the AQ patch...

[Atak_Snajpera]

You claim that sky got less bits but by a miracle sky in AQ example got more bits therefore it's more detailed. If you understand source code that well explain me how come?

[Sharktooth]

do you have ANY proof sky has more bits?
the AQ screen looks bad for me...
the source is really simple, there's a treshold... if that threshold isnt hit then there's no change (it's also commented!).

[Atak_Snajpera]

Yes I've got. Blocks are about 4 times smaller :) . How are you going to achieve that without using extra bits? If you still don't believe me I have very simple example specially for you. First screenshot shows jpg image @ 25% (lower compression) and second is compressed to 75% (higher compression). If you use less bits images get less detailed and blocks are getting bigger.

http://img524.imageshack.us/img524/5921/25yw3.th.jpg (http://img524.imageshack.us/my.php?image=25yw3.jpg)

http://img378.imageshack.us/img378/5033/75vm3.th.jpg (http://img378.imageshack.us/my.php?image=75vm3.jpg)

[Sharktooth]

you cant compare apples and peaches...
the AQ patch source code is there for everyone. have a nice reading...
Index: encoder/encoder.c
===================================================================
--- encoder/encoder.c (revision 598)
+++ encoder/encoder.c (working copy)
@@ -457,6 +457,8 @@
if( !h->param.b_cabac )
h->param.analyse.i_trellis = 0;
h->param.analyse.i_trellis = x264_clip3( h->param.analyse.i_trellis, 0, 2 );
+ if( h->param.analyse.b_aq && h->param.analyse.f_aq_strength <= 0 )
+ h->param.analyse.b_aq = 0;
h->param.analyse.i_noise_reduction = x264_clip3( h->param.analyse.i_noise_reduction, 0, 1<<16 );

{
Index: encoder/analyse.c
===================================================================
--- encoder/analyse.c (revision 598)
+++ encoder/analyse.c (working copy)
@@ -28,6 +28,7 @@
#include <limits.h>

#include "common/common.h"
+#include "common/cpu.h"
#include "macroblock.h"
#include "me.h"
#include "ratecontrol.h"
@@ -1999,8 +2000,73 @@
}
}

+static void x264_add_dctq(int16_t dct[8][8], int *p_total) {
+ int i, t = 0;
+ int16_t *p = &dct[0][0];

+ for (i = 1; i < 64; ++i)
+ t += abs(p[i]) * x264_dct8_weight_tab[i];
+
+ *p_total += t;
+}
+
/*****************************************************************************
+ * x264_adaptive_quant:
+ * check if mb is "flat", i.e. has most energy in low frequency components, and
+ * adjust qp down if it is
+ *****************************************************************************/
+void x264_adaptive_quant( x264_t *h, x264_mb_analysis_t *a )
+{
+ DECLARE_ALIGNED( static uint8_t, zero[FDEC_STRIDE*8], 16 );
+
+ int16_t dct[8][8];
+ int total = 0;
+ int i_qp = h->mb.i_qp, i_qp_adj;
+ float fc;
+
+ if( i_qp <= 10 ) /* AQ is probably not needed at such low QP */
+ return;
+
+ if( h->pixf.sad[PIXEL_16x16](h->mb.pic.p_fenc[0], FENC_STRIDE, zero, 16) > 64*16*16 )
+ { /* light places */
+ if( h->pixf.count_8x8(h->mb.pic.p_fenc[1], FENC_STRIDE, 0x81818181) < 40 )
+ /* not enough "blue" pixels */
+ return;
+
+ if( h->pixf.count_8x8(h->mb.pic.p_fenc[2], FENC_STRIDE, 0x87878787) > 24 )
+ /* too many "red" pixels */
+ return;
+ }
+
+ h->dctf.sub8x8_dct8( dct, h->mb.pic.p_fenc[0], zero );
+ x264_add_dctq( dct, &total );
+ h->dctf.sub8x8_dct8( dct, h->mb.pic.p_fenc[0] + 8, zero );
+ x264_add_dctq( dct, &total );
+ h->dctf.sub8x8_dct8( dct, h->mb.pic.p_fenc[0] + 8*FENC_STRIDE, zero );
+ x264_add_dctq( dct, &total );
+ h->dctf.sub8x8_dct8( dct, h->mb.pic.p_fenc[0] + 8*FENC_STRIDE + 8, zero );
+ x264_add_dctq( dct, &total );
+
+ if( total == 0 ) /* sum is 0, nothing to do */
+ return;
+
+ x264_cpu_restore( h->param.cpu );
+
+ fc = (float)expf(-5e-13 * total * total);
+
+ //printf("AQ: %d %.3f\n", total, fc);
+ /* the function is chosen such that it stays close to 0 in almost all
+ * range of 0..1, and rapidly goes up to 1 near 1.0 */
+ i_qp_adj = (int)(i_qp * h->param.analyse.f_aq_strength / pow(2 - fc, h->param.analyse.f_aq_sensitivity));
+
+ /* don't adjust by more than this amount */
+ i_qp_adj = X264_MIN(i_qp_adj, i_qp/2);
+
+ h->mb.i_qp = a->i_qp = i_qp - i_qp_adj;
+ h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
+}
+
+/*****************************************************************************
* x264_macroblock_analyse:
*****************************************************************************/
void x264_macroblock_analyse( x264_t *h )
@@ -2012,6 +2078,10 @@
/* init analysis */
x264_mb_analyse_init( h, &analysis, x264_ratecontrol_qp( h ) );

+ /* try to do adaptive quantization */
+ if( h->param.analyse.b_aq )
+ x264_adaptive_quant( h, &analysis);
+
/*--------------------------- Do the analysis ---------------------------*/
if( h->sh.i_type == SLICE_TYPE_I )
{
Index: x264.c
===================================================================
--- x264.c (revision 598)
+++ x264.c (working copy)
@@ -236,6 +236,12 @@
" - 2: enabled on all mode decisions\n", defaults->analyse.i_trellis );
H0( " --no-fast-pskip Disables early SKIP detection on P-frames\n" );
H0( " --no-dct-decimate Disables coefficient thresholding on P-frames\n" );
+ H0( " --aq-strength <float> Amount to adjust QP per MB [%.1f]\n"
+ " 0.0: no AQ\n"
+ " 1.1: strong AQ\n", defaults->analyse.f_aq_strength );
+ H0( " --aq-sensitivity <float> \"Flatness\" threshold to trigger AQ [%.1f]\n"
+ " 5: applies to almost all blocks\n"
+ " 22: only flat blocks\n", defaults->analyse.f_aq_sensitivity );
H0( " --nr <integer> Noise reduction [%d]\n", defaults->analyse.i_noise_reduction );
H1( "\n" );
H1( " --deadzone-inter <int> Set the size of the inter luma quantization deadzone [%d]\n", defaults->analyse.i_luma_deadzone[0] );
@@ -395,6 +401,8 @@
{ "trellis", required_argument, NULL, 't' },
{ "no-fast-pskip", no_argument, NULL, 0 },
{ "no-dct-decimate", no_argument, NULL, 0 },
+ { "aq-strength", required_argument, NULL, 0 },
+ { "aq-sensitivity", required_argument, NULL, 0 },
{ "deadzone-inter", required_argument, NULL, '0' },
{ "deadzone-intra", required_argument, NULL, '0' },
{ "level", required_argument, NULL, 0 },
Index: common/pixel.c
===================================================================
--- common/pixel.c (revision 598)
+++ common/pixel.c (working copy)
@@ -215,7 +215,17 @@
PIXEL_SATD_C( x264_pixel_satd_4x8, 4, 8 )
PIXEL_SATD_C( x264_pixel_satd_4x4, 4, 4 )

+static int x264_pixel_count_8x8( uint8_t *pix1, int i_pix1, uint32_t threshold )
+{
+ int i, j, sum = 0;
+
+ for ( i = 0; i < 8; ++i, pix1 += i_pix1 )
+ for ( j = 0; j < 8; ++j )
+ sum += pix1[j] > (uint8_t)threshold;

+ return sum;
+}
+
/****************************************************************************
* pixel_sa8d_WxH: sum of 8x8 Hadamard transformed differences
****************************************************************************/
@@ -470,6 +480,8 @@
pixf->ads[PIXEL_16x8] = pixel_ads2;
pixf->ads[PIXEL_8x8] = pixel_ads1;

+ pixf->count_8x8 = x264_pixel_count_8x8;
+
#ifdef HAVE_MMXEXT
if( cpu&X264_CPU_MMX )
{
Index: common/pixel.h
===================================================================
--- common/pixel.h (revision 598)
+++ common/pixel.h (working copy)
@@ -26,6 +26,7 @@

typedef int (*x264_pixel_cmp_t) ( uint8_t *, int, uint8_t *, int );
typedef int (*x264_pixel_cmp_pde_t) ( uint8_t *, int, uint8_t *, int, int );
+typedef int (*x264_pixel_count_t) ( uint8_t *, int, uint32_t );
typedef void (*x264_pixel_cmp_x3_t) ( uint8_t *, uint8_t *, uint8_t *, uint8_t *, int, int[3] );
typedef void (*x264_pixel_cmp_x4_t) ( uint8_t *, uint8_t *, uint8_t *, uint8_t *, uint8_t *, int, int[4] );

@@ -84,6 +85,7 @@
/* multiple parallel calls to sad. */
x264_pixel_cmp_x3_t sad_x3[7];
x264_pixel_cmp_x4_t sad_x4[7];
+ x264_pixel_count_t count_8x8;

/* abs-diff-sum for successive elimination.
* may round width up to a multiple of 8. */
Index: common/common.c
===================================================================
--- common/common.c (revision 598)
+++ common/common.c (working copy)
@@ -123,6 +123,9 @@
param->analyse.i_chroma_qp_offset = 0;
param->analyse.b_fast_pskip = 1;
param->analyse.b_dct_decimate = 1;
+ param->analyse.b_aq = 0;
+ param->analyse.f_aq_strength = 0.0;
+ param->analyse.f_aq_sensitivity = 15;
param->analyse.i_luma_deadzone[0] = 21;
param->analyse.i_luma_deadzone[1] = 11;
param->analyse.b_psnr = 1;
@@ -449,6 +452,13 @@
p->analyse.b_fast_pskip = atobool(value);
OPT("dct-decimate")
p->analyse.b_dct_decimate = atobool(value);
+ OPT("aq-strength")
+ {
+ p->analyse.f_aq_strength = atof(value);
+ p->analyse.b_aq = (p->analyse.f_aq_strength > 0.0);
+ }
+ OPT("aq-sensitivity")
+ p->analyse.f_aq_sensitivity = atof(value);
OPT("deadzone-inter")
p->analyse.i_luma_deadzone[0] = atoi(value);
OPT("deadzone-intra")
@@ -927,6 +937,9 @@
s += sprintf( s, " zones" );
}

+ if( p->analyse.b_aq )
+ s += sprintf( s, " aq=1:%.1f:%.1f", p->analyse.f_aq_strength, p->analyse.f_aq_sensitivity );
+
return buf;
}

Index: x264.h
===================================================================
--- x264.h (revision 598)
+++ x264.h (working copy)
@@ -218,6 +218,9 @@
int i_trellis; /* trellis RD quantization */
int b_fast_pskip; /* early SKIP detection on P-frames */
int b_dct_decimate; /* transform coefficient thresholding on P-frames */
+ int b_aq; /* psy adaptive QP */
+ float f_aq_strength;
+ float f_aq_sensitivity;
int i_noise_reduction; /* adaptive pseudo-deadzone */

/* the deadzone size that will be used in luma quantization */

[Dark Shikari]

do you have ANY proof sky has more bits?
the AQ screen looks bad for me...
the source is really simple, there's a treshold... if that threshold isnt hit then there's no change (it's also commented!).
If he can upload a clip of the AQ and non-AQ encode, I can analyze it with Elecard Streameye Tools and give a nice unbiased result :cool:

[Sharktooth]

maybe there's enough blue so maybe there are more bits there.
however if the sky was of another color (in particular RED) then AQ wont assign any bit. Actual AQ is not a solution, it's a workaround that may or may not work. that's the real reason why it hadnt been commited...
if you dont trust me, ask akupenguin... or haali.

[Terranigma]

Well, the image color's light blue.

Adaptive Quantisation allows different parts of a frame to use different quantisers, as opposed to the whole frame using the same quantizer. The common use for this in XviD was that it would compress light and dark areas more, thus freeing bits for parts that were more noticable. Adaptive Quantisation works differently in x264 because it was added by Haali to address a problem that large blue areas could sometimes look blocky. Adaptive Quantisation in x264 assigns more bits to dark and blue areas to help stop them blocking up. This particular setting instructs the amount to adjust QP per Macroblock. Setting 0.0 is no Adaptive Quantisation, and 1.1 is considered strong. It is recommended that you only use this setting if your source is particularly blocky in blue areas only wheras the rest of the image looks fine.

[Sharktooth]

exactly.

[Atak_Snajpera]

If he can upload a clip of the AQ and non-AQ encode, I can analyze it with Elecard Streameye Tools and give a nice unbiased result

http://forum.doom9.org/showthread.php?p=1043895#post1043895

[foxyshadis]

There's nothing related to dark or blue in that patch. Dark and blue areas are just the most common "flat" textures, but it'll apply to light grey (fog) as well as any color of shallow gradient. (In fact it could also apply on any texture that uses very few bits on just a few of high frequencies, rather than low freq, but that's so rare it's not important. I used to see it happen more often with MPEG-1, and still do with JPEG, but it was probably a bad encoder artifact.)

[Dark Shikari]

http://forum.doom9.org/showthread.php?p=1043895#post1043895
Done, here's some images of a B-frame with Elecard StreamEye (I love this software!)

Without AQ (Partition sizes):
http://i9.tinypic.com/5yv1d9x.png

With AQ (Partition sizes):
http://i14.tinypic.com/6c450mh.png

Now this is interesting: the non-AQ encode has smaller partitions! But we're not done yet.

Without AQ (Bits per macroblock):
http://i2.tinypic.com/4thowop.png

With AQ (Bits per macroblock):
http://i10.tinypic.com/6apron5.png

Well that looks pretty obvious right there. But if that isn't good enough:

Without AQ (Quantizer):
http://i2.tinypic.com/63kf0vo.png

With AQ (Quantizer):
http://i5.tinypic.com/66nmckh.png

Yes, AQ is being activated :p

[Sharktooth]

There's nothing related to dark or blue in that patch. Dark and blue areas are just the most common "flat" textures, but it'll apply to light grey (fog) as well as any color of shallow gradient. (In fact it could also apply on any texture that uses very few bits on just a few of high frequencies, rather than low freq, but that's so rare it's not important. I used to see it happen more often with MPEG-1, and still do with JPEG, but it was probably a bad encoder artifact.)
it counts blue pixels and red pixels for a reason... in bright non blue places and when there's too much red, there wont be any AQ.
again... look at the code FFS!
+void x264_adaptive_quant( x264_t *h, x264_mb_analysis_t *a )
+{
+ DECLARE_ALIGNED( static uint8_t, zero[FDEC_STRIDE*8], 16 );
+
+ int16_t dct[8][8];
+ int total = 0;
+ int i_qp = h->mb.i_qp, i_qp_adj;
+ float fc;
+
+ if( i_qp <= 10 ) /* AQ is probably not needed at such low QP */
+ return;
+
+ if( h->pixf.sad[PIXEL_16x16](h->mb.pic.p_fenc[0], FENC_STRIDE, zero, 16) > 64*16*16 )
+ { /* light places */
+ if( h->pixf.count_8x8(h->mb.pic.p_fenc[1], FENC_STRIDE, 0x81818181) < 40 )
+ /* not enough "blue" pixels */
+ return;
+
+ if( h->pixf.count_8x8(h->mb.pic.p_fenc[2], FENC_STRIDE, 0x87878787) > 24 )
+ /* too many "red" pixels */
+ return;
+ }
+
+ h->dctf.sub8x8_dct8( dct, h->mb.pic.p_fenc[0], zero );
+ x264_add_dctq( dct, &total );
+ h->dctf.sub8x8_dct8( dct, h->mb.pic.p_fenc[0] + 8, zero );
+ x264_add_dctq( dct, &total );
+ h->dctf.sub8x8_dct8( dct, h->mb.pic.p_fenc[0] + 8*FENC_STRIDE, zero );
+ x264_add_dctq( dct, &total );
+ h->dctf.sub8x8_dct8( dct, h->mb.pic.p_fenc[0] + 8*FENC_STRIDE + 8, zero );
+ x264_add_dctq( dct, &total );
+
+ if( total == 0 ) /* sum is 0, nothing to do */
+ return;
+
+ x264_cpu_restore( h->param.cpu );
+
+ fc = (float)expf(-5e-13 * total * total);
+
+ //printf("AQ: %d %.3f\n", total, fc);
+ /* the function is chosen such that it stays close to 0 in almost all
+ * range of 0..1, and rapidly goes up to 1 near 1.0 */
+ i_qp_adj = (int)(i_qp * h->param.analyse.f_aq_strength / pow(2 - fc, h->param.analyse.f_aq_sensitivity));
+
+ /* don't adjust by more than this amount */
+ i_qp_adj = X264_MIN(i_qp_adj, i_qp/2);
+
+ h->mb.i_qp = a->i_qp = i_qp - i_qp_adj;
+ h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
+}
so, no, it wont apply on light gray fog!

@dark shikari: the sky is blue there... and if you look at the pics you posted AQ is ONLY activated on blue or dark areas...

[Terranigma]

I have to take Sharktooth's and Zero1's side here.
I love this part: BIG FAT WARNING:

This disclaimer is as much to cover my ass, as it is to ensure you are not being mis-informed.
Also, about aq-sensitivity

--aq-sensitivity
This switch is to be used in conjunction with --aq-strength, and will not work if --aq-strength is set to 0.0, or not specified (since it is disabled by default). This controls the "flatness" threshold at which to apply AQ (Adaptive Quantisation). Setting it to 5 will apply AQ to almost all blocks, and setting it to 22 will apply it only to flat blocks. A flat block would be described as an area of solid colour or little detail.

[Sharktooth]

My preferred way to fix blockiness is using a custom quantization matrix. a very low coefficient (lower than 8) for low frequencies will ensure a better quantization for blocky areas.
with h.264 you can also trim luma and chroma matrices...

[Atak_Snajpera]

Dark Shikari could you do the same but with frame (for example frame no. 304) where I see blocks if I don't use AQ

[Dark Shikari]

Dark Shikari could you do the same but with frame (for example frame no. 304) where I see blocks if I don't use AQ
Done, Display Frame 304.

x264, no AQ, Macroblock Type:
http://i3.tinypic.com/33bk9kz.png

x264, AQ, Macroblock Type:
http://i13.tinypic.com/67gwrxv.png

Notice the vastly lower number of skipped blocks (yellow) in the sky when AQ is used.

x264, no AQ, Motion Vectors:
http://i18.tinypic.com/4p4avqp.png

x264, AQ, Motion Vectors
http://i5.tinypic.com/6cpj6n5.png

The newly non-skipped blocks introduced in AQ not surprisingly carry a lot of motion vectors with them. Quite long ones, sometimes, in fact.

x264, no AQ, Quantizer + Partitions:
http://i16.tinypic.com/6b0o09e.png

x264, AQ, Quantizer + Partitions:
http://i2.tinypic.com/4uudshe.png

Without AQ, the entire sky has basically no partitions at all. AQ adds some partitions and lowers the quantizer a lot.

x264, no AQ, Bits per Macroblock:
http://i4.tinypic.com/4porrr6.png

x264, AQ, Bits per Macroblock:
http://i18.tinypic.com/6f7v281.png

Should be obvious.

[Sharktooth]

it's no surprise at this point since the sky is indeed bright blue...
however AQ is producing a lot of MVs in the sky and those require even more bits.
now let's see if we can get a better picture without AQ just using a custom matrix...
please encode the same clip using EQM AVC-HR and lets see the results...
however why no one asked aku the reason AQ isnt in the x264 SVN?
im curious to hear his answer...
also, if you cant read the code, ask him what AQ exactly does...

[Atak_Snajpera]

Source is here
http://www.dvinfo.net/conf/showthread.php?t=77116
3. Tripod James & me Triple auto focus Size: 51.9Mb

right click and select Save Link As

I only need to know why XviD has smaller blocks than newer technology. Is this a part of H.264 standard?

[deets]

i did indeed try your custom matrix last night on a clip and its very good :)

one question though, how are encodes with custom matrices handled on non PC devices, will they play fine on PS3, xbox and PSP for example?

[Terranigma]

I know sharktooth doesn't recommend the prestige matrix(:p), but you should give it a try as well to see how it fares against blocking.

[Atak_Snajpera]

eqm_avc_hr.cfg
http://img262.imageshack.us/img262/1064/eqmavchrra7.th.png (http://img262.imageshack.us/my.php?image=eqmavchrra7.png)

Gamma 0.4
http://img510.imageshack.us/img510/1452/eqmavchrgamma04dc1.th.png (http://img510.imageshack.us/my.php?image=eqmavchrgamma04dc1.png)

[Terranigma]

Atak, can you try with the Prestige CQM (http://forum.doom9.org/showpost.php?p=989214&postcount=2) matrix?

[Sharktooth]

my opinion is it looks just better than the non-AQ pic...
better coefficients distribution may completely "fix" the blockiness.
ill work on a new anti-blocking matrix ASAP.

@deets: if the device fully supports the AVC high profile, then there will be no problems

[Atak_Snajpera]

Prestige
http://img126.imageshack.us/img126/6744/prestigecg6.th.png (http://img126.imageshack.us/my.php?image=prestigecg6.png)

gamma 0.4
http://img252.imageshack.us/img252/2865/prestigegamma04ua3.th.png (http://img252.imageshack.us/my.php?image=prestigegamma04ua3.png)

my opinion is it looks just better than the non-AQ pic...
Image quality is similar to --aq-strength 0.5 --aq-sensitivity 5 but custom matrixes work only with high profile. AQ works with all profiles.

[Terranigma]

Prestige looks good and this is without aq ?
I wonder how it'd look with adaptive quantization. Maybe it'd block more since it may conflict with the matrix's decision.

[Sharktooth]

noise and stuff like that is high frequency, and since the matrix is "supposed" to be tuned to preserve high frequencies, it is quite useless in this case where low frequencies are poorly quantized...
infact the prestige encode is as blocky as the original... however i still cant see blocks at the correct brightness... (FYI my monitor is this: http://www.asus.com/products.aspx?l1=10&l2=89&l3=356&l4=0&model=1382&modelmenu=1)
about AQ, matrices and profiles, high profile (FRExt) was explicitly made to reach a better visual quality and if a device isnt compatible with high profile then its not aimed to produce a high quality playback.
iPods & co. are a clear example of low quality devices. the high profile doesnt make almost any difference so they support lower specs...

[Terranigma]

better coefficients distribution may completely "fix" the blockiness.
ill work on a new anti-blocking matrix ASAP.
I can't wait.:D How long does it usually take you to create a custom matrix?

[Sharktooth]

creating? 2 minutes... tuning? several days...

EDIT: my current idea is to crate a "ripple-shaped" coefficients distribution for luma, having the lower ones on the corners to preserve both low and high frequencies (flat areas and noise/grain), sacrificing some apparently random intermediate ones a bit (in pure HVS style).
thinking about it... the idea may also work for MPEG-4 ASP too...

[Atak_Snajpera]

Prestige looks good and this is without aq ?
Sharktooth's matrix looks better

[Sharktooth]

time to reboot... if everything will work (usually an "emerge -uavDN world" is a real pain in the a$$... but the true pain comes when emerging system...) i will try render my idea into a matrix for the weekend...

[Sharktooth]

luckily the linux box restarted without major problems so...
this is more or less the idea for ASP:
8 8 9 9 9 9 8 8
8 9 10 10 10 10 9 8
9 10 11 12 12 11 10 9
9 10 12 11 11 12 10 9
9 10 12 11 11 12 10 9
9 10 11 12 12 11 10 9
8 9 10 10 10 10 9 8
8 8 9 9 9 9 8 8

16 16 17 17 17 17 16 16
16 17 18 18 18 18 17 16
17 18 19 20 20 19 18 17
17 18 20 19 19 20 18 17
17 18 20 19 19 20 18 17
17 18 19 20 20 19 18 17
16 17 18 18 18 18 17 16
16 16 17 17 17 17 16 16
http://img253.imageshack.us/img253/803/ripplematrixbg7.th.png (http://img253.imageshack.us/my.php?image=ripplematrixbg7.png)

***WARNING*** - this stuff is completely untested!

[Sharktooth]

can you please re-encode the clip with xvid and the previously posted matrix and the one that follows then post the usual screenshots?
8 8 9 9 9 9 8 8
8 9 10 10 10 10 9 8
9 10 11 12 12 11 10 9
9 10 12 11 11 12 10 9
9 10 12 11 11 12 10 9
9 10 11 12 12 11 10 9
8 9 10 10 10 10 9 8
8 8 9 9 9 9 8 8

8 8 9 9 9 9 8 8
8 9 10 10 10 10 9 8
9 10 11 12 12 11 10 9
9 10 12 11 11 12 10 9
9 10 12 11 11 12 10 9
9 10 11 12 12 11 10 9
8 9 10 10 10 10 9 8
8 8 9 9 9 9 8 8

[fields_g]

If you want some fun trying to kill these blocks off, I'd recommend the new Disney production logo sequence that was unveiled with "Pirates: Dead Man's Chest" (still shot (http://upload.wikimedia.org/wikipedia/en/d/dd/Waltdisneypictureslogo.png)). It is on "Bridge to Terabinthia" also. (Any other sources guys?)

Animated CGI footage with dark, blue, motion, and fine details everywhere! No need to be playing with gamma and tilting your lcd screens and adjusting brightness/contrast. The problems are so evident. It will give AQ/CQM/dead zone settings a run for the money!

[Dark Shikari]

If you want some fun trying to kill these blocks off, I'd recommend the new Disney production logo sequence that was unveiled with "Pirates: Dead Man's Chest" (still shot (http://upload.wikimedia.org/wikipedia/en/d/dd/Waltdisneypictureslogo.png)). I think it is on "Bridge to Terabinthia" also. I'll check tonight. (Any other sources guys?)

Animated CGI footage with dark, blue, motion, and fine details everywhere! It will give AQ/CQM/dead zone settings a run for the money!
Perhaps this is another chance for the old idea of mine... instead of AQ, here's what you do:

Encoding a macroblock:
1. Run a blocking metric on the source that measures how "blocky" that macroblock already is.
2. Do the motion estimation, etc, etc.
3. Encode the block at the normal quantizer.
4. Run the blocking metric again. If the blocking metric is much higher than it was before, re-encode the block at higher and higher quantizer until it gets within a certain tolerance of the original value.

Suddenly, no more blocks... without screwing up intra/inter prediction with a broken matrix like Prestige, and without adding extra bits where they aren't needed like AQ sometimes does.

[Sharktooth]

that would mean a major slowdown. during 4) a sort of prediction of the needed QP variance to reencode the block would waste less time.

[Dark Shikari]

that would mean a major slowdown. during 4) a sort of prediction of the needed QP variance to reencode the block would waste less time.It wouldn't be a horrible slowdown, as most of the time is already spent (motion estimation, etc).

A simple prediction heuristic would help though.

[Sharktooth]

yep, expecially if you have a lookup table with metric ranges and quantizer variations based on empirical tests or a simple formula for calculating quant variation in function of metric test variation (also empirical).
an even better solution would be:
1) Run a blocking metric on the source that measures how "blocky" that macroblock already is.
2) if metric indicates blocking then apply a debanding/deblocking filter using a dithering method and proceed with normal encoding else
3) encode at normal quant and run the blocking metric again.
4) If the blocking metric is much higher than it was before, apply a debanding/deblocking filter using a dithering method and proceed re-encoding the block.

This method will even improve visual quality if source has already blocking artifacts

[Dark Shikari]

yep, expecially if you have a lookup table with metric ranges and quantizer variations based on empirical tests or a simple formula for calculating quant variation in function of metric test variation (also empirical).
an even better solution would be:
1) Run a blocking metric on the source that measures how "blocky" that macroblock already is.
2) if metric indicates blocking then apply a debanding/deblocking filter using a dithering method and proceed with normal encoding else
3) encode at normal quant and run the blocking metric again.
4) If the blocking metric is much higher than it was before, apply a debanding/deblocking filter using a dithering method and proceed re-encoding the block.

This method will even improve visual quality if source has already blocking artifacts
Problems with that:

1. What if its meant to be a block to begin with? There are square objects that might get "deblocked" inadvertently.
2. Debanding adds dithering noise, which cannot be kept effectively without a much lower quantizer. It will quite literally be dumped by x264 during encoding.

[Sharktooth]

remove 2) then...
and no, ditering doesnt add noise. dithering doesnt create moving pixels but static pixels, so there are no additional MVs, etc... but since the block will contain more data a lower quantization is needed. just some additional analysis...
there is a possible problem though for referenced blocks...
this procedure may create flickering so the debanding should be temporal...

[woah!]

If you want some fun trying to kill these blocks off, I'd recommend the new Disney production logo sequence that was unveiled with "Pirates: Dead Man's Chest" (still shot (http://upload.wikimedia.org/wikipedia/en/d/dd/Waltdisneypictureslogo.png)). I think it is on "Bridge to Terabinthia" also. I'll check tonight. (Any other sources guys?)

Animated CGI footage with dark, blue, motion, and fine details everywhere! No need to be playing with gamma and tilting your lcd screens and adjusting brightness/contrast. The problems are so evident. It will give AQ/CQM/dead zone settings a run for the money!

i gave your idea a go and use the intro 800frames of POTC2.

heres 2 screenies of it at 1280x720 at 1000 bitrate. low i know but this way it shows the blocks well. i will do a 8000 bitrate next to show the same frames:

Prestige.cfg 1000bitrate
http://www.cif-forums.com/png/115_Pres.png

EQM AVC-HR.CFG 1000bitrate
http://www.cif-forums.com/png/115_AVC_HR.png

Prestige.cfg 8000bitrate
http://www.cif-forums.com/png/720P_INTRO_Prestige_8000BR.png

EQM AVC-HR.CFG 8000bitrate
http://www.cif-forums.com/png/720P_INTRO_EQM AVC-HR_8000BR.png

[Sharktooth]

EQM AVC-HR does a good job at both 1000 and 8000kbps.
however im working to a new matrix that will be even better ;)

[fields_g]

Prestige @ 1k.... I could have made that with legos. :) I know its a starved of its intended bitrate, but if a CQM can make something look so bad, Sharktooth's though of using a CQM to make it better might not be a bad one either!

What command line did you use for these encodes? Do you have the output stats?

After Sharktooth finishes his new CQM, it might be nice to run these and other CQM's in both CRF and VBR modes of various bitrates/levels. Anyone want to partner up with me a do this?

Here's another idea..... wanna move this conversation over to the MPEG-4 AVC forum instead of having it under a thread in the GUI forum under a title for a suggestion that shouldn't happen unless AQ is put into the SVN? (no wonder why people can't find this in search!)

[Terranigma]

woah!, test and compare*.mp4 guy's M4G-V3 CQM ;)

M4G-V3:
INTRA4X4_LUMA =
4,9,18,45,
9,16,35,180,
18,35,225,255,
45,180,255,255

INTRA4X4_CHROMAU =
14,22,32,72,
22,24,41,144,
32,41,96,200,
72,144,200,255

INTRA4X4_CHROMAV =
14,22,32,72,
22,24,41,144,
32,41,96,200,
72,144,200,255

INTER4X4_LUMA =
4,18,18,18,
18,18,18,18,
18,18,18,18,
18,18,18,18

INTER4X4_CHROMAU =
16,22,22,22,
22,22,22,56,
22,22,72,96,
22,56,96,128

INTER4X4_CHROMAV =
16,22,22,22,
22,22,22,56,
22,22,72,96,
22,56,96,128

INTRA8X8_LUMA =
5,8,11,15,18,21,24,35,
8,9,12,17,18,22,25,70,
11,12,18,19,22,30,40,125,
15,17,19,24,35,50,100,155,
18,18,22,35,70,135,160,180,
21,22,30,50,135,175,200,205,
24,25,40,100,160,200,225,230,
35,70,125,155,180,205,230,255

INTER8X8_LUMA =
5,14,16,18,18,18,19,19,
14,15,17,17,17,18,18,18,
16,17,17,17,17,18,18,18,
18,17,17,17,17,18,18,19,
18,17,17,17,17,18,18,19,
18,18,18,18,18,18,18,19,
19,18,18,18,18,18,18,20,
19,19,19,19,19,19,20,20

It's meant for this purpose.

--
I just found this (http://forum.doom9.org/showpost.php?p=799957&postcount=76) and I have to say that that's awesome :eek:

So AVC-HR is meant to retain this kind of detail as well? A miracle matrix? :p

[woah!]

M4G-V3.CFG 1000bitrate
http://www.cif-forums.com/png/720P_INTRO_M4G-V3.CFG_1000BR.png

[Terranigma]

Thanks. So it seems that sharktooth's matrix is better at retaining more detail and being less blocky than m4g-v3's coefficients, but then again, there's a 7000 bitrate difference compared to the other examples. :p

[Sharktooth]

my matrix was thought to retain most of the visible details, dropping frequencies without causing too much "damage" and better quantize lower frequencies to limit the blockiness. also since the matrix forces the encoder to put more bits than flat for every quantizer that will rise the QP for each encoded picture. doing that, the deblocking filter will be triggered more frequently (helping again reducing blocking), while at high bitrates this behaviour will be less evident so details get preserved.
since the matrix it's not 100% tuned there is still room for improving it, but keep in mind that it is "unbalanced"... and that could be bad for some x264 internal stuff (i said x264 since akupenguin once said that unbalanced matrices may screw something, i still dont know if other encoders can bear with that).
however if you encode something and it looks good, then all went good :)

@fields_g: yes, it's time to split the thread...

@Terranigma: i dont do miracles...

[Terranigma]

Sharktooth, do you exactly recall cases where direct auto conflicted with b-rdo? From my personal experiences, I haven't noticed blocking caused by using them together.

[Sharktooth]

no, sorry. my memory is too short to remember. however im sure there was a discussion on the AVC forum

[Terranigma]

No Problem. I have another question. What in-loop values would you recommend to use with your matrix?
-1, -1?

[MarcioAB]

Can I say the conclusion here is that blockness on dark/flat/low motion areas, may be better resolved with a special kind of "Quantization Matrix" than with frame-Adaptive Quantization (AQ) ?

I would like to try on my trouble encode (*).
Can someone indicate where can I get such "special" Quantization matrix ?

Thank you.

(*) By the way, the blockness of my trouble encode, compared with the Bike example posted here, is so worst that I can say the Bike encode is perfect.

[Atak_Snajpera]

Do whatever you want but I still prefer AQ over custom matrix. It's a lot easier for me to adjust two settings instead of playing with those matrixes. Prestige or EQM AVC-HR or maybe M4G-V3... eee waste of time... :P

[Terranigma]

Well you can use adaptive quantization with a custom matrix as well. It would'nt hurt you know. :p
Anyways, i'd like to hear from sharktooth on his recommended in-loop setting with the AVC-HR matrix. :)

[Atak_Snajpera]

Well you can use adaptive quantization with a custom matrix as well.
I see no reason to use these two options together. EQM AVC-HR works well and gives similar quality as AQ with standard matrix but it's restricted only to high profile. I prefer more compatible solution.

[MarcioAB]

I'm using the "EQM AVC-HR" that is in Sharktooth signature, but I understand he is working in something "better" and it's not ready yet.

Anyway I see 3 scenarios:

1) EQM matrix without AQ
2) default matrix with AQ
3) EQM matrix with AQ

My visual classification after quick tests is: (3) > (2) > (1) on the blocky area.

[Sharktooth]

@Terranigma: it all depends on your preferences. I like how it looks with -2,-1 but with different monitors/TV and different eyes it may look completely off.
so try some encodings with short clips and find the values you like most.

[akupenguin]

my matrix was thought to retain most of the visible details, dropping frequencies without causing too much "damage" and better quantize lower frequencies to limit the blockiness. also since the matrix forces the encoder to put more bits than flat for every quantizer that will rise the QP for each encoded picture. doing that, the deblocking filter will be triggered more frequently (helping again reducing blocking), while at high bitrates this behaviour will be less evident so details get preserved.
since the matrix it's not 100% tuned there is still room for improving it, but keep in mind that it is "unbalanced"... and that could be bad for some x264 internal stuff (i said x264 since akupenguin once said that unbalanced matrices may screw something, i still dont know if other encoders can bear with that).
Unbalanced means uses different average number of bits per QP than the flat matrix (or different quality per QP; I'm not sure that bits is exaclty the right measure). If you want to adjust deblocking strength, do that with --deblock not --cqm.
But even more important than matching bitrate of the encode as a whole, is matching the bitrate of each individual matrix to eachother. Prestige is very unbalanced in that the intra matrices take more bits than the inter matrices at the same QP, thus RDO rarely chooses intra even in blocks that don't benefit from inter prediction.

[woah!]

I'm using the "EQM AVC-HR" that is in Sharktooth signature, but I understand he is working in something "better" and it's not ready yet.

Anyway I see 3 scenarios:

1) EQM matrix without AQ
2) default matrix with AQ
3) EQM matrix with AQ

My visual classification after quick tests is: (3) > (2) > (1) on the blocky area.

i would concur with this by using this screenshot :

EQM AVC-HR.CFG 1000bitrate with --aq-sensitivity 10 --aq-strength 0.4

http://www.cif-forums.com/png/720P_INTRO_EQM AVC-HR_1000BR_AQ_0.4_10.bmp

[Sharktooth]

Unbalanced means uses different average number of bits per QP than the flat matrix (or different quality per QP; I'm not sure that bits is exaclty the right measure). If you want to adjust deblocking strength, do that with --deblock not --cqm.
But even more important than matching bitrate of the encode as a whole, is matching the bitrate of each individual matrix to eachother. Prestige is very unbalanced in that the intra matrices take more bits than the inter matrices at the same QP, thus RDO rarely chooses intra even in blocks that don't benefit from inter prediction.
correct me if im wrong, but rising the deblocking settings will compromise picture sharpness too.
the intent here is to have a good sharpness preserving fine details and reduce blocking in flat areas.
also there are some situations where deblocking doesnt help (blue sky issue...) while a CQM or AQ can.

[akupenguin]

correct me if im wrong, but rising the deblocking settings will compromise picture sharpness too.
Right, exactly the same as modifying the CQM to raise QP per bit. Except that modifying the QP per bit also confuses x264's other decisions, whereas deblocking settings only affect deblocking.

[woah!]

heres the 4 encodes of the intro for you to compare overall quality as screenshots dont tell the whole story most of the time:

720P_INTRO_EQM AVC-HR_1000BR_AQ_0.4_10.mkv
720P_INTRO_EQM AVC-HR_1000BR.CFG.mkv
720P_INTRO_M4G-V3.CFG_1000BR.CFG.mkv
720P_INTRO_Prestige_1000BR.cfg.mkv

encodes.rar (http://www.cif-forums.com/png/encodes.rar)

[Sharktooth]

Right, exactly the same as modifying the CQM to raise QP per bit. Except that modifying the QP per bit also confuses x264's other decisions, whereas deblocking settings only affect deblocking.
well the idea is exactly to force the encoder to make it assign more bits in flat areas since there is an unwanted blocking. the counter-effect is those bits should be reclaimed from other "places" but in most cases the blocking is the worst hurting thing in the picture.
i think this method is more coherent than AQ (that's assigning more bits only in blue or dark areas without distinction of flat or detailed areas) but perhaps there's a problem in x264 since this "blocking" problem seems to require workarounds for many sources.

[akupenguin]

Of course a CQM changes the distribution of bits. But what it shouldn't do is
since the matrix forces the encoder to put more bits than flat for every quantizer that will rise the QP for each encoded picture. doing that, the deblocking filter will be triggered more frequently

[Dark Shikari]

heres the 4 encodes of the intro for you to compare overall quality as screenshots dont tell the whole story most of the time:

720P_INTRO_EQM AVC-HR_1000BR_AQ_0.4_10.mkv
720P_INTRO_EQM AVC-HR_1000BR.CFG.mkv
720P_INTRO_M4G-V3.CFG_1000BR.CFG.mkv
720P_INTRO_Prestige_1000BR.cfg.mkv

encodes.rar (http://www.cif-forums.com/png/encodes.rar)
I tested these blindly with the MSU Human Visual Quality tester. Results:

Task method, Continuous quality evaluation (MSUCQE)
Method description, Two videos are shown simultaneously. During playback expert should click on video he or she dislikes at the moment.
Mark description, Mark belongs from 0 to 10, the higher the better. In pair of sequences if expert clicked on first one c1 times and on second c2 times, first sequence gets mark (10.0 * c2/(c1+c2)) and second gets (10.0 * c1(c1+c2)) or both get 5.0 if no clicks were made.

AVERAGE MARKS
720P_INTRO_EQM AVC-HR_1000BR.CFG
7.40

720P_INTRO_EQM AVC-HR_1000BR_AQ_0.4_10
9.02

720P_INTRO_M4G-V3.CFG_1000BR.CFG
5.57

720P_INTRO_Prestige_1000BR.cfg
0.02

Simple conclusion: these matrices (M4G and Prestige) tend to look worse at low bitrates, and AQ is good.

[Sharktooth]

EQM AVC-HR was not meant to be used at those bitrates...
it would be useful to have the flat matrix in that comparison too

[foxyshadis]

Sharktooth, would you care to rescale the matrix from the original coefficients to make it give approximately the same bitrate per quant? Call it the balanced version or something, I don't know.

[Sharktooth]

yes, however it depends on the source too... but i have the suspect the actual AVC-HR will look better...

[foxyshadis]

whoa, would you be willing try another encode, this time with m4g-smooth? Its scaling is a little more stronger than eqm-hr and I'd just like to see if it makes any visual difference.