Variance AQ Megathread (AQ v0.48 update--defaults changed)

[Dark Shikari]

So, after collecting enough magic pixie dust, I've come up with an AQ algorithm that just might work. Its purpose is to avoid blocking in flat areas like regular AQ, but more importantly, avoid blurring in relatively flat textured areas, such as grass at a football game or film grain. It seems to be relatively ineffective (or worse than no AQ at all) on low-bitrate anime, but I have gotten reports that its quite useful at higher bitrates, so try it yourself and see.

Patch (http://files.x264.nl/AQ/AQ_0.48.diff)
Build (http://files.x264.nl/AQ/x264.736.dark.aq.0.48.exe)
PthreadGC2.dll if you need it (http://mirror05.x264.nl/Dark/force.php?file=./pthreadGC2.dll).
A 5 minute or so 1080p sample encoded with the new AQ (0.45) at 5 megabits per second (http://mirror05.x264.nl/Dark/force.php?file=./PlanetEarthSample.mkv)
a 44 second sample of 1080p encoded at 10 megabits with AQ 0.47 (strength 0.9, sensitivity 14, qcomp=1) (http://x264.nl/x264.736.aq.0.47.mkv)

How to use AQ:

1. AQ is on by default at strength 0.5. Change --aq-strength to make it stronger or weaker.
2. In 2pass, use AQ on both passes with the same settings.
3. Watch in wonder as all the blurred details come back to life and the SSIM of your video rises.

Version history:

0.48: AQ strength 0.5, sensitivity 13 made the defaults. Updated to r736. Qcomp is now scaled based on AQ strength automatically.
0.47: Rounding error fixed at low QPs. Code cleanup/optimization by Akupenguin.
0.46: Code cleanup, documentation updated, defaults changed based on testing, and RDRC removed from the main patch in preparation for putting AQ in SVN. RDRC builds and patches can still be found on Mirror05/Dark.
0.45: Fixed bug if variance=0. Additionally, when static sensitivity is used, no limit is put on the quantizers other than qpmin/qpmax; this allows one to use AQ as a form of quasi-ratecontrol to redistribute more bits to flatter frames to improve quality.
0.44: Bug in x264 CABAC encoding (mb_qp_delta) fixed. While this isn't a bug in AQ, it only showed up when using CABAC, interlaced mode, and AQ.
0.431: Crash bug fixed.
0.43: Code cleaned and re-organized. Massive speed increase of the AQ itself due to performance optimizations.
0.42: Lambda-based AQ removed due to incompatibilities with deadzone that will take some work to resolve.

0.4 (Huge overhaul):

1. Totally rewritten AQ. Same basic concept, but now uses a logarithmic scale instead of a hackneyed exponential one.
2. For B-frames, uses a tricky bit of lambda-changing instead of QP changing; this requires absolutely no bits for QP-deltas!
3. For P-frames, uses a slight bit of trickery to reduce the bit cost of QP deltas.
4. Totally rewritten, far faster automatic sensitivity. Respects bitrate in CRF mode better also.
5. Now based on r720.

0.3 (Major overhaul of automatic thresholding and options)
0.21 (Fixed overflow bug with variance function. Code now is allowed to raise quantizers. If this causes problems I may restrict it somewhat.)
0.2 (Re-introduced pre-pass for automagic thresholding)
0.11 (Heavily optimized code)
0.1 (Initial release, fixed scaling formula, removed pre-pass)
0.01 (Initial algorithm)

Results from a test of AQ 0.42:

(1-0.9769377) / (1- 0.9799512) = 15% SSIM boost
(37.751-36.792)/0.05 = 19% PSNR drop
7416.06 / 6960.12 - 1 = 6.55% bitrate drop

[Inventive Software]

Trust you to come up with this an hour before I go home for 3 weeks! :D

I'll have a play next week, if you're not too busy changing the innards of the algorithm. ;)

[ToS_Maverick]

OMG Dark Shikari you are a HERO :D

i could only do a quick test for now, but @CRF20 it was COMPLETELY transparent, at CRF22 it was VERY good...

a few questions:
- how does it work, it's very effective?
- where can i donate :D

more detailed test with screens coming soon ;)

[Dark Shikari]

OMG Dark Shikari you are a HERO :D

i could only do a quick test for now, but @CRF20 it was COMPLETELY transparent, at CRF22 it was VERY good...

a few questions:
- how does it work, it's very effective?
- where can i donate :D

more detailed test with screens coming soon ;)
It works by using the AC energy of each macroblock as a metric. Or, in other words, it takes the average of the block's pixels, subtracts those from the original pixel data, and then takes the sum of squares of the result. This means that blocks that aren't completely flat but have a lot of texture still get hit by AQ.

There are some slight optimizations to this (like the fact that this is mathematically equivalent to SSD - SAD^2, and so forth). Also note I made a major change to the build that's up there (I reuploaded)--it now has --aq-sensitivity affect the algorithm. This affects the thresholding in the following way: lower values mean that blocks have to be "flatter" to be affected by AQ, while higher values mean blocks don't have to be as flat.

[AGDenton]

Can you do an svn diff against some revision of x264 ? I'd like to test this, but I'm not under win32...

[Dark Shikari]

Can you do an svn diff against some revision of x264 ? I'd like to test this, but I'm not under win32...

Index: encoder/encoder.c
===================================================================
--- encoder/encoder.c (revision 712)
+++ encoder/encoder.c (working copy)
@@ -472,6 +472,8 @@
if( !h->param.b_cabac )
h->param.analyse.i_trellis = 0;
h->param.analyse.i_trellis = x264_clip3( h->param.analyse.i_trellis, 0, 2 );
+ if( h->param.analyse.b_aq && h->param.analyse.f_aq_strength <= 0 )
+ h->param.analyse.b_aq = 0;
h->param.analyse.i_noise_reduction = x264_clip3( h->param.analyse.i_noise_reduction, 0, 1<<16 );

{
Index: encoder/analyse.c
===================================================================
--- encoder/analyse.c (revision 712)
+++ encoder/analyse.c (working copy)
@@ -29,6 +29,7 @@
#endif

#include "common/common.h"
+#include "common/cpu.h"
#include "macroblock.h"
#include "me.h"
#include "ratecontrol.h"
@@ -2031,8 +2032,61 @@
}
}

+//Finds the total AC energy of the block in all planes.
+static int ac_energy_mb(x264_t *h)
+{
+ DECLARE_ALIGNED( static uint8_t, zero[FDEC_STRIDE*16], 16 );
+ int avg[3];
+ int x,y;
+ for(y = 0; y < 16; y++)
+ for(x = 0; x < 16; x++)
+ zero[FDEC_STRIDE*y+x]=0;
+ avg[0] = h->pixf.sad[PIXEL_16x16](zero,FDEC_STRIDE,h->mb.pic.p_fenc[0],FENC_STRIDE) >> 8;
+ avg[1] = h->pixf.sad[PIXEL_8x8](zero,FDEC_STRIDE,h->mb.pic.p_fenc[1],FENC_STRIDE) >> 6;
+ avg[2] = h->pixf.sad[PIXEL_8x8](zero,FDEC_STRIDE,h->mb.pic.p_fenc[2],FENC_STRIDE) >> 6;
+ int totalSSD = 0;
+ for(y = 0; y < 16; y++)
+ for(x = 0; x < 16; x++)
+ zero[FDEC_STRIDE*y+x]=avg[0];
+ totalSSD += h->pixf.ssd[PIXEL_16x16](zero,FDEC_STRIDE,h->mb.pic.p_fenc[0],FENC_STRIDE);
+ for(y = 0; y < 8; y++)
+ for(x = 0; x < 8; x++)
+ zero[FDEC_STRIDE*y+x]=avg[1];
+ totalSSD += h->pixf.ssd[PIXEL_8x8](zero,FDEC_STRIDE,h->mb.pic.p_fenc[1],FENC_STRIDE);
+ for(y = 0; y < 8; y++)
+ for(x = 0; x < 8; x++)
+ zero[FDEC_STRIDE*y+x]=avg[2];
+ totalSSD += h->pixf.ssd[PIXEL_8x8](zero,FDEC_STRIDE,h->mb.pic.p_fenc[2],FENC_STRIDE);
+ return totalSSD;
+}

/*****************************************************************************
+ * x264_adaptive_quant:
+ * check if mb is "flat", i.e. has most energy in low frequency components, and
+ * adjust qp down if it is
+ *****************************************************************************/
+void x264_adaptive_quant( x264_t *h, x264_mb_analysis_t *a )
+{
+ int qp = h->mb.i_qp;
+ int ac_energy = ac_energy_mb(h);
+ x264_cpu_restore(h->param.cpu);
+ float result = ac_energy;
+ const float expconst = 0.367879441;
+ float threshold = powf(h->param.analyse.f_aq_sensitivity,4)/2;
+ if(result < threshold)
+ {
+ if(result == 0) result = 1;
+ else
+ result = (expconst-expf(-powf(threshold/result,0.2))) * 2.71828183;
+ }
+ else result = 0;
+ int qp_adj = (qp * result * h->param.analyse.f_aq_strength) / 2;
+ qp_adj = x264_clip3(qp_adj, 0, qp/2);
+ h->mb.i_qp = a->i_qp = qp - qp_adj;
+ h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
+}
+
+/*****************************************************************************
* x264_macroblock_analyse:
*****************************************************************************/
void x264_macroblock_analyse( x264_t *h )
@@ -2040,9 +2094,14 @@
x264_mb_analysis_t analysis;
int i_cost = COST_MAX;
int i;
+
+ h->mb.i_qp = x264_ratecontrol_qp( h );

+ if( h->param.analyse.b_aq )
+ x264_adaptive_quant( h, &analysis );
+
/* init analysis */
- x264_mb_analyse_init( h, &analysis, x264_ratecontrol_qp( h ) );
+ x264_mb_analyse_init( h, &analysis, h->mb.i_qp );

/*--------------------------- Do the analysis ---------------------------*/
if( h->sh.i_type == SLICE_TYPE_I )
Index: x264.c
===================================================================
--- x264.c (revision 712)
+++ x264.c (working copy)
@@ -243,6 +243,12 @@
" - 2: enabled on all mode decisions\n", defaults->analyse.i_trellis );
H0( " --no-fast-pskip Disables early SKIP detection on P-frames\n" );
H0( " --no-dct-decimate Disables coefficient thresholding on P-frames\n" );
+ H0( " --aq-strength <float> Amount to adjust QP per MB [%.1f]\n"
+ " 0.0: no AQ\n"
+ " 1.1: strong AQ\n", defaults->analyse.f_aq_strength );
+ H0( " --aq-sensitivity <float> \"Flatness\" threshold to trigger AQ [%.1f]\n"
+ " 5: applies to almost no blocks\n"
+ " 35: applies to almost all blocks\n", defaults->analyse.f_aq_sensitivity );
H0( " --nr <integer> Noise reduction [%d]\n", defaults->analyse.i_noise_reduction );
H1( "\n" );
H1( " --deadzone-inter <int> Set the size of the inter luma quantization deadzone [%d]\n", defaults->analyse.i_luma_deadzone[0] );
@@ -406,6 +412,8 @@
{ "trellis", required_argument, NULL, 't' },
{ "no-fast-pskip", no_argument, NULL, 0 },
{ "no-dct-decimate", no_argument, NULL, 0 },
+ { "aq-strength", required_argument, NULL, 0 },
+ { "aq-sensitivity", required_argument, NULL, 0 },
{ "deadzone-inter", required_argument, NULL, '0' },
{ "deadzone-intra", required_argument, NULL, '0' },
{ "level", required_argument, NULL, 0 },
Index: common/pixel.c
===================================================================
--- common/pixel.c (revision 712)
+++ common/pixel.c (working copy)
@@ -213,6 +213,14 @@
PIXEL_SATD_C( x264_pixel_satd_4x8, 4, 8 )
PIXEL_SATD_C( x264_pixel_satd_4x4, 4, 4 )

+static int x264_pixel_count_8x8( uint8_t *pix, int i_pix, uint32_t threshold )
+{
+ int x, y, sum = 0;
+ for( y=0; y<8; y++, pix += i_pix )
+ for( x=0; x<8; x++ )
+ sum += pix[x] > (uint8_t)threshold;
+ return sum;
+}

/****************************************************************************
* pixel_sa8d_WxH: sum of 8x8 Hadamard transformed differences
@@ -473,6 +481,8 @@
pixf->ads[PIXEL_16x8] = pixel_ads2;
pixf->ads[PIXEL_8x8] = pixel_ads1;

+ pixf->count_8x8 = x264_pixel_count_8x8;
+
#ifdef HAVE_MMX
if( cpu&X264_CPU_MMX )
{
Index: common/pixel.h
===================================================================
--- common/pixel.h (revision 712)
+++ common/pixel.h (working copy)
@@ -84,6 +84,8 @@
void (*ads[7])( int enc_dc[4], uint16_t *sums, int delta,
uint16_t *res, int width );

+ int (*count_8x8)( uint8_t *pix, int i_pix, uint32_t threshold );
+
/* calculate satd of V, H, and DC modes.
* may be NULL, in which case just use pred+satd instead. */
void (*intra_satd_x3_16x16)( uint8_t *fenc, uint8_t *fdec, int res[3] );
Index: common/common.c
===================================================================
--- common/common.c (revision 712)
+++ common/common.c (working copy)
@@ -123,6 +123,9 @@
param->analyse.i_chroma_qp_offset = 0;
param->analyse.b_fast_pskip = 1;
param->analyse.b_dct_decimate = 1;
+ param->analyse.b_aq = 0;
+ param->analyse.f_aq_strength = 0.0;
+ param->analyse.f_aq_sensitivity = 15;
param->analyse.i_luma_deadzone[0] = 21;
param->analyse.i_luma_deadzone[1] = 11;
param->analyse.b_psnr = 1;
@@ -455,6 +458,13 @@
p->analyse.b_fast_pskip = atobool(value);
OPT("dct-decimate")
p->analyse.b_dct_decimate = atobool(value);
+ OPT("aq-strength")
+ {
+ p->analyse.f_aq_strength = atof(value);
+ p->analyse.b_aq = (p->analyse.f_aq_strength > 0.0);
+ }
+ OPT("aq-sensitivity")
+ p->analyse.f_aq_sensitivity = atof(value);
OPT("deadzone-inter")
p->analyse.i_luma_deadzone[0] = atoi(value);
OPT("deadzone-intra")
@@ -939,6 +949,9 @@
s += sprintf( s, " zones" );
}

+ if( p->analyse.b_aq )
+ s += sprintf( s, " aq=1:%.1f:%.1f", p->analyse.f_aq_strength, p->analyse.f_aq_sensitivity );
+
return buf;
}

Index: x264.h
===================================================================
--- x264.h (revision 712)
+++ x264.h (working copy)
@@ -230,6 +230,9 @@
int i_trellis; /* trellis RD quantization */
int b_fast_pskip; /* early SKIP detection on P-frames */
int b_dct_decimate; /* transform coefficient thresholding on P-frames */
+ int b_aq; /* psy adaptive QP */
+ float f_aq_strength;
+ float f_aq_sensitivity;
int i_noise_reduction; /* adaptive pseudo-deadzone */

/* the deadzone size that will be used in luma quantization */

[akupenguin]

In addition to being unnecessary as discussed before, your zero array is not thread safe. And if you did for whatever reason need a dc array, its stride should be 0 to reduce the amount of data to initialize.

[Dark Shikari]

In addition to being unnecessary as discussed before, your zero array is not thread safe. And if you did for whatever reason need a dc array, its stride should be 0 to reduce the amount of data to initialize.Yup, yup, I will fix the code. Wait, the zero array isn't threadsafe though? Isn't that what ordinary AQ uses?

[akupenguin]

That zero array contains zeros. Yours gets modified. In short: there shouldn't be any non-const static variables.

[Dark Shikari]

That zero array contains zeros. Yours gets modified. In short: there shouldn't be any non-const static variables.Bleh, fixed code.

Not bit-equivalent to the old one, but close enough, and faster.

Index: encoder/encoder.c
===================================================================
--- encoder/encoder.c (revision 712)
+++ encoder/encoder.c (working copy)
@@ -472,6 +472,8 @@
if( !h->param.b_cabac )
h->param.analyse.i_trellis = 0;
h->param.analyse.i_trellis = x264_clip3( h->param.analyse.i_trellis, 0, 2 );
+ if( h->param.analyse.b_aq && h->param.analyse.f_aq_strength <= 0 )
+ h->param.analyse.b_aq = 0;
h->param.analyse.i_noise_reduction = x264_clip3( h->param.analyse.i_noise_reduction, 0, 1<<16 );

{
Index: encoder/analyse.c
===================================================================
--- encoder/analyse.c (revision 712)
+++ encoder/analyse.c (working copy)
@@ -29,6 +29,7 @@
#endif

#include "common/common.h"
+#include "common/cpu.h"
#include "macroblock.h"
#include "me.h"
#include "ratecontrol.h"
@@ -2031,8 +2032,51 @@
}
}

+//Finds the total AC energy of the block in all planes.
+static int ac_energy_mb(x264_t *h)
+{
+ DECLARE_ALIGNED( static uint8_t, zero[16], 16 );
+ int sad,ssd;
+ int totalSSD = 0;
+ sad = h->pixf.sad[PIXEL_16x16](zero,0,h->mb.pic.p_fenc[0],FENC_STRIDE);
+ ssd = h->pixf.ssd[PIXEL_16x16](zero,0,h->mb.pic.p_fenc[0],FENC_STRIDE);
+ totalSSD += ssd - ((sad * sad) >> 8);
+ sad = h->pixf.sad[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[1],FENC_STRIDE);
+ ssd = h->pixf.ssd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[1],FENC_STRIDE);
+ totalSSD += ssd - ((sad * sad) >> 6);
+ sad = h->pixf.sad[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[2],FENC_STRIDE);
+ ssd = h->pixf.ssd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[2],FENC_STRIDE);
+ totalSSD += ssd - ((sad * sad) >> 6);
+ return totalSSD;
+}

/*****************************************************************************
+ * x264_adaptive_quant:
+ * check if mb is "flat", i.e. has most energy in low frequency components, and
+ * adjust qp down if it is
+ *****************************************************************************/
+void x264_adaptive_quant( x264_t *h, x264_mb_analysis_t *a )
+{
+ int qp = h->mb.i_qp;
+ int ac_energy = ac_energy_mb(h);
+ x264_cpu_restore(h->param.cpu);
+ float result = ac_energy;
+ const float expconst = 0.367879441;
+ float threshold = powf(h->param.analyse.f_aq_sensitivity,4)/2;
+ if(result < threshold)
+ {
+ if(result == 0) result = 1;
+ else
+ result = (expconst-expf(-powf(threshold/result,0.2))) * 2.71828183;
+ }
+ else result = 0;
+ int qp_adj = (qp * result * h->param.analyse.f_aq_strength) / 2;
+ qp_adj = x264_clip3(qp_adj, 0, qp/2);
+ h->mb.i_qp = a->i_qp = qp - qp_adj;
+ h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
+}
+
+/*****************************************************************************
* x264_macroblock_analyse:
*****************************************************************************/
void x264_macroblock_analyse( x264_t *h )
@@ -2040,9 +2084,14 @@
x264_mb_analysis_t analysis;
int i_cost = COST_MAX;
int i;
+
+ h->mb.i_qp = x264_ratecontrol_qp( h );

+ if( h->param.analyse.b_aq )
+ x264_adaptive_quant( h, &analysis );
+
/* init analysis */
- x264_mb_analyse_init( h, &analysis, x264_ratecontrol_qp( h ) );
+ x264_mb_analyse_init( h, &analysis, h->mb.i_qp );

/*--------------------------- Do the analysis ---------------------------*/
if( h->sh.i_type == SLICE_TYPE_I )
Index: x264.c
===================================================================
--- x264.c (revision 712)
+++ x264.c (working copy)
@@ -243,6 +243,12 @@
" - 2: enabled on all mode decisions\n", defaults->analyse.i_trellis );
H0( " --no-fast-pskip Disables early SKIP detection on P-frames\n" );
H0( " --no-dct-decimate Disables coefficient thresholding on P-frames\n" );
+ H0( " --aq-strength <float> Amount to adjust QP per MB [%.1f]\n"
+ " 0.0: no AQ\n"
+ " 1.1: strong AQ\n", defaults->analyse.f_aq_strength );
+ H0( " --aq-sensitivity <float> \"Flatness\" threshold to trigger AQ [%.1f]\n"
+ " 5: applies to almost no blocks\n"
+ " 35: applies to almost all blocks\n", defaults->analyse.f_aq_sensitivity );
H0( " --nr <integer> Noise reduction [%d]\n", defaults->analyse.i_noise_reduction );
H1( "\n" );
H1( " --deadzone-inter <int> Set the size of the inter luma quantization deadzone [%d]\n", defaults->analyse.i_luma_deadzone[0] );
@@ -406,6 +412,8 @@
{ "trellis", required_argument, NULL, 't' },
{ "no-fast-pskip", no_argument, NULL, 0 },
{ "no-dct-decimate", no_argument, NULL, 0 },
+ { "aq-strength", required_argument, NULL, 0 },
+ { "aq-sensitivity", required_argument, NULL, 0 },
{ "deadzone-inter", required_argument, NULL, '0' },
{ "deadzone-intra", required_argument, NULL, '0' },
{ "level", required_argument, NULL, 0 },
Index: common/pixel.c
===================================================================
--- common/pixel.c (revision 712)
+++ common/pixel.c (working copy)
@@ -213,6 +213,14 @@
PIXEL_SATD_C( x264_pixel_satd_4x8, 4, 8 )
PIXEL_SATD_C( x264_pixel_satd_4x4, 4, 4 )

+static int x264_pixel_count_8x8( uint8_t *pix, int i_pix, uint32_t threshold )
+{
+ int x, y, sum = 0;
+ for( y=0; y<8; y++, pix += i_pix )
+ for( x=0; x<8; x++ )
+ sum += pix[x] > (uint8_t)threshold;
+ return sum;
+}

/****************************************************************************
* pixel_sa8d_WxH: sum of 8x8 Hadamard transformed differences
@@ -473,6 +481,8 @@
pixf->ads[PIXEL_16x8] = pixel_ads2;
pixf->ads[PIXEL_8x8] = pixel_ads1;

+ pixf->count_8x8 = x264_pixel_count_8x8;
+
#ifdef HAVE_MMX
if( cpu&X264_CPU_MMX )
{
Index: common/pixel.h
===================================================================
--- common/pixel.h (revision 712)
+++ common/pixel.h (working copy)
@@ -84,6 +84,8 @@
void (*ads[7])( int enc_dc[4], uint16_t *sums, int delta,
uint16_t *res, int width );

+ int (*count_8x8)( uint8_t *pix, int i_pix, uint32_t threshold );
+
/* calculate satd of V, H, and DC modes.
* may be NULL, in which case just use pred+satd instead. */
void (*intra_satd_x3_16x16)( uint8_t *fenc, uint8_t *fdec, int res[3] );
Index: common/common.c
===================================================================
--- common/common.c (revision 712)
+++ common/common.c (working copy)
@@ -123,6 +123,9 @@
param->analyse.i_chroma_qp_offset = 0;
param->analyse.b_fast_pskip = 1;
param->analyse.b_dct_decimate = 1;
+ param->analyse.b_aq = 0;
+ param->analyse.f_aq_strength = 0.0;
+ param->analyse.f_aq_sensitivity = 15;
param->analyse.i_luma_deadzone[0] = 21;
param->analyse.i_luma_deadzone[1] = 11;
param->analyse.b_psnr = 1;
@@ -455,6 +458,13 @@
p->analyse.b_fast_pskip = atobool(value);
OPT("dct-decimate")
p->analyse.b_dct_decimate = atobool(value);
+ OPT("aq-strength")
+ {
+ p->analyse.f_aq_strength = atof(value);
+ p->analyse.b_aq = (p->analyse.f_aq_strength > 0.0);
+ }
+ OPT("aq-sensitivity")
+ p->analyse.f_aq_sensitivity = atof(value);
OPT("deadzone-inter")
p->analyse.i_luma_deadzone[0] = atoi(value);
OPT("deadzone-intra")
@@ -939,6 +949,9 @@
s += sprintf( s, " zones" );
}

+ if( p->analyse.b_aq )
+ s += sprintf( s, " aq=1:%.1f:%.1f", p->analyse.f_aq_strength, p->analyse.f_aq_sensitivity );
+
return buf;
}

Index: x264.h
===================================================================
--- x264.h (revision 712)
+++ x264.h (working copy)
@@ -230,6 +230,9 @@
int i_trellis; /* trellis RD quantization */
int b_fast_pskip; /* early SKIP detection on P-frames */
int b_dct_decimate; /* transform coefficient thresholding on P-frames */
+ int b_aq; /* psy adaptive QP */
+ float f_aq_strength;
+ float f_aq_sensitivity;
int i_noise_reduction; /* adaptive pseudo-deadzone */

/* the deadzone size that will be used in luma quantization */

EXE updated.

[LigH]

Hooray!

Thanks for this patch. I bet some friends in the german board will test it too.

[Sagekilla]

So now the general starting point for using AQ-strength would be 1.0 and now 0.5 as before? If so that would be quite nice, since I'd imagine it'd give me some more leeway with only using tiny amounts of AQ in those pesky movies where theres relatively few dark scenes.

[Dark Shikari]

I found a bug with my energy function where I get an integer overflow in some extremely bright blocks, resulting in AQ not being activated even if the block is flat. A fix will come in a bit.

I'm also working on a magical algorithm to automatically find a good threshold value for each frame. :)

[Sagekilla]

I found a bug with my energy function where I get an integer overflow in some extremely bright blocks, resulting in AQ not being activated even if the block is flat. A fix will come in a bit.

I'm also working on a magical algorithm to automatically find a good threshold value for each frame. :)

Could this be magically added to a multi-patched x264 with all your other wonderful patches too? :)

[Dark Shikari]

Could this be magically added to a multi-patched x264 with all your other wonderful patches too? :)Soon. In the meantime, a teaser of the latest algorithm:

(1-pass ABR, 1000 kbit, a comparison of two I-frames)

Original:

http://i6.tinypic.com/72riyc7.pnghttp://i2.tinypic.com/6lb7k2g.png

AQ:

http://i4.tinypic.com/8ftnotu.pnghttp://i17.tinypic.com/8borxbp.png

Note most of the ringing is from the original source, which was not a very well-encoded DVD (and so blurring obscures the ringing when AQ isn't used).

If you want a huge contrast between the two, look at the wheel in the background on the first image. Or the bricks in the background on the second image.

[Sagekilla]

Very nice, some good detail retention in areas that I'd imagine would otherwise be killed off..

[kumi]

I see a huge difference in CRF output size with --aq-sensitivity 0. Normal?

--crf 21.5
Size: 9.99 MB
Bitrate (Avg): 1.169

--crf 21.5 --aq-str 1.0
Size: 8.12 MB
Bitrate (Avg): 0.950

--crf 21.5 --aq-str 1.0 --aq-sens 0
Size: 36.8 MB
Bitrate (Avg): 4.309

[Sagekilla]

I see a huge difference in CRF output size with --aq-sensitivity 0. Normal?

--crf 21.5
Size: 9.99 MB
Bitrate (Avg): 1.169

--crf 21.5 --aq-str 1.0
Size: 8.12 MB
Bitrate (Avg): 0.950

--crf 21.5 --aq-str 1.0 --aq-sens 0
Size: 36.8 MB
Bitrate (Avg): 4.309

Perhaps it may be borked with 0, like one of those divide by zero errors.

[kumi]

Yes, but
"2. For the automagic thresholding algorithm, use --aq-sensitivity 0."

[Dark Shikari]

I see a huge difference in CRF output size with --aq-sensitivity 0. Normal?

--crf 21.5
Size: 9.99 MB
Bitrate (Avg): 1.169

--crf 21.5 --aq-str 1.0
Size: 8.12 MB
Bitrate (Avg): 0.950

--crf 21.5 --aq-str 1.0 --aq-sens 0
Size: 36.8 MB
Bitrate (Avg): 4.309Try with bitrate mode to make the results more comparable. Its likely screwing up ratecontrol--I will try to see what I can do to make it avoid blowing up the filesize.

That is natural though--AQ does drastically raise filesize with CRF. Its just in this case its raised it a bit more than usual.

[check]

what do you get with a sensitivity very near 1?

[Sagekilla]

I have to say, I find your AQ to be quite interesting.. I actually got a -huge- reduction in bitrate when I used it. 1738 kbps without vs 1475 kbps with, in one of my tests. That was using a simple --aq-strength 0.5 --aq-sensitivity 15.

@Check: At that point I think it'd be still running at the regular non-adaptive sensitivity so it would activate on very few blocks according to what the help says (low aq = less blocks activated on, high aq = more blocks activated on)

[Dark Shikari]

0 = adaptive, any other value = regular scheme.

Its possible adaptive could be a bit too strong by default, so experiment with lower strength values (and I could experiment with slightly better adaptive schemes).

[Sagekilla]

0 = adaptive, any other value = regular scheme.

Its possible adaptive could be a bit too strong by default, so experiment with lower strength values (and I could experiment with slightly better adaptive schemes).

It seems like it, because I tried the adaptive mode (Wouldn't that make it a... adaptive adaptive quantization?) myself and I ended up with a severely bloated file over have it at the default sensitivity of 15. Personally I like how it decreases the file sizes while not really decreasing the quality at all, so that's just about reason enough for me to just go with strength 0.9 and sensitivity 15.

[Dark Shikari]

I found some serious problems with automagic thresholding--it was consistently overestimating the necessary threshold.

As a result, I implemented a much more brute-force (and as a result slightly slower) algorithm that should be able to find a better threshold. Try it out--unlike before, it shouldn't screw up ratecontrol.

Strength has also been moved to a different part of the formula for easier control over the results of the algorithm.

The goal of this latest algorithm is to keep the average QP per frame the same. This, in most cases, keeps the bits per frame relatively similar, which means AQ should no longer drastically increase or decrease bitrate at a given CRF/QP.

[kumi]

Great! Can't wait to test :D

[ToS_Maverick]

what i found out about 0.3 with BlackPearl:
- --aq-strength 1.0 --aq-sensitivity 20 and CRF20 is transparent
- your new AQ produces bigger files, but the quality is better than ever!
- str 1.0 is very balanced
- below sens 20 some areas get left behind
- auto mode (sens 0) is producing heavily undersized files
- --aq-strength 1.0 --aq-sensitivity 20 and CRF20 = --aq-strength 1.0 and CRF15, about the same size and quality (only a small difference)
- SSIM is the same (CRF20 with AQ compared to CRF17 without, same size)
- PSNR is 1 dB lower (OMG ;))

why is your adaptive-mode acting so weird? what is it supposed to do?

[Sagekilla]

what i found out about 0.3 with BlackPearl:
- --aq-strength 1.0 --aq-sensitivity 20 and CRF20 is transparent
- your new AQ produces bigger files, but the quality is better than ever!
- str 1.0 is very balanced
- below sens 20 some areas get left behind
- auto mode (sens 0) is producing heavily undersized files
- --aq-strength 1.0 --aq-sensitivity 20 and CRF20 = --aq-strength 1.0 and CRF15, about the same size and quality (only a small difference)
- SSIM is the same (CRF20 with AQ compared to CRF17 without, same size)
- PSNR is 1 dB lower (OMG ;))

why is your adaptive-mode acting so weird? what is it supposed to do?

The "adaptive mode" for adaptive quantization is supposed to dynamically choose the best sensitivity for each frame, so it can change the qps accordingly, or that's what it seems to be doing from what I can infer.

[Dark Shikari]

The "adaptive mode" for adaptive quantization is supposed to dynamically choose the best sensitivity for each frame, so it can change the qps accordingly, or that's what it seems to be doing from what I can infer.And it defines "best" as the sensitivity that results in the average QP for that frame not changing--i.e. if it raises 20 QPs by 5, it also has to lower other QPS by a total of 100.

[Sagekilla]

And it defines "best" as the sensitivity that results in the average QP for that frame not changing--i.e. if it raises 20 QPs by 5, it also has to lower other QPS by a total of 100.

Does this affect the qps of each block after x264 chooses a qp for a given frame or does this intermix with the qp decision to give the frame?

[Dark Shikari]

Does this affect the qps of each block after x264 chooses a qp for a given frame or does this intermix with the qp decision to give the frame?After, because x264's frame-QP decision is already based on the relative complexity of the frame.

[Sagekilla]

After, because x264's frame-QP decision is already based on the relative complexity of the frame.

Gotcha, so in this case the new adaptive mode will be just be adding and removing bits here and there without actually reducing or increasing the bitrate significantly?

[Dark Shikari]

Gotcha, so in this case the new adaptive mode will be just be adding and removing bits here and there without actually reducing or increasing the bitrate significantly?Ideally, yes.

[ToS_Maverick]

then why does it lead to a massive undersize with this sample, while it actually should increase the bitrate?

and why are the final quants so low (15-17)?

[Dark Shikari]

then why does it lead to a massive undersize with this sample, while it actually should increase the bitrate?

and why are the final quants so low (15-17)?Can you upload the .h264 stream so I can look at it?

[ToS_Maverick]

you should have the sample, try it with these settings:
--crf 20.0 --level 4.1 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 9781 --vbv-maxrate 29400 --threads auto --thread-input --progress --no-dct-decimate --output "output" "input" --aq-strength 1.0

i can't upload it today, if you need it i'll upload it tomorrow!

thx and good night ;)

[Dark Shikari]

you should have the sample, try it with these settings:What sample? What do you mean, "you should have the sample"? That's not exactly descriptive... :rolleyes:

[kumi]

It seems the oversizing is a little better now, only +14% @ 0.9 strength. I haven't encountered any undersizing yet :rolleyes:

25.1 MB --crf 21.5
25.4 MB --crf 21.5 --aq-strength 0.3 --aq-sensitivity 0
26.7 MB --crf 21.5 --aq-strength 0.6 --aq-sensitivity 0
29.2 MB --crf 21.5 --aq-strength 0.9 --aq-sensitivity 0



I compared AQ on a 2% sample of bright, outdoor, well-shot prosumer SD DV movie. Scenes consist of lots of close-ups of people's faces talking, with lots of action in the background. No dark scenes at all.

25.1 MB --crf 21.5
vs
25.2 MB --crf 22.5 --aq-strength 0.9 --aq-sensitivity 0

Right away the most noticable improvement is in the increased facial detail and skin tones. I mean HUGE improvement. Mosquito noise, blocking and banding are all much less visible. And not just the dark and/or detailed flat areas, either. Everywhere there is detail to bring out, like humari hair, it seems to bring it out. In fact I can't find areas that look worse than before... where are the extra bits coming from?! This is voodoo magic! :eek:

If there's anything I would ask, it would be to speed it up a bit (if possible), and release a fast-ref-search/AQ binary :p But this is #$%@ing great work you've done here, thank you. :cool:

[Dark Shikari]

Right away the most noticable improvement is in the increased facial detail and skin tones. I mean HUGE improvement. Mosquito noise, blocking and banding are all much less visible. And not just the dark and/or detailed flat areas, either. Everywhere there is detail to bring out, like humari hair, it seems to bring it out. In fact I can't find areas that look worse than before... where are the extra bits coming from?! This is voodoo magic! :eek:It takes the bits from the areas with the highest variance--a very sharp boundary with strong brightness differences, for example, would get bits taken away. I'm not sure if this would have a negative effect in anime--in live action any negative effect seems to be nearly invisible.

One thing you'll find when looking at bit distribution of non-AQ encodes is that often the vast majority of the bits are concentrated in very small areas; one can easily take a few away without there being much noticeable difference.

[Sagekilla]

Everywhere there is detail to bring out, like humari hair, it seems to bring it out. In fact I can't find areas that look worse than before... where are the extra bits coming from?! This is voodoo magic! :eek:

Voodoo magic? No.. This.. Is.. SPARTA!!

@Dark Shikari: If it were to be that way, wouldn't it be a good idea to use the mode you're using right now as a "real life" mode, and then use a different type of AQ as an "anime" mode? Because, I do encode a few anime sources where I do need to use AQ, and if the new AQ will harm anime then I really think you should consider adding a switch to choose anime/real life mode or something to that effect.

[Sharktooth]

why dont you try the new AQ on your anime, see it with your eyes and report back?
it would be a really usefull info...

[Dark Shikari]

why dont you try the new AQ on your anime, see it with your eyes and report back?
it would be a really usefull info...I've tried it, and its hard to tell. It really does salvage some detail, much like in ordinary encodes, but I'm really not that sure about it.

[Sharktooth]

... it was directed to sagekilla ...

i know you probably tested it on animes too, but a second POV would be usefull...

[ToS_Maverick]

Dark Shikari, i meant the BlackPearlSample, my main testsample ;)

from your screens i could see you still got it, anyway, i posted the link here:
http://forum.doom9.org/showthread.php?p=1028047#post1028047

i used this script:
DGDecode_mpeg2source("Black.Pearl.Sample.d2v", idct=7)
trim(2,0)
crop(0,58,0,-62)
LanczosResize(768,320)

crf 15 --aq-strength 1.0:
--[NoImage] Job commandline: "C:\Programme\megui\tools\x264\x264.exe" --crf 15.0 --level 4.1 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 9781 --vbv-maxrate 29400 --threads auto --thread-input --progress --no-dct-decimate --output "F:\Video\BlackPearl\Black.Pearl.Sample crf 15 newaq10.mkv" "F:\Video\BlackPearl\Black.Pearl.Sample.avs" --aq-strength 1.0
--[Information] [16.12.2007 13:08:58] Encoding started
--[NoImage] Standard output stream
--[NoImage] Standard error stream
---[NoImage] avis [info]: 768x320 @ 23.98 fps (3622 frames)
---[NoImage] x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 SSSE3 Cache64
---[NoImage] x264 [info]: slice I:98 Avg QP:14.66 size: 33513 PSNR Mean Y:46.78 U:48.70 V:49.63 Avg:47.40 Global:46.44
---[NoImage] x264 [info]: slice P:1955 Avg QP:16.93 size: 16632 PSNR Mean Y:44.88 U:47.55 V:48.51 Avg:45.69 Global:45.52
---[NoImage] x264 [info]: slice B:1569 Avg QP:18.58 size: 6651 PSNR Mean Y:43.48 U:46.79 V:47.66 Avg:44.39 Global:44.23
---[NoImage] x264 [info]: mb I I16..4: 14.4% 24.3% 61.2%
---[NoImage] x264 [info]: mb P I16..4: 8.8% 21.7% 13.7% P16..4: 21.7% 21.7% 9.9% 0.0% 0.0% skip: 2.5%
---[NoImage] x264 [info]: mb B I16..4: 2.1% 5.5% 2.1% B16..8: 42.6% 3.6% 8.0% direct:14.7% skip:21.4%
---[NoImage] x264 [info]: 8x8 transform intra:47.9% inter:29.8%
---[NoImage] x264 [info]: ref P 74.8% 17.5% 7.7%
---[NoImage] x264 [info]: ref B 79.5% 16.6% 3.9%
---[NoImage] x264 [info]: SSIM Mean Y:0.9818539
---[NoImage] x264 [info]: PSNR Mean Y:44.322 U:47.254 V:48.177 Avg:45.170 Global:44.935 kb/s:2448.41
---[NoImage] encoded 3622 frames, 31.19 fps, 2448.63 kb/s

crf 20 --aq-strength 1.0 --aq-sensitivity 20
--[NoImage] Job commandline: "C:\Programme\megui\tools\x264\x264.exe" --crf 20.0 --level 4.1 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 9781 --vbv-maxrate 29400 --threads auto --thread-input --progress --no-dct-decimate --output "F:\Video\BlackPearl\Black.Pearl.Sample crf 20 newaq10 sens10.mkv" "F:\Video\BlackPearl\Black.Pearl.Sample.avs" --aq-strength 1.0 --aq-sensitivity 20
--[Information] [16.12.2007 11:58:52] Encoding started
--[NoImage] Standard output stream
--[NoImage] Standard error stream
---[NoImage] avis [info]: 768x320 @ 23.98 fps (3622 frames)
---[NoImage] x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 SSSE3 Cache64
---[NoImage] x264 [info]: slice I:98 Avg QP:19.66 size: 28360 PSNR Mean Y:45.00 U:47.60 V:48.59 Avg:45.74 Global:43.20
---[NoImage] x264 [info]: slice P:1955 Avg QP:21.93 size: 15696 PSNR Mean Y:44.31 U:47.18 V:48.16 Avg:45.15 Global:44.86
---[NoImage] x264 [info]: slice B:1569 Avg QP:23.58 size: 6291 PSNR Mean Y:42.89 U:46.36 V:47.24 Avg:43.83 Global:43.60
---[NoImage] x264 [info]: mb I I16..4: 15.5% 28.0% 56.5%
---[NoImage] x264 [info]: mb P I16..4: 8.2% 21.3% 14.6% P16..4: 22.7% 22.1% 8.9% 0.0% 0.0% skip: 2.3%
---[NoImage] x264 [info]: mb B I16..4: 2.3% 6.0% 2.1% B16..8: 43.3% 3.6% 7.1% direct:14.3% skip:21.2%
---[NoImage] x264 [info]: 8x8 transform intra:47.9% inter:29.6%
---[NoImage] x264 [info]: ref P 74.8% 17.6% 7.6%
---[NoImage] x264 [info]: ref B 79.4% 16.6% 4.0%
---[NoImage] x264 [info]: SSIM Mean Y:0.9807162
---[NoImage] x264 [info]: PSNR Mean Y:43.712 U:46.837 V:47.774 Avg:44.595 Global:44.223 kb/s:2294.88
---[NoImage] encoded 3622 frames, 32.24 fps, 2295.10 kb/s

for crf 20 --aq-strength 1.0 i get
918 kb/s
SSIM 0.969
which is way too low in size, SSIM and visual quality

[Dark Shikari]

for crf 20 --aq-strength 1.0 i get
918 kb/s
SSIM 0.969
which is way too low in size, SSIM and visual qualityHow about you compare two different videos, at the same bitrate, visually?

[ToS_Maverick]

ok, i think i have to express myself a bit more clearly ;)

the crf 15 vid has 150 kb/s more bitrate than the crf 20 one, that's 6.5 %. for me, thats close enough. i compared them visually of course.

to give you an idea:
vid1=directshowsource("Black.Pearl.Sample crf 15 newaq10.mkv", audio=false).lanczosresize(1280,536)
vid2=directshowsource("Black.Pearl.Sample crf 20 newaq10 sens20.mkv", audio=false).lanczosresize(1280,536)
interleave(vid1,vid2)

normally, i predict the quality of my encodes with the aveage quant/ratefactor. now a sample, that is transparent at 20, suddenly needs 15. that's a bit weird for me.

[Dark Shikari]

ok, i think i have to express myself a bit more clearly ;)

the crf 15 vid has 150 kb/s more bitrate than the crf 20 one, that's 6.5 %. for me, thats close enough. i compared them visually of course.

to give you an idea:
vid1=directshowsource("Black.Pearl.Sample crf 15 newaq10.mkv", audio=false).lanczosresize(1280,536)
vid2=directshowsource("Black.Pearl.Sample crf 20 newaq10 sens20.mkv", audio=false).lanczosresize(1280,536)
interleave(vid1,vid2)

normally, i predict the quality of my encodes with the aveage quant/ratefactor. now a sample, that is transparent at 20, suddenly needs 15. that's a bit weird for me.I'll have to do some testing with this to see why the bitrate is changing so much--whether its the fact that it doesn't need all that bitrate when AQ is applied, or whether AQ is just being applied unevenly.

I might have found the problem. It might be because of the black on the bottom and the top, the letterbox padding--this is counted as "flat" and so the formula screws up completely. But I'm not 100% sure about this... /goes back to testing.

[foxyshadis]

Maybe the quant accounting is just off, since it'll average out to the same quant but a very different bit allocation. It probably doesn't really matter that much, forcing the same bitrate would require modifying the RC or accounting for how many bits every quant change adds or subtracts, a lot of work for questionable gain.

[Gilgamesh83]

Hi!

Was just wondering about what the aq does in a codec,
does it:

a) pull bitrate from dynamic parts of the frame to non dynamic part of the frame?

b) pull bitrate from brighter parts of the frame to darker parts of the frame? (does it have to do with the fact that codecs give less bitrate in darker areas with the same quantizer in bright areas? or something like that.)

again a quick answer is ok since im a noob or rather, I know nothing about programming but am an avid user of x264 in megui.
I have always imagined that aq could in a frame have different quantizers to e.g. only a small dynamic parts of a frame. (like when in an anime there is a frame that is still but has a tv that shows static, that frame would still get a great about of bitrate cause of the dynamics in the static tv part.)

[Dark Shikari]

a) pull bitrate from dynamic parts of the frame to non dynamic part of the frame?This would be some sort of motion-based AQ. I've never seen one, personally.
b) pull bitrate from brighter parts of the frame to darker parts of the frame? (does it have to do with the fact that codecs give less bitrate in darker areas with the same quantizer in bright areas? or something like that.)That's called brightness-based AQ. Elecard supports this, and regular x264 AQ, though not brightness-based, is thresholded by brightness.

Regular x264 AQ finds the flattest parts of the frame (least complex) and gives the more bits. Mine is somewhat similar, but does it using different math and is much more willing to move bits around.

[Dark Shikari]

A test of the AQ, at 2 megabits per second:

Original (no AQ) (http://mirror05.x264.nl/Dark/force.php?file=./Original.mkv)

New AQ (http://mirror05.x264.nl/Dark/force.php?file=./AQTest.mkv)

Notice the massively better grain retention with the AQ.

[Sharktooth]

what settings did you use for those 2 encodings?
also how does it behave at very low quantizers (below 18)?
it seems in the original one, background grain is there on the i frame, but on Bs it gets washed. it does the same with AQ, but it "drops" much less.

[Dark Shikari]

what settings did you use for those 2 encodings?
also how does it behave at very low quantizers (below 18)?
it seems in the original one, background grain is there on the i frame, but on Bs it gets washed. it does the same with AQ, but it "drops" much less.I used 3pass with pretty much maxed settings, with --no-dct-decimate. Trellis 1.

I-frames will obviously have better grain because they have lower quantizers.

I'm testing a 5 megabit encode right now to see behavior at lower quantizers.

Also note that the original video clip isn't the best quality--some blurring and even blocking is actually from the original.

[Gilgamesh83]

This would be some sort of motion-based AQ. I've never seen one, personally.
That's called brightness-based AQ. Elecard supports this, and regular x264 AQ, though not brightness-based, is thresholded by brightness.

Regular x264 AQ finds the flattest parts of the frame (least complex) and gives the more bits. Mine is somewhat similar, but does it using different math and is much more willing to move bits around.

thx for the answer. would be cool if a motion-based aq existed.

[foxyshadis]

Motion-based isn't that interesting. You'd really want some sort of area of visual interest AQ, but that is firmly in the Hard Problem territory. You'd practically need the director or someone intimately familiar with the film highlight the area the eye is focusing on in every frame. Researching is progressing in this every year, though, there's a lot of papers out there if anyone ever wants to take a (quixotic imho) stab at it.

The quantizer, deadzone, and custom matrix are all there to tweak how the codec responds to motion and detail, and generally do a fine job; it's grain that has generated nearly all of the complaints over the years. The lousy performance of h.264 with grain is the main reason VC-1 even exists.

btw, xvid and ffmpeg use method b and call it lumimasking. x264 doesn't need that since it doesn't have the same overquantization problems in dark areas they do, although there's some overlap when dark areas are flat but grainy.

[ToS_Maverick]

@Dark Shikari:
did you find something out during your testing? just curious ;)

[foxyshadis]

Also: It definitely helps reduce blocking of gradients in anime. It can increase bitrate by a pretty inordinate amount in some scenes, though, even with zero visual difference - anime's probably so flat that the algorithm goes a little crazy.

[Valeron]

hi, Dark Shikari, not good news from my anime encode experience.
crf18 same setting, one with ur new AQ strength 1.0 and threshold 0(automatic) enable, the other disable AQ, the AQ enable one looks bad compare to the no AQ encode. And is 27MB larger in size.
If u would like some screen shot, I can post here tomorrow.

[Dark Shikari]

hi, Dark Shikari, not good news from my anime encode experience.
crf18 same setting, one with ur new AQ strength 1.0 and threshold 0(automatic) enable, the other disable AQ, the AQ enable one looks bad compare to the no AQ encode. And is 27MB larger in size.
If u would like some screen shot, I can post here tomorrow."Bad" is sort of a bad term--screenshots are definitely useful to illustrate. Also, comparing to files with different sizes is generally bad, too.

[burfadel]

"Bad" is sort of a bad term--screenshots are definitely useful to illustrate. Also, comparing to files with different sizes is generally bad, too.

Especially since CRF is constant quality, not constant bitrate (which would end up with the same file size). The use of P and B frames, as well as macroblocks would also be different with constant quality and AQ enabled (?), so although the image may have the same CRF, the filesize will end up being different! The filesize could go either way?...

[Dark Shikari]

Especially since CRF is constant quality, not constant bitrate (which would end up with the same file size). The use of P and B frames, as well as macroblocks would also be different with constant quality and AQ enabled (?), so although the image may have the same CRF, the filesize will end up being different! The filesize could go either way?...And of course comparing individual frames is also bad unless the GOPs have the same structure--comparing a B-frame to an I-frame, for example, is just retarded.

Best way to compare is just to run two two-pass encodes, one with AQ and one without, and comparing the result.

[Sharktooth]

i'd suggest to encode at a target bitrate and see if the AQ encode looks better...

[zbutsam]

I read in a post on this topic that this AQ patch could improve the encoding of scenes with grass textures and I had a great clip to test it on. It is a trailer of the film "Kicking and Screaming" which contains a lot of action on football fields.
The clip (originally in 720p) was resized to 560x304 for speed's sake and encoded in MeGUI using the HQ-Fast profile and with a two-pass target bitrate of 700kbits (rather low I know but I wanted to see how the AQ would react with lower bitrates).
As a reference I used the latest x264 build that MeGUI would download (709).
For the AQ x264.exe I added the switches
--aq-strength 1.0 --aq-sensitivity 20.

The results were great:) Whereas the original would struggle with the low bitrate having to blur details on the grass and producing a flat effect, the new AQ managed to preserve much more detail without noticable loss of quality anywhere else.

On a 1000 kbit 2-pass I tried just with the AQ build it retained much more detail and the picture was sharp (however I will have to go back and repeat this last test with the unmodified build to see how that does).

Speed-wise the build with the AQ patch is about 10% slower for me.

I will try to post more tests later but, so far, congratulations :) it seems to be working great

[Sagekilla]

I read in a post on this topic that this AQ patch could improve the encoding of scenes with grass textures and I had a great clip to test it on. It is a trailer of the film "Kicking and Screaming" which contains a lot of action on football fields.
The clip (originally in 720p) was resized to 560x304 for speed's sake and encoded in MeGUI using the HQ-Fast profile and with a two-pass target bitrate of 700kbits (rather low I know but I wanted to see how the AQ would react with lower bitrates).
As a reference I used the latest x264 build that MeGUI would download (709).
For the AQ x264.exe I added the switches
--aq-strength 1.0 --aq-sensitivity 20.

The results were great:) Whereas the original would struggle with the low bitrate having to blur details on the grass and producing a flat effect, the new AQ managed to preserve much more detail without noticable loss of quality anywhere else.

On a 1000 kbit 2-pass I tried just with the AQ build it retained much more detail and the picture was sharp (however I will have to go back and repeat this last test with the unmodified build to see how that does).

Speed-wise the build with the AQ patch is about 10% slower for me.

I will try to post more tests later but, so far, congratulations :) it seems to be working great

700 kbps actually isn't that too low for 560x304. I manage to get around 1.1 mbps (300, surprisingly) to about 2 mbps on most of my encodes @ 864x480. Then again, I do typically enable most settings except for esa. On some of my encodes I've decided to just go with the full 16 refs since they're so slow to begin with because of preprocessing.


In any case, that's very interesting to hear. I'm waiting for the preprocessing to finish on one of my videos before I decide how to tackle it with AQ. Last encode I ran on it, I ended up using an older build of the new AQ and the newer builds seem to be doing an even better job, so I'm redoing it for the probably 8th time now.

[Dark Shikari]

700 kbps actually isn't that too low for 560x304. I manage to get around 1.1 mbps (300, surprisingly) to about 2 mbps on most of my encodes @ 864x480.The main issue that I find, however, is that without my AQ, you need very high bitrates to retain fine background detail, like grass; it simply doesn't put the bits where they need to be. As a result, you end up needing vastly higher bitrates to achieve transparency, even though most of those bits end up wasted.

[Sagekilla]

The main issue that I find, however, is that without my AQ, you need very high bitrates to retain fine background detail, like grass; it simply doesn't put the bits where they need to be. As a result, you end up needing vastly higher bitrates to achieve transparency, even though most of those bits end up wasted.


Neat, all the more for me to be excited about re-encoding 300 for the zillionth time. At the bitrates I said above, I usually find the videos to be mostly transparent (except backgrounds which tend to be slightly blocked but manageable) I just hope the rate control won't get screwed up badly using crf 18. By the way, why did you say that grass tends to be unfairly smeared by x264 again? I always found that a bit of an odd quirk.

[Sharktooth]

metrics do not represent the eyes perception. they're just numbers that represent an average deviation from the source picture. since x264 internal stuff was made to get the best compression keeping high level of metrics, sometimes the codec produce unwanted (visually speaking) results.
however every codec developer is more or less using the same method coz it's easier to compare eventual (metric) improvements and that leads to a faster development. when algos are optimized and the compression gets close to the theoretical maximum, then visual optimizations (psy and other stuff) are introduced to obtain a visually pleasing picture.
in other words, x264 has a very good compression but the picture quality can be improved drastically.

[rhester72]

Since there are no current diffs, any chance we could get the test EXEs compiled with MP4 output support?

Rodney

[Dark Shikari]

Here's a diff... (http://pastebin.com/f54db05b9)

I really need to work on the automatic thresholding and the formula though--there are some cases in which the AQ really doesn't work well. I've been busy lately though--final exams and watching Haruhi.

[Sagittaire]

Well here a well know psy optimisation for noise/grain: for HVS noise in dark area is useless.

1) make pre-process for reduce noise in dark area. Make strong denoising/degraining in dark area (with lumi < 40 for example).

2) Use lower quant in dark area for better HVS quality in dark area. After "dark denoising" dark area will be more compressible. Use high quality (low quantizer) for dark area is really important for TFT screen.

3) Use higher quant in for complex texture. With this HVS AQ the quality for flat area will be really better. Use "dark denoising pre-process" and "spacial complexity AQ" will produce directly better quality for dark area.

[ToS_Maverick]

Dear Dark Shikari, I got a pre-christmas present for you :D

i recorded PotC1 some time ago from HDTV and can now present you, the same sample in 1080p broadcasted at about 6 MBit:
http://www.megaupload.com/de/?d=O262L4JJ

to get the same screen size and picture area, i use this script:
directshowsource("Black.Pearl.Sample HD.mkv",audio=false,fps=25)
trim(82,3703)
ColorMatrix(mode="Rec.709->Rec.601")
crop(0,140,0,-140)
lanczosresize(768,320)

the broadcast isn't perfect, but the very fine detail in the background is preserved very well, which should be good input for your AQ!

have fun :cool:

[kumi]

Good luck with your exams, and Haruhi :) I hope you find time to adjust the automatic thresholding for use with crf mode. I know that you said this isn't like constant bitrate mode and we shouldn't expect filesize parity, but a little more predictability would be real nice. I just finished an encode that came out massively oversized, (2.3GB vs 1.4GB without AQ). Other movies haven't been nearly as bad, though.

Well I guess I should have been doing a prediction pass from the start... stupid me :p

Happy holidays, everyone

[Dark Shikari]

Good luck with your exams, and Haruhi :) I hope you find time to adjust the automatic thresholding for use with crf mode. I know that you said this isn't like constant bitrate mode and we shouldn't expect filesize parity, but a little more predictability would be real nice. I just finished an encode that came out massively oversized, (2.3GB vs 1.4GB without AQ). Other movies haven't been nearly as bad, though.

Well I guess I should have been doing a prediction pass from the start... stupid me :p

Happy holidays, everyoneThe other issue is that edges are really getting screwed up in some cases with my AQ; my algorithm seems to work at its absolute best when there are no sharp edges in the video, and worst when there are plenty--so perhaps I have to deal with edge masking or similar.

[Sagekilla]

Hmm, the latest build (0.3) is rather quirky in your transmagical adaptive sensitivity mode with crf . Strength 1 I got 1.6 mbps on a 720p resized portion of that PotC source. Increasing strength to 2 made the bitrate go down further, to 1.3 mbps! I'm decreasing crf to 16.5 to see if that'll change anything, but the rate control really gets screwed up with your transmorphmagical mode :(

[kumi]

Rate control gets out of whack, but it does have potential... I've been trying to match the (overall) quality achieved with aq-sensitivity 0 on a certain source... and no amount of fiddling with >0 aq-sensitivity values was able to approach it.

[Dark Shikari]

Rate control gets out of whack, but it does have potential... I've been trying to match the (overall) quality achieved with aq-sensitivity 0 on a certain source... and no amount of fiddling with >0 aq-sensitivity values was able to approach it.It seems to be absurdly source-dependent--which is what AQ should try to avoid.

This winter, I'll try to find a good way to fix it if I can. Its a tough problem.

[Gromozeka]

When you can add AQ in official build? :)
It already now gives very big positive effect.
It would allow more better testing yours AQ many people From the different countries
Thank you

[Sharktooth]

AQ is not ready yet. stop asking those silly questions.
all Dark Shikari needs is testing from qualified persons, not all idiots on the planet.

[ToS_Maverick]

@Dark Shikari and Sharktooth:
What type of samples, genres, ... still need to be tested? It would be nice to have a list or sth, to know which samples are still open.

maybe i got some other useful stuff that could be tested.

[Gromozeka]

To Sharktooth
Слышь ты, умник, производное децибела, аля мозг планеты, будь поскромнее! Мои вопросы может и не блещут ни познанием английского языка, ни алгоритмами программирования, но я подозреваю, что ты мог бы покумекать своими мозгами и быть несколько вежливее.
А если ты на это не способен то закрой свой ротик, зачехли рога на башке и сопи в тряпочку!
С нескрытым раздражением, но уважением, Игорь

[Sagekilla]

When you can add AQ in official build? :)
It already now gives very big positive effect.
It would allow more better testing yours AQ many people From the different countries
Thank you


Personally I don't think it's ready yet. Currently the rate control for constant rate factor is completely off when using adaptive sensitivity for AQ, so it'll take a lot of testing before AQ will be introduced to svn since it's completely failing with crf.

[sp@rrow]

Gromozeka
Бугага, эта 5 :-)))

[fields_g]

To Sharktooth
Слышь ты, умник, производное децибела, аля мозг планеты, будь поскромнее! Мои вопросы может и не блещут ни познанием английского языка, ни алгоритмами программирования, но я подозреваю, что ты мог бы покумекать своими мозгами и быть несколько вежливее.
А если ты на это не способен то закрой свой ротик, зачехли рога на башке и сопи в тряпочку!
С нескрытым раздражением, но уважением, Игорь

Igor (Gromozeka),
You've missed years of people asking for AQ to be added to SVN. Not only that, but you have also missed years of different methods, tweaks, and revisions. AQ (or other psy enhancements) is greatly needed, but is not currently stable.

I think the x264 community can be proud of the integrity of our SVN. Patches are accepted into SVN when the developers are comfortable with what it does and feel they can maintain the addition properly. Feel free to compile x264 yourself with any available patches you choose. In fact, I would guess that the majority of people here use builds that are NOT pure SVN.

BTW.... You might want to brush up on Rule 13.

[Sharktooth]

just as a reminder...
13) The official language is English. Outside the translator forum English is the only allowed language.

[Gromozeka]

I think the x264 community can be proud of the integrity of our SVN. Patches are accepted into SVN when the developers are comfortable with what it does and feel they can maintain the addition properly. Feel free to compile x264 yourself with any available patches you choose. In fact, I would guess that the majority of people here use builds that are NOT pure SVN.

BTW.... You might want to brush up on Rule 13.

I have understood you on 75 %. Russian language is combined even for Russian people at times. And to communicate here on it it is wrong. Thanks for all

[Chainmax]

fields_g: Gromozeka's post was polite and, more importantly, was volunteering for testing. What you described is not an excuse for Shartooth's awfully rude, insulting and snobbish retort. He should know better.


just as a reminder...
13) The official language is English. Outside the translator forum English is the only allowed language.

Just another reminder:
4) Be nice to each other and respect the moderator. Profanity and insults will not be tolerated. If you have a problem with another member turn to the respective moderator and if the moderator can't help you send a private message to Doom9

[bond]

yeah everyone keep cool please...

[desta]

Sorry to ask probably an obvious question, but does "--aq-sensitivity 0" have to be input to use the 'automatic' thresholding? Going by --help and the info in the first post, I would've assumed that automatic is.. automatic, but going from certain other posts in this thread, it seems it does need to be input.

[Dark Shikari]

Sorry to ask probably an obvious question, but does "--aq-sensitivity 0" have to be input to use the 'automatic' thresholding? Going by --help and the info in the first post, I would've assumed that automatic is.. automatic, but going from certain other posts in this thread, it seems it does need to be input.Automatic wasn't the default until version 0.3. Now it is.

[desta]

Ah I see. Thanks for clarifying. :)

[Ranguvar]

I have a video that would be very good for testing this, IMO. I'd like to do so and provide screenshots.

It's an HD trailer for a video game. Not capped by myself, provided by GameTrailers. Would it be a Rule 6?

[Dark Shikari]

I have a video that would be very good for testing this, IMO. I'd like to do so and provide screenshots.

It's an HD trailer for a video game. Not capped by myself, provided by GameTrailers. Would it be a Rule 6?Nope, those are fine to use.

[Sharktooth]

fields_g: Gromozeka's post was polite and, more importantly, was volunteering for testing. What you described is not an excuse for Shartooth's awfully rude, insulting and snobbish retort. He should know better.
ppl need to learn to speak only when necessary and just not to blow air out of their mouth.
the fact there were like 1 million of ppl requesting AQ in the SVN it doesnt mean it DESERVES to be there.
There are several test builds and i cant see the reason why incomplete and experimental code should be put in the x264 SVN.
Also, since it has been asked SO MUCH TIMES (and the answer was always the same), he could SEARCH before posting (you should too).
No offense, it's just my way...
and excuse me for the OT.

[bond]

guys, again, keep cool and nice please. next offense, no matter which one, will get striked

[ToS_Maverick]

@Dark Shikari
i played around with VC-1 a bit, while an idea struck me:
would it make sense, to apply AQ only to I or I and P frames?

i think it would be interesting to see the effect of this. quality and metric-wise and if you could save some bitrate by leaving the B frames out.

[Dark Shikari]

@Dark Shikari
i played around with VC-1 a bit, while an idea struck me:
would it make sense, to apply AQ only to I or I and P frames?

i think it would be interesting to see the effect of this. quality and metric-wise and if you could save some bitrate by leaving the B frames out.That's not a bad idea, since the main disadvantage of my AQ is actually the cost of encoding the qp_delta bits.

[ToS_Maverick]

great that i could be helpful!

well since VC1 is using things from AVC, why not have a look at what they are doing.

at the MS VC1 codec you can set this. i got the idea because the background grain started to "update" with every I frame. i could not test the I/P setting since AVS2ASF seems bit buggy.

[mahsah]

Any idea what settings (if any) for AQ I could use to retain the dithering added by Gradfun2db?

[Sagekilla]

Any idea what settings (if any) for AQ I could use to retain the dithering added by Gradfun2db?

Dithering is actually quite "complex" since it's not a uniform, flat color like the sky would be. Since it makes use of bunching together a bunch of different colors in a set pattern to make it LOOK like another color, it'll end up getting mushed because it looks like grain or noise.

And since AQ looks to increase the bits allocated towards flat areas so that it isn't blocky (dithering looks flat, but to the human eye, not to an algorithm) but since dithering isn't like this, like I just explained above, it won't work well.

[Dark Shikari]

And since AQ looks to increase the bits allocated towards flat areas so that it isn't blocky (dithering looks flat, but to the human eye, not to an algorithm) but since dithering isn't like this, like I just explained above, it won't work well.My algorithm will most definitely consider dithered blocks to be very very close to flat, and as a result will decrease their quantizer (though not necessarily by enough to keep the dither accurately).

[Sagekilla]

My algorithm will most definitely consider dithered blocks to be very very close to flat, and as a result will decrease their quantizer (though not necessarily by enough to keep the dither accurately).

Oho, that's interesting. Is this something unique to your latest version?

[Dark Shikari]

Oho, that's interesting. Is this something unique to your latest version?No, its inherent in the concept--dithered blocks will have extremely low variance, and so will get the strongest AQ applied.

[burfadel]

Will the latest changes made to x264 with version 717 affect the patch? is it possible for a new build :)?!

[Dark Shikari]

Will the latest changes made to x264 with version 717 affect the patch? is it possible for a new build :)?!717 just looks like an ESA improvement.

[burfadel]

ah ok! I see that now :) wouldn't a 1.3x increase (30 percent) in ESA speed bring it reasonably close to the speed of Multi hex?

[akupenguin]

ah ok! I see that now :) wouldn't a 1.3x increase (30 percent) in ESA speed bring it reasonably close to the speed of Multi hex?
No. Maybe if you also pull in UMH's early termination and range adaption, so that only the multi-hexagon part is replaced by ESA.

[Dark Shikari]

No. Maybe if you also pull in UMH's early termination and range adaption, so that only the multi-hexagon part is replaced by ESA.While we're at it, how much does this speed up ESA SATD?

[Sagekilla]

While we're at it, how much does this speed up ESA SATD?

Pardon my slightly off topic question but isn't the current esa actually a completely different algo? I remember you referring to it as "SEA," what exactly does that stand for?


On a side note, nice to hear that esa got a 30% boost in speed. I've been using esa (along with a number of other insane settings) on relatively short clips and I've been pleased with the (slight, but noticeable) improvement over umh.

[Dark Shikari]

Pardon my slightly off topic question but isn't the current esa actually a completely different algo? I remember you referring to it as "SEA," what exactly does that stand for?

On a side note, nice to hear that esa got a 30% boost in speed. I've been using esa (along with a number of other insane settings) on relatively short clips and I've been pleased with the (slight, but noticeable) improvement over umh.ESA SATD, at least Aku's version, uses SEA.

SEA is Sequential Elimination, and it uses a layered image representation to losslessly eliminate candidates (IIRC).

[Sagekilla]

Also, how compatible are the various speed and quality patches (fast ref, new aq) with rev 715? I'm looking to compile my own build with fast ref search, the older AQ algo (new one is dodgy with crf), and get the nice esa speed boost from the latest patch.

[ToS_Maverick]

just for the record

the new AQ at
--aq-strength 1.0 --aq-sensitivity 17

hast the same size as the old AQ @
--aq-strength 0.6 --aq-sensitivity 10 or --aq-strength 0.9

with all CRFs (tested on 18, 20 and 22)
with a higher visual quality.

[JvA_]

Dark Shikari, could you please make a diff against the latest SVN checkout, if possible?

Except knowledge of C/C++, how much mathematical knowledge is required to start hacking on x264? I have no previous knowledge on video compression, but I've had some courses about fourier-transform and similar math. I've seen JPEG2000 uses a lot of the math I've studied so far, so if MPEG4 has similarities I suppose I would understand a lot ;)

So, what do you recommend? Start digging the source code right away, or are there documents that I should read first that will give me a better "hands on"?

[Dark Shikari]

Dark Shikari, could you please make a diff against the latest SVN checkout, if possible?I don't think there should be any incompatibilities between my old patch and the current SVN. If there are, I'll fix them.
Except knowledge of C/C++, how much mathematical knowledge is required to start hacking on x264? I have no previous knowledge on video compression, but I've had some courses about fourier-transform and similar math. I've seen JPEG2000 uses a lot of the math I've studied so far, so if MPEG4 has similarities I suppose I would understand a lot ;)You don't need much math knowledge. There are dozens of places in x264 that you can start hacking at without even knowing more than a small amount about video compression, since one can easily treat every other part of the code as a "black box" and ignore how it works. This is why I started on me.c and was able to do a few useful things even as a clueless newbie back in the day.

So, what do you recommend? Start digging the source code right away, or are there documents that I should read first that will give me a better "hands on"?Dig through the source code while asking every question you can think of in #x264dev on Freenode. You'll learn faster than you thought you ever could.

[JvA_]

Thanks for your reply!

I tried to patch the source code with your web-published diff using patch -Np0 -i ../path/to/the/file.txt standing in the checked out x264 directory. Got a reject on every line it tried to add. If you got time, please change the diff so it matches. Can't wait to try this AQ :)

[Dark Shikari]

Thanks for your reply!

I tried to patch the source code with your web-published diff using patch -Np0 -i ../path/to/the/file.txt standing in the checked out x264 directory. Got a reject on every line it tried to add. If you got time, please change the diff so it matches. Can't wait to try this AQ :)Are you patching the SVN code, or the web code? The code on mirror05 is already patched with the other AQ, which would result in the errors.

[JvA_]

I'm patching the code I've fetched from:
svn co svn://svn.videolan.org/x264/trunk x264

[Dark Shikari]

New AQ is out. Massive changes.

1. Totally rewritten AQ. Same basic concept, but now uses a logarithmic scale instead of a hackneyed exponential one.
2. For B-frames, uses a tricky bit of lambda-changing instead of QP changing; this requires absolutely no bits for QP-deltas!
3. Totally rewritten, far faster automatic sensitivity. Respects bitrate in CRF mode better also.

Will post it in the original post soon.

[Sagekilla]

Very nice, I'll go try this out and see how well it behaves on my encoding. I've been holding out on a number of movies because I couldn't get decent flat detailed areas with good quality.

[akupenguin]

Pardon my slightly off topic question but isn't the current esa actually a completely different algo? I remember you referring to it as "SEA," what exactly does that stand for?
Technically ESA and SEA are different algorithms. But since they produce the same result and differ only in implementation details, I saw no reason to rename the commandline option when I switched algorithm in r388.

[mitsubishi]

The link for 0.4 doesn't seem to work: "Invalid Quickkey. This error has been forwarded to MediaFire's development team."

[Dark Shikari]

The link for 0.4 doesn't seem to work:Fixed.

[Atak_Snajpera]

Thanks Dark Shikari !
Finally I'm fully convinced to new AQ algorithm :)
No more dancing blocks on blue sky :)

BTW strength 1.0 gives me the best result. I've noticed also that sometimes file size is even lower than 0.5 in CQ mode.

[MasterNobody]

Dark Shikari
May be you will also upload source diff with current x264 trunk so people can compile own's builds (for example, I want to make experimental x264vfw version with this new AQ patch)

[Dark Shikari]

Dark Shikari
May be you will also upload source diff with current x264 trunk so people can compile own's builds (for example, I want to make experimental x264vfw version with this new AQ patch)
Patch version 0.41 (http://pastebin.com/f7fb3770d). 0.41 is just mainly documentation/code cleanup/code comments.

[ToS_Maverick]

Dark Shikari, i got bad news for you:

just did a quick test with these settings:

--crf 20.0 --level 3 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 1835 --vbv-maxrate 10000 --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "output" "input" --aq-strength 1.0
and got the following result
http://img444.imageshack.us/img444/2190/newaqpy2.th.png (http://img444.imageshack.us/my.php?image=newaqpy2.png)

for comparison your 0.3 algo:
http://img256.imageshack.us/img256/7029/aqck8.th.png (http://img256.imageshack.us/my.php?image=aqck8.png)

otherwise, your new AQ looks very promising, but i got the feeling that the grain "stutters" a little bit? like it has only 1/2 or 1/4th of the fps.

[Dark Shikari]

Dark Shikari, i got bad news for you:

just did a quick test with these settings:

--crf 20.0 --level 3 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 1835 --vbv-maxrate 10000 --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "output" "input" --aq-strength 1.0
and got the following result
http://img444.imageshack.us/img444/2190/newaqpy2.th.png (http://img444.imageshack.us/my.php?image=newaqpy2.png)

for comparison your 0.3 algo:
http://img256.imageshack.us/img256/7029/aqck8.th.png (http://img256.imageshack.us/my.php?image=aqck8.png)

otherwise, your new AQ looks very promising, but i got the feeling that the grain "stutters" a little bit? like it has only 1/2 or 1/4th of the fps.Whoa... something went wrong there... :p

Its likely the "stuttering" problem is due to the lambda trick in B-frames, since that really doesn't retain grain very well. I can add a commandline to either do AQ on all frames, AQ on non-B-frames and lambda on B-frames, or AQ on just P-frames.

Edit: Tested your command on the exact same source and didn't get the problem you got...

[DeathTheSheep]

Is this AQ patch applicable on top of hadamard, me-prepass, etc for r720?

[Dark Shikari]

Is this AQ patch applicable on top of hadamard, me-prepass, etc for r720?fpel-cmp seems to be incompatible as of r717, so you'll have to ask Pengvado to fix that.

ME-prepass should continue to be fine. Both should have no problem with AQ, and AQ should be fine with RDRC also.

[DeathTheSheep]

RDRC? Rate distortion rate control? I must have missed this...!

Anyway, I guess I'm backtracking to r680 until the patches gain some steam again. I'm guessing it was the ESA speedup that caused the incompatibilities, and r680 is pretty much the same speed/quality-wise for me without it.

Will this AQ work with 680 on top of the other patches?

PS: I'm not very compelling when it comes to asking people to do things, so a new fpel-cmp feels like a pipe dream. :)

[Dark Shikari]

RDRC? Rate distortion rate control? I must have missed this...!

Anyway, I guess I'm backtracking to r680 until the patches gain some steam again. I'm guessing it was the ESA speedup that caused the incompatibilities, and r680 is pretty much the same speed/quality-wise for me without it.

Will this AQ work with 680 on top of the other patches?

PS: I'm not very compelling when it comes to asking people to do things, so a new fpel-cmp feels like a pipe dream. :)AQ should work fine on r680 as far as I can think.

fpel-cmp just has to be updated--it doesn't have to be rewritten.

RDRC... come on #x264dev, and learn all about the potential 1db+ PSNR gain :p

[ToS_Maverick]

ok, now i reencoded the sample and got 3!!! different file-sizes?!

please reencode the sample and look at the file sizes, i hope you get the same result...

obviously there is something strange happening here

[Dark Shikari]

ok, now i reencoded the sample and got 3!!! different file-sizes?!

please reencode the sample and look at the file sizes, i hope you get the same result...

obviously there is something strange happening here
I just ran it three times... bitwise equivalent result. Something must be wrong with your machine.

[Atak_Snajpera]

I have no problems neither. Core2Duo 1.86GHz overclocked to 2.8GHz :)

[DeathTheSheep]

Overclock? Strange results? Data processing corruption? Hmm, something smells...like burning CPU!! Is your system stable, ToS_Maverick?

Oh and DS, hadamard doesn't work with r716 either. Compiles but crashes (and burns). :)

I'm wondering if r681 is a good choice, now that I look carefully, since it seems to integrate your improved subme7, correct? (As you can see, I want to test this with some insane options. Adaptive quantization goes best with static insanity, does it not?).

[Atak_Snajpera]

Download Go-orthos and check if your cpu is stable (run at least few hours)

[G_M_C]

Whoa... something went wrong there... :p

Its likely the "stuttering" problem is due to the lambda trick in B-frames, since that really doesn't retain grain very well. I can add a commandline to either do AQ on all frames, AQ on non-B-frames and lambda on B-frames, or AQ on just P-frames.

Edit: Tested your command on the exact same source and didn't get the problem you got...

The grainpulsing might be less when you preprocess the source with grainoptimizer (http://forum.doom9.org/showthread.php?p=1052870#post1052870).

[ToS_Maverick]

thank you all for your tips!

if my pc would be instable, many other programs would crash, i should get freezes, BSODs and whatever, but my machine is rock solid. i even underclocked it now to test.

i did a lot of tests now, but could not always get bit identical results.

what i tested:
my machine (C2D 6600@2.33GHz)
AQ 0.4 build - 4 of 6 identical
AQ 0.4 build no AQ - none
rev 720 std - 2 of 3 identical

father's machine (Athlon X2 3800 all std)
AQ 0.4 build - none
rev 720 std - none

the identical files seem to be related with the current system load. while doing nothing during the encoding process, the files have a higher chance to get identical. it's very strange that the athlon doesn't produce identical files. shouldn't a program deliver correct results, no matter what runs nearby?

maybe someone else has a better explanation to this? and sry for being a bit off topic.

[akupenguin]

Before trying to debug nondeterministic r680 (if that's what you're doing), read r713 (http://trac.videolan.org/x264/changeset/713). That wouldn't cause colored snow like ToS_Maverick saw, but it was a bug.

[Dark Shikari]

Two bugs have been discovered.

1. The modified lambda system doesn't handle chroma QPs properly, which causes breakage at high QPs. This has been fixed in my latest internal build. It will be uploaded soon. In the meantime, here's the updated patch (http://pastebin.com/f499214d3).

2. The deadzone lambda changing is somewhat broken. Using trellis is recommended until this is fixed.

[Razorholt]

trellis 1 or 2? Doesn't matter?

Thanks
- Dan

[Dark Shikari]

trellis 1 or 2? Doesn't matter?

Thanks
- Dan2 will completely eliminate the use of deadzone, ensuring no problems ever. 1 is probably sufficient.

[DeathTheSheep]

Aku: Then could you update your SATD patch for 720? It would make testing this more beneficial in the extreme scenarios.

Also, Dark Shikari: How about a refresh of that me-prepass? Last I checked it was thrown all over the place (b0rked diffs, missing lines, updates, etc)?

'Twould truly be much appreciated on my part. :)

[desta]

So using this AQ + deadzones is a no go (for now)?

[Dark Shikari]

Use this new patch:
]Index: encoder/encoder.c
===================================================================
--- encoder/encoder.c (revision 720)
+++ encoder/encoder.c (working copy)
@@ -374,7 +374,7 @@
h->param.analyse.i_direct_mv_pred = X264_DIRECT_PRED_SPATIAL;
}
}
-
+
if( h->param.rc.i_rc_method < 0 || h->param.rc.i_rc_method > 2 )
{
x264_log( h, X264_LOG_ERROR, "no ratecontrol method specified\n" );
@@ -472,6 +472,8 @@
if( !h->param.b_cabac )
h->param.analyse.i_trellis = 0;
h->param.analyse.i_trellis = x264_clip3( h->param.analyse.i_trellis, 0, 2 );
+ if( h->param.analyse.b_aq && h->param.analyse.f_aq_strength <= 0 )
+ h->param.analyse.b_aq = 0;
h->param.analyse.i_noise_reduction = x264_clip3( h->param.analyse.i_noise_reduction, 0, 1<<16 );

{
@@ -1020,6 +1022,32 @@
x264_macroblock_slice_init( h );
}

+//Finds the total AC energy of the block in all planes.
+static int ac_energy_mb(x264_t *h)
+{
+ DECLARE_ALIGNED( static uint8_t, zero[16], 16 );
+ int sad = h->pixf.sad[PIXEL_16x16](zero,0,h->mb.pic.p_fenc[0],FENC_STRIDE) >> 4;
+ int ssd = h->pixf.ssd[PIXEL_16x16](zero,0,h->mb.pic.p_fenc[0],FENC_STRIDE);
+ int totalSSD = ssd - (sad * sad);
+ sad = h->pixf.sad[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[1],FENC_STRIDE) >> 3;
+ ssd = h->pixf.ssd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[1],FENC_STRIDE);
+ totalSSD += ssd - (sad * sad);
+ sad = h->pixf.sad[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[2],FENC_STRIDE) >> 3;
+ ssd = h->pixf.ssd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[2],FENC_STRIDE);
+ totalSSD += ssd - (sad * sad);
+ return totalSSD;
+}
+
+//Find the total SATD score of a block. Represents the block's overall complexity (bit cost) for intra encoding.
+static int satd_mb(x264_t *h)
+{
+ DECLARE_ALIGNED( static uint8_t, zero[16], 16 );
+ int totalSATD = h->pixf.satd[PIXEL_16x16](zero,0,h->mb.pic.p_fenc[0],FENC_STRIDE);
+ totalSATD += h->pixf.satd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[1],FENC_STRIDE);
+ totalSATD += h->pixf.satd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[2],FENC_STRIDE);
+ return totalSATD;
+}
+
static void x264_slice_write( x264_t *h )
{
int i_skip;
@@ -1045,7 +1073,51 @@
}
h->mb.i_last_qp = h->sh.i_qp;
h->mb.i_last_dqp = 0;
-
+ x264_cpu_restore(h->param.cpu);
+ /* Adaptive AQ sensitivity algorithm. */
+ if(h->param.analyse.f_aq_sensitivity == 0 && h->param.analyse.f_aq_strength != 0)
+ {
+ double total = 0;
+ double n = 0;
+ /* FIXME: Easier way to iterate over MBs? Do we need to do the full cache_load? */
+ /* FIXME: Some of the SATDs might be already calculated elsewhere (ratecontrol?). Can we reuse them? */
+ /* FIXME: Store the data, then do the logs after, to avoid the cpu_restores every single cycle? */
+ /* FIXME: Is chroma SATD necessary? */
+ for( mb_xy = h->sh.i_first_mb; mb_xy < h->sh.i_last_mb; )
+ {
+ const int i_mb_y = mb_xy / h->sps->i_mb_width;
+ const int i_mb_x = mb_xy % h->sps->i_mb_width;
+ x264_macroblock_cache_load( h, i_mb_x, i_mb_y );
+ int energy = ac_energy_mb(h);
+ x264_cpu_restore(h->param.cpu);
+ /* Weight the energy value by the SATD value of the MB. This represents the fact that
+ the more complex blocks in a frame should be weighted more when calculating the optimal sensitivity.
+ This also helps diminish the negative effect of large numbers of simple blocks in a frame, such as in the case
+ of a letterboxed film. */
+ if(energy != 0)
+ {
+ int satd = satd_mb(h);
+ x264_cpu_restore(h->param.cpu);
+ total += log(energy) * satd;
+ n += satd;
+ }
+ if( h->sh.b_mbaff )
+ {
+ if( (i_mb_y&1) && i_mb_x == h->sps->i_mb_width - 1 )
+ mb_xy++;
+ else if( i_mb_y&1 )
+ mb_xy += 1 - h->sps->i_mb_width;
+ else
+ mb_xy += h->sps->i_mb_width;
+ }
+ else
+ mb_xy++;
+ }
+ x264_cpu_restore(h->param.cpu);
+ /* Calculate and store the threshold. */
+ if(n == 0) h->aq_threshold = 100000;
+ else h->aq_threshold = expf(total / n);
+ }
for( mb_xy = h->sh.i_first_mb, i_skip = 0; mb_xy < h->sh.i_last_mb; )
{
const int i_mb_y = mb_xy / h->sps->i_mb_width;
Index: encoder/analyse.c
===================================================================
--- encoder/analyse.c (revision 720)
+++ encoder/analyse.c (working copy)
@@ -29,6 +29,7 @@
#endif

#include "common/common.h"
+#include "common/cpu.h"
#include "macroblock.h"
#include "me.h"
#include "ratecontrol.h"
@@ -2037,8 +2038,68 @@
}
}

+//Finds the total AC energy of the macroblock in all planes.
+static int ac_energy_mb(x264_t *h)
+{
+ DECLARE_ALIGNED( static uint8_t, zero[16], 16 );
+ int sad = h->pixf.sad[PIXEL_16x16](zero,0,h->mb.pic.p_fenc[0],FENC_STRIDE) >> 4;
+ int ssd = h->pixf.ssd[PIXEL_16x16](zero,0,h->mb.pic.p_fenc[0],FENC_STRIDE);
+ int totalSSD = ssd - (sad * sad);
+ sad = h->pixf.sad[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[1],FENC_STRIDE) >> 3;
+ ssd = h->pixf.ssd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[1],FENC_STRIDE);
+ totalSSD += ssd - (sad * sad);
+ sad = h->pixf.sad[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[2],FENC_STRIDE) >> 3;
+ ssd = h->pixf.ssd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[2],FENC_STRIDE);
+ totalSSD += ssd - (sad * sad);
+ return totalSSD;
+}

/*****************************************************************************
+* x264_adaptive_quant:
+ * adjust macroblock QP based on variance (AC energy) of the MB.
+ * high variance = higher QP
+ * low variance = lower QP
+ * This generally increases SSIM and lowers PSNR.
+ * To save bits in B-frames, adaptive lambda is used instead of adaptive quantization.
+*****************************************************************************/
+void x264_adaptive_quant( x264_t *h, x264_mb_analysis_t *a )
+{
+ int qp = h->mb.i_qp;
+ int ac_energy = ac_energy_mb(h);
+ x264_cpu_restore(h->param.cpu);
+ float result = ac_energy;
+ float threshold;
+ /* In the case of adaptive AQ sensitivity, grab the value from the frame pre-process. Otherwise, calculate
+ the AQ sensitivity value for the current frame. */
+ if(h->param.analyse.f_aq_sensitivity == 0)
+ threshold = h->aq_threshold;
+ else threshold = powf(h->param.analyse.f_aq_sensitivity,4)/2;
+ /* Adjust the QP based on the AC energy of the macroblock. */
+ int qp_adj = -3.0 * h->param.analyse.f_aq_strength * log(result / threshold);
+ qp_adj = x264_clip3(qp_adj,-5*h->param.analyse.f_aq_strength,5*h->param.analyse.f_aq_strength);
+ int new_qp = x264_clip3(qp - qp_adj,h->param.rc.i_qp_min,h->param.rc.i_qp_max);
+ /* Change the lambda values. */
+ /* If the QP of this MB is within 1 of the previous MB, code the same QP as the previous MB, to lower the bit
+ cost of the qp_delta. */
+ if(abs(new_qp - h->mb.i_last_qp) == 1) new_qp = h->mb.i_last_qp;
+ a->i_lambda = i_qp0_cost_table[new_qp];
+ a->i_lambda2 = i_qp0_cost_table[new_qp];
+ //h->i_mod_qp = new_qp;
+ //h->i_mod_chroma_qp = i_chroma_qp_table[x264_clip3( new_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
+ /* Adaptive quantization only applies to I/P frames. Applying it to B-frames generally results in a lot of
+ unnecessary bits spent on delta_qp. Instead, the lambdas are changed.
+ FIXME: Choose whether to use adaptive lambda or adaptive quantization on a per-block basis?
+ FIXME: Choose which to use based on bit cost?
+ FIXME: Is this optimal? */
+ //if(h->sh.i_type != SLICE_TYPE_B )
+ {
+ /* Save the final QP and update the chroma QP. */
+ h->mb.i_qp = a->i_qp = new_qp;
+ h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
+ }
+}
+
+/*****************************************************************************
* x264_macroblock_analyse:
*****************************************************************************/
void x264_macroblock_analyse( x264_t *h )
@@ -2046,9 +2107,19 @@
x264_mb_analysis_t analysis;
int i_cost = COST_MAX;
int i;
+
+ h->mb.i_qp = x264_ratecontrol_qp( h );
+
+ if( h->param.analyse.b_aq )
+ x264_adaptive_quant( h, &analysis );
+ //else
+ //{
+ // h->i_mod_qp = h->mb.i_qp;
+ // h->i_mod_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
+ //}

/* init analysis */
- x264_mb_analyse_init( h, &analysis, x264_ratecontrol_qp( h ) );
+ x264_mb_analyse_init( h, &analysis, h->mb.i_qp );

/*--------------------------- Do the analysis ---------------------------*/
if( h->sh.i_type == SLICE_TYPE_I )
Index: x264.c
===================================================================
--- x264.c (revision 720)
+++ x264.c (working copy)
@@ -243,6 +243,14 @@
" - 2: enabled on all mode decisions\n", defaults->analyse.i_trellis );
H0( " --no-fast-pskip Disables early SKIP detection on P-frames\n" );
H0( " --no-dct-decimate Disables coefficient thresholding on P-frames\n" );
+ H0( " --aq-strength <float> Amount to adjust QP/lambda per MB [%.1f]\n"
+ " 0.0: no AQ\n"
+ " 0.7: medium AQ\n"
+ " 1.4: strong AQ\n", defaults->analyse.f_aq_strength );
+ H0( " --aq-sensitivity <float> \"Center\" of AQ curve. [%.1f]\n"
+ " 0: automatic sensitivity (recommended)\n"
+ " 5: almost all QPs are raised\n"
+ " 35: almost all QPs are lowered\n", defaults->analyse.f_aq_sensitivity );
H0( " --nr <integer> Noise reduction [%d]\n", defaults->analyse.i_noise_reduction );
H1( "\n" );
H1( " --deadzone-inter <int> Set the size of the inter luma quantization deadzone [%d]\n", defaults->analyse.i_luma_deadzone[0] );
@@ -406,6 +414,8 @@
{ "trellis", required_argument, NULL, 't' },
{ "no-fast-pskip", no_argument, NULL, 0 },
{ "no-dct-decimate", no_argument, NULL, 0 },
+ { "aq-strength", required_argument, NULL, 0 },
+ { "aq-sensitivity", required_argument, NULL, 0 },
{ "deadzone-inter", required_argument, NULL, '0' },
{ "deadzone-intra", required_argument, NULL, '0' },
{ "level", required_argument, NULL, 0 },
Index: common/common.c
===================================================================
--- common/common.c (revision 720)
+++ common/common.c (working copy)
@@ -123,6 +123,9 @@
param->analyse.i_chroma_qp_offset = 0;
param->analyse.b_fast_pskip = 1;
param->analyse.b_dct_decimate = 1;
+ param->analyse.b_aq = 0;
+ param->analyse.f_aq_strength = 0.0;
+ param->analyse.f_aq_sensitivity = 0;
param->analyse.i_luma_deadzone[0] = 21;
param->analyse.i_luma_deadzone[1] = 11;
param->analyse.b_psnr = 1;
@@ -455,6 +458,13 @@
p->analyse.b_fast_pskip = atobool(value);
OPT("dct-decimate")
p->analyse.b_dct_decimate = atobool(value);
+ OPT("aq-strength")
+ {
+ p->analyse.f_aq_strength = atof(value);
+ p->analyse.b_aq = (p->analyse.f_aq_strength > 0.0);
+ }
+ OPT("aq-sensitivity")
+ p->analyse.f_aq_sensitivity = atof(value);
OPT("deadzone-inter")
p->analyse.i_luma_deadzone[0] = atoi(value);
OPT("deadzone-intra")
@@ -939,6 +949,9 @@
s += sprintf( s, " zones" );
}

+ if( p->analyse.b_aq )
+ s += sprintf( s, " aq=1:%.1f:%.1f", p->analyse.f_aq_strength, p->analyse.f_aq_sensitivity );
+
return buf;
}

Index: common/common.h
===================================================================
--- common/common.h (revision 720)
+++ common/common.h (working copy)
@@ -232,6 +232,10 @@

struct x264_t
{
+ float aq_threshold;
+ //int i_mod_qp;
+ //int i_mod_chroma_qp;
+
/* encoder parameters */
x264_param_t param;

Index: x264.h
===================================================================
--- x264.h (revision 720)
+++ x264.h (working copy)
@@ -230,6 +230,9 @@
int i_trellis; /* trellis RD quantization */
int b_fast_pskip; /* early SKIP detection on P-frames */
int b_dct_decimate; /* transform coefficient thresholding on P-frames */
+ int b_aq; /* psy adaptive QP */
+ float f_aq_strength;
+ float f_aq_sensitivity;
int i_noise_reduction; /* adaptive pseudo-deadzone */

/* the deadzone size that will be used in luma quantization */
This should compile (I haven't tried it) and should have absolutely no potential for bugs/breakage. It just entirely disables the lambda-based AQ for the meantime, since its only on B-frames and not really a big deal.

[Dark Shikari]

New executable uploaded. Removing the lambda AQ didn't seem to have much of a negative or positive effect for now, but will fix most of the bugs with this version. I may add it back in later.

[Dark Shikari]

I have posted results in the OP of a test of test of the new AQ. Yes, that's right, roughly 21.5% bitrate-adjusted SSIM improvement. :eek:

[acrespo]

I didn't use version 0.4 but v0.42 crashes in Vista x64. When I execute I receive a message from windows that the pthreadGC2.dll is missing. I returned to version 0.3 and don't have problems.

[Dark Shikari]

I didn't use version 0.4 but v0.42 crashes in Vista x64. When I execute I receive a message from windows that the pthreadGC2.dll is missing. I returned to version 0.3 and don't have problems.That's because... you need pthreadGC2.dll?

You can get it here (http://mirror05.x264.nl/Dark/force.php?file=./pthreadGC2.dll).

[ToS_Maverick]

you somehow forgot to replace the link, it still points to 0.4
0.42 is there but not labeled as .exe:
http://mirror05.x264.nl/Dark/force.php?file=./x264_Experimental_AQ_0.42

[CruNcher]

@Dark Shikari
Great work, you really solved parts of the Banding Problem with this, im amazed by the results :)

Without Dark Shikaris Magic AQ
http://rapidshare.com/files/84197179/testseq-pearl-nodarkaq.mkv

With Dark Shikaris Magic AQ (HVS Quality is greatly improved)
http://rapidshare.com/files/84197485/testseq-pearl-darkaq.mkv

Get it now, it makes your "darkest" dreams come true ;D

i tested an older version i think of this tough (found the patch in your folder) with --aq-strength 0.9, bellow that it wouldn't look good enough and --aq-strength 1.0 coused a problem (gonna retest with the new patch) in this test cut :)

and yes im as crazy as you where with Vendeta 3 Mbit for 1080p ;)

Here is the Bug with --aq-strength 1.0 (also happens for --aq-strength 0.9 if --no-dct-decimate is used), excuse me if this problem allready has been encountered before i didn't read the whole thread yet (and maybe it's not even happening with the new patch anymore)
http://rapidshare.com/files/84200572/testseq-pearl-darkaq-bug.mkv (look @ the top left when the calendar pages are turned)

[burfadel]

This patch definately makes a better quality image, without sacrificing bitrate, at least in crf mode :) Its to the point where I can say it should be enabled by default, and with a strength of 1.0 (100 percent? :) ) by default only because it works so well, and unlike the old patch such a high AQ seems to be better. Except of course, for the bug that cruncher has mentioned which I haven't seen! Hopefully this is a real step towards including it in the svn!

[acrespo]

That's because... you need pthreadGC2.dll?

You can get it here (http://mirror05.x264.nl/Dark/force.php?file=./pthreadGC2.dll).

Why the new version needs this file? I notice that the file size decrease too. Is that because you remove this library inside the .EXE?

[Atak_Snajpera]

(look @ the top left when the calendar pages are turned)

Could you make a screenshot and mark spot because I can't see it.
BTW 3MBps for 1440x1080... You are crazy :)

Good advice use mediafire.com instead of Rapidshare.
(I've just reached the limit for user)

[Dark Shikari]

Why the new version needs this file? I notice that the file size decrease too. Is that because you remove this library inside the .EXE?I'm too lazy to fix it for now ;_;

Also, I fixed the link.

[Sharktooth]

nice one DS! really... :)

[ToS_Maverick]

Dark Shikari, could you implement the additional switches you suggested, in the next release? i'd really like to play around with a fully enabled AQ (also on B frames).

[Dark Shikari]

Dark Shikari, could you implement the additional switches you suggested, in the next release? i'd really like to play around with a fully enabled AQ (also on B frames).This one is fully enabled on B-frames :)

I completely took out all frame-specific code and lambda-based code for the meantime.

[DeathTheSheep]

Even so, with r681, it yields an average of 33.4% bitrate-adjusted SSIM increase on my anime test clips (as compared to r680) in conjunction with hadamard, me-prepass, and 681's subme7 tweak. (Baseline profile, relatively high quantizers too).

Hmm, I wonder what all of these quality patches have in common? Like common place of origin, etc? Hmmm, nope, nothing. :p

[Dark Shikari]

Even so, with r681, it yields an average of 33.4% bitrate-adjusted SSIM increase on my anime test clips:eek:

[Dark Shikari]

Here is the Bug with --aq-strength 1.0 (also happens for --aq-strength 0.9 if --no-dct-decimate is used), excuse me if this problem allready has been encountered before i didn't read the whole thread yet (and maybe it's not even happening with the new patch anymore)
http://rapidshare.com/files/84200572/testseq-pearl-darkaq-bug.mkv (look @ the top left when the calendar pages are turned)
Something is HORRIBLY screwed up with the quantizers in that frame.

What commandline are you using, and where can I get your source? There seems to be something like a complete AQ reversal (!?!?!) It almost looks like the AQ sensitivity for that frame is negative (?!!!) I'm guessing it has something to do with the dark/light areas and an overflow of some sort in the calculations.

[DeathTheSheep]

...and the more modest 17.1% without hadamard, me-prepass, and subme7 patch (r680).

(This "expectedly unexpected" development only serves to add fuel to the fire in favor of updating the old patches; for some reason, quality increase is almost doubled when all used together! :D).

[edit]By the way, I'm testing in constant quantizer mode only. Are these results at all expected?

[Sagekilla]

...and the more modest 17.1% without hadamard, me-prepass, and subme7 patch (r680).

(This "expectedly unexpected" development only serves to add fuel to the fire in favor of updating the old patches; for some reason, quality increase is almost doubled when all used together! :D).

[edit]By the way, I'm testing in constant quantizer mode only. Are these results at all expected?

That could have something to do with it... You should try a similar constant quality mode instead to see what kind of quality gains can be had.

[DeathTheSheep]

What do you mean? As in the qcomped "crf" instead of "qp"?

It is bitrate adjusted, after all (bitrate delta compared with SSIM delta over 3 qp range works too).

Also, it's fun to note that q29 aq strength 0.5 produces nearly identical filesize (on 2 of my anime test clips) as q30 without aq, but SSIM goes up from [av] 0.9714289 to [av] 0.9747350.

[bob0r]

@Dark Shikari
Great work, you really solved parts of the Banding Problem with this, im amazed by the results :)

Without Dark Shikaris Magic AQ
http://rapidshare.com/files/84197179/testseq-pearl-nodarkaq.mkv

With Dark Shikaris Magic AQ (HVS Quality is greatly improved)
http://rapidshare.com/files/84197485/testseq-pearl-darkaq.mkv

Get it now, it makes your "darkest" dreams come true ;D

...

Wow what a difference!!!
Truely stunning to see this must improvement... i have mirrored the files: http://files.x264.nl/cruncher/

[CruNcher]

Thanks bob0r for mirroring, but i might have bad news the new Patch does nothing on this scene, seems the adaptive --aq-sensitivity is failing and even if i set it manualy to 15 and strength to 0.9 or 1.0 nothing changes anymore as with the old patch :(

CMD is


Old patch (x264_aq-brdo.diff) (improves visual percepted quality of this scene massively)
x264-oldaq pearl.avs --bitrate 3000 --level 4.1 --min-keyint 1 -
-keyint 15 --aq-strength 0.9 --no-fast-pskip --bframes 0 --ref 1 --weightb --sub
me 1 --8x8dct --qpmin 15 --ipratio 1.1 --trellis 0 --deadzone-inter 11 --deadzon
e-intra 20 --nf --vbv-bufsize 14754 --vbv-maxrate 29400 --vbv-init 1.0 --me dia
--threads auto --no-chroma-me --thread-input --aud --progress --sar 16:9 -o pear
l-aq.mkv

Old patch Reveresed AQ Bug (x264_aq-brdo.diff) (still improves but shows a problem in one sequence)
x264-oldaq pearl.avs --bitrate 3000 --level 4.1 --min-keyint 1 -
-keyint 15 --aq-strength 1.0 --no-fast-pskip --bframes 0 --ref 1 --weightb --sub
me 1 --8x8dct --qpmin 15 --ipratio 1.1 --trellis 0 --deadzone-inter 11 --deadzon
e-intra 20 --nf --vbv-bufsize 14754 --vbv-maxrate 29400 --vbv-init 1.0 --me dia
--threads auto --no-chroma-me --thread-input --aud --progress --sar 16:9 -o pear
l-aq.mkv

New Patch (No reaction, also non with strength 0.9/1.0 and sensitvitiy 15)
x264-newaq pearl.avs --bitrate 3000 --level 4.1 --min-keyint 1 -
-keyint 15 --aq-strength 1.0 --no-fast-pskip --bframes 0 --ref 1 --weightb --sub
me 1 --8x8dct --qpmin 15 --ipratio 1.1 --trellis 0 --deadzone-inter 11 --deadzon
e-intra 20 --nf --vbv-bufsize 14754 --vbv-maxrate 29400 --vbv-init 1.0 --me dia
--threads auto --no-chroma-me --thread-input --aud --progress --sar 16:9 -o pear
l-aq.mkv

[Dark Shikari]

Thanks bob0r for mirroring, but i might have bad news the new Patch does nothing on this scene, seems the adaptive --aq-sensitivity is failing and even if i set it manualy to 15 and strength to 0.9 or 1.0 nothing changes anymore as with the old patch :(

CMD is


Old patch (x264_aq-brdo.diff)
x264-oldaq pearl.avs --bitrate 3000 --level 4.1 --min-keyint 1 -
-keyint 15 --aq-strength 0.9 --no-fast-pskip --bframes 0 --ref 1 --weightb --sub
me 1 --8x8dct --qpmin 15 --ipratio 1.1 --trellis 0 --deadzone-inter 11 --deadzon
e-intra 20 --nf --vbv-bufsize 14754 --vbv-maxrate 29400 --vbv-init 1.0 --me dia
--threads auto --no-chroma-me --thread-input --aud --progress --sar 16:9 -o pear
l-aq.mkv

Old patch Reveresed AQ Bug (x264_aq-brdo.diff)
x264-oldaq pearl.avs --bitrate 3000 --level 4.1 --min-keyint 1 -
-keyint 15 --aq-strength 1.0 --no-fast-pskip --bframes 0 --ref 1 --weightb --sub
me 1 --8x8dct --qpmin 15 --ipratio 1.1 --trellis 0 --deadzone-inter 11 --deadzon
e-intra 20 --nf --vbv-bufsize 14754 --vbv-maxrate 29400 --vbv-init 1.0 --me dia
--threads auto --no-chroma-me --thread-input --aud --progress --sar 16:9 -o pear
l-aq.mkv

New Patch (No reaction, also non with strength 0.9/1.0 and sensitvitiy 15)
x264-newaq pearl.avs --bitrate 3000 --level 4.1 --min-keyint 1 -
-keyint 15 --aq-strength 1.0 --no-fast-pskip --bframes 0 --ref 1 --weightb --sub
me 1 --8x8dct --qpmin 15 --ipratio 1.1 --trellis 0 --deadzone-inter 11 --deadzon
e-intra 20 --nf --vbv-bufsize 14754 --vbv-maxrate 29400 --vbv-init 1.0 --me dia
--threads auto --no-chroma-me --thread-input --aud --progress --sar 16:9 -o pear
l-aq.mkv
Can you get me the source for this so I can find and fix the bug?

[Sharktooth]

bobor, i hope you've messed up the file names on your mirror, coz the nodarkaq looks better and less blocky than the darkaq sequence...

[Dark Shikari]

bobor, i hope you've messed up the file names on your mirror, coz the nodarkaq looks better and less blocky than the darkaq sequence...That's probably due to the bug in my AQ that seems to show its ugly face on certain parts of this clip.

[Sharktooth]

no, i saw blockiness in bright areas.

[DeathTheSheep]

Funny I noticed similar... but at my bitrates, the increase in quality of everything else more than makes up for it.

[Dark Shikari]

no, i saw blockiness in bright areas.Uh, that's part of the bug. My AQ has absolutely zero, zip, zilch to do with "brightness" or "darkness."

[Sharktooth]

but the blockiness was accentuated from the nodarkaq encode... so IMHO you should watch out from stealing bits from those areas unless you want them to look worse than without AQ.

[Dark Shikari]

but the blockiness was accentuated from the nodarkaq encode... so IMHO you should watch out from stealing bits from those areas unless you want them to look worse than without AQ.The entire bug is that AQ is reversed in certain frames. This means its INCREASING blockiness in blocky areas--doing the opposite of what it should. How much do I have to explain this? Look at the quantizer distribution yourself--its obviously completely reversed.

So far, I have been unable to replicate the bug using his encoded source as my source, with a CRF pass.

[CruNcher]

The percived quality improvement here is in the really dark scenes but even with standard VMR9 and calibrated you can see those problems without Dark Shikaris AQ (especialy LCD). They even become visible in the gradient in the non Dark (Green/Blue) Background with the text (The Rise and fall of our Empire is at stake) you can clearly see how Darks AQ improves this Background Gradient drasticly :). Sure you wont see the stuff in the Dark area before that if you turn down the Brightness a little, but in this Area you would still see this Gradient Problems even if you lower your Brightness. And for me Personaly that (Banding) is more anoying then the little more blocks it introduces mostly in the Fast Action :D, never forget we working lossy here we have to set visual priorities and my is clear in that case ;)

[DeathTheSheep]

In my case, it seems there were flaws in the original mpeg2, which I misconstrued as problems with the AQ. No worries for me...

What was the exact binary, avs script, decoder, and commandline used to produce the bug? I want to replicate this thing--I can't believe it would favor one system over another.

[CruNcher]

Indeed the source is bad one of the first Blu-Ray Mpeg-2 encodes (Full of Filmgrain mixed with Encoder Quantization Noise, and Banding couseing problems when cleaning it), so perfect to test extreme cases ;). Ok found a problem useing --sar 16:9 makes this hardware incompatible :( --sar 4:3 works but it's huge now im sure it wasn't intended to be displayed that way, jesus i never gonna understand AR as it's AR is 2.40 it's wrong i think that it's displayed now with --sar 4:3 @ AR 16/9 how crazy is that ;).
Ok so no go for staying hardware compatible and also in the right AR (2.40) onscreen itself, you must include the letterbox i see no other way or im blind ;)

Ok here another one with the old AQ, this time it should be almost 100% Hardware Decoding Compatible (and it's low complexity) :)
http://mirror05.x264.nl/CruNcher/new-olddarkaq-hdc.mkv

Another thing i encountered now is that with VM9 Windowed (at least on my Nvidia 8800 GT) i see every mini block specialy now the problems in the Letterbox area (grey/white flashing because brightness is much higher now seems the old TV Level thing most probably).
With VMR9 Renderless it looks much better (because it's darker and so alot of problems aren't visible (Luma HVS tricks) on the first sight and even hard to spot on the second). Darks AQ improves the rest mostly the visible banding in the bright fire flash gradient, then the background gradient of the Text scene and also in the dark blue gradient background outside :) I also left out --no-fast-pskip, this seems to have overall improved it a little. Viewing this from aprox 75cm-1m away looks really nice :). Only thing im not happy with is the color representation compared to the source it seems wrong much duller it's strange it seems to be no Colormatrix problem as the uncompressed sample FFV1 shows all the color fidelity. As expected it's the way Cyberlinks Decoder does it under VMR9 and even Overlay Mixer the only Decoder doing it right from the start is CoreAVC it shows imidiatly the Full Color Representation as the Source (and by doing that it even gets rid of more of the visual flaws that you would see with other decoders even Mplayer and Videolan show the same dull colors :(

[Dark Shikari]

Same settings as you cruncher--and absolutely no problem like I encountered in your stream. QPs came out exactly as I expected:

http://i18.tinypic.com/8dvvvqq.png

Also, I vastly optimized the AQ, removing the need for duplicate cache_loads and SADs and SSDs. The main result is that the CPU cost of AQ is cut drastically.

[CruNcher]

Dark did you used the old AQ patch or the new one ? the new one didn't worked @ all for me and the old one showed that bug @ --aq-strength 1.0, strange this is.
This here is the one i mean http://files.x264.nl/force.php?file=./dark/x264_aq-brdo.diff that one i patched on 720 and got those --aq-strength 1.0 results.
gcc.exe (GCC) 3.4.5 (mingw special) <--- now im scared i better try a newer gcc version asap

[Dark Shikari]

Dark did you used the old AQ patch or the new one ? the new one didn't worked @ all for me and the old one showed that bug @ --aq-strength 1.0, strange this is.
This here is the one i mean http://files.x264.nl/force.php?file=./dark/x264_aq-brdo.diff that one i patched on 720 and got those --aq-strength 1.0 results.That's an ancient and bugged AQ patch.

Here, I'll give you a little something that will make you all warm and fuzzy inside: a combined patch of RDRC and AQ... AQ 0.43, not 0.42! Nice and fast... until you turn on RCRD ;)

Patch (http://pastebin.com/f227baccb)

Build (http://www.mediafire.com/?bcd2dd5ygtj)

[CruNcher]

Nice and where is Lookahead ? a complete RDRC but no Lookahead for ABR geez ;)

[Sharktooth]

could you encode the clip again with this one? just AQ, no RDRC...

[Dark Shikari]

could you encode the clip again with this one? just AQ, no RDRC...That's what I just did--just AQ, not RCRD.

RCRD is unrelated and I'm not using it for most of the AQ work; its just something some people were requesting.

[CruNcher]

Dark ehh i tried your new build (not patched it your .exe) but --aq-strength 1.0 does nothing compared to the old (that you call buggy AQ) in the key spots im talked about it just ignores them and when doing --aq-strength 1.0 --aq-sensitivity 15 your .exe is crashing the encode right after the 1 frame :(

[Dark Shikari]

Dark ehh i tried your new build (not patched it your .exe) but --aq-strength 1.0 does nothing compared to the old (that you call buggy AQ) in the key spots im talked about it just ignores them and when doing --aq-strength 1.0 --aq-sensitivity 15 your .exe is crashing the encode right after the 1 frame :(Well it works just fine here... :rolleyes: same source, same commandlines...

Can you come on Freenode and we'll try to resolve this?

[mahsah]

Wow, this is great! I tried this on a test clip of Futurama, and while the results are not very noticable (and I was doing two pass encoding), there are fewer blocks with AQ on. Also the png filesize is lower, which I guess means it is more compressable.

NoAQ:
http://img47.imageshack.us/my.php?image=noaqtq1.png

AQ:
http://img337.imageshack.us/img337/5960/aqgx4.png

command line:

C:\Program Files\megui\tools\x264\x264.exe" --pass 2 --bitrate 1350 --stats "hfyu_go.stats" --ref 8 --mixed-refs --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,1 --subme 6 --trellis 1 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 12 --threads auto --thread-input --sar 243:200 --progress --no-psnr --no-ssim --output "hfyu_go.mkv" "hfyu_go.avs"

[CruNcher]

Dark wich commandline i pasted 3 :)

so ok i did a test again with the old aq and your new aq both same settings --aq-streangth 0.9 this time as 1.0 is extreme buggy as you said no --aq-sensitivity was used for both (not correct the old one has --aq-sensitivity 15 as default) and here is now a visual comparsion analysis of they key spots i talked about and their subjectivly percepted visual differences.



Not many guys for sure have such trained eyes as me and maybe that's the problem why i realize such scenes so fast and they really hurt me (please note it's not about the individual frames here it's about the scenes they mark and the time they stay in focus and are perceptable to the eye) :(

New AQ
http://s3.directupload.net/images/080117/llnhtezh.png
http://s1.directupload.net/images/080117/tu7hg58j.png
http://s2.directupload.net/images/080117/gueurani.png
http://s1.directupload.net/images/080117/ft79axio.png

Old AQ
http://s1.directupload.net/images/080117/gh3325fq.png
http://s2.directupload.net/images/080117/2dp7ut4p.png
http://s5.directupload.net/images/080117/dvit3x7j.png
http://s5.directupload.net/images/080117/hgc6ar9j.png

These are the Key Areas for me that decide what is good and what is bad and how the whole sequence is rated by the viewer more rated by me as Sharktooth for example doesn't like the blocks in the luminant areas but those are so fast anyway this really hurts my eyes about your new AQ compared to the old :)
I have overexergerated the Decoding brightness Problems (useing VMR9 Windowed and ffdshow is the best combination to reach that hehe) to show those Key areas this is ofcourse not the same what you would see with a good calibration on screen and the best to rate this on Windows seems CoreAVC (Perfect Rendering Quality right color levels right brigthness levels everything perfect) :)

These Key Spots have to be seen in Motion (especialy the color gradients) and then it's clear that the new AQ is not for everyone :)

[Dark Shikari]

It'd be far more useful if you showed me what you thought was a "good" quantizer distribution and what you thought was a "bad" one.

I can't tell crap from those images without seeing the quantizers.

[Sagekilla]

Dark Shikari, does your latest linked build feature your RCRD as well? I downloaded what I thought had this feature and I found it produced bit-for-bit identical encodes as the current r720 build, with the exception of r720 being marginally faster (6.88 vs 6.47 fps).

Nevermind, I forgot to enable the switches to begin with. Are there any suggestions for --rcrd-lambda, like a starting point for example? :)

[Dark Shikari]

For lambda, note that lambda does not correspond to a specific CRF--the bitrate result of lambda will vary based on your window size, source, and AQ settings.

As I mentioned in the OP, lambda 500 will give you something roughly similar to CRF 24-26.

[CruNcher]

@Dark
Sorry gonna upload the results from both showing the quantizers and some results showing both rendered by CoreAVC in the key spots (also interesting) :)


This is the nices Representation of the Encode (wich comes closest to the source in color fidelity and brightness)

CoreAVC New AQ
http://s3.directupload.net/images/080117/g7vcy5ab.png
http://s3.directupload.net/images/080117/jtx66peq.png
http://s3.directupload.net/images/080117/o3jlae5c.png

CoreAVC Old AQ
http://s5.directupload.net/images/080117/wz44kip6.png
http://s3.directupload.net/images/080117/y73468f4.png
http://s4.directupload.net/images/080117/pgqe79ch.png

Even with Hardware Decoding it's not possible to reach this Quality at least not with MPCHC + Cyberlink and Nvidia on VMR9 or Overlay the colors will allways be duller then the actuall source was :(

[Sagekilla]

@CruNcher: Sounds like you're not properly adjusted for PC levels.. If you're not, then video will always look dull and washed out (black will be gray, whites won't be as white, etc) I forgot which setting, but it involves YUV conversion in hardware. Don't do that, that's what causes it to be set to TV scale instead of PC scale.


On a side note, is there any possibility of speed optimizations for RCRD or is it inherently that slow?

[Dark Shikari]

On a side note, is there any possibility of speed optimizations for RCRD or is it inherently that slow?I have already made a number of optimizations, but more are possible.

1) Take more shortcuts when doing the lookahead encodes.

2) Take more shortcuts when choosing the QP.

3) The full effect of RDRC can be simulated by analyzing the video on the first pass to figure out, for every case in which data is added to the video stream, how much that data is used in all future frames. This is an extremely difficult problem, but if we could somehow get an answer to that question, we would have an RC method that would get results close to RDRC and absolutely blow away all other video codecs out there. I already have some theories on how to do this, but I'm not sure how good an approximation they would be.

[Sagekilla]

Interesting work.. Any chance of any portion of this being added to the RC for crf mode? Or is this something that'll only work for 2-pass?

[Dark Shikari]

Interesting work.. Any chance of any portion of this being added to the RC for crf mode? Or is this something that'll only work for 2-pass?The speed cost is extremely heavy; the only thing I could think of for general use would be to use it to make I-frame/scenecut quantizer decisions or something.

It doesn't need to be twopass--that's just to make the coding easier (to avoid having to decide frametypes).

[CruNcher]

Here dark as you can see in the gradient areas the Quantization is in the new AQ to high or uneven distributed

New AQ Quantization Distribution
http://s4.directupload.net/images/080117/qiil5rwn.png
http://s6.directupload.net/images/080117/92elpdmw.png
http://s4.directupload.net/images/080117/dj435r2x.png
http://s3.directupload.net/images/080117/5xdn8u82.png

Old AQ Quantization Distribution
http://s1.directupload.net/images/080117/kgtd8olt.png
http://s3.directupload.net/images/080117/p86p3bdv.png
http://s3.directupload.net/images/080117/st6beg4c.png
http://s3.directupload.net/images/080117/9ar89obb.png

Maybe a Quantization of 15-20 (oldaq is most of the time really low like 15-17 over several frames) will help in such areas :) 24-28 seems to be to much and then it seems the result is very visible banding.

Here are also both clips for you to analyze
http://mirror05.x264.nl/CruNcher/pearl-aq-newdark.mkv
http://mirror05.x264.nl/CruNcher/new-olddarkaq-hdc.mkv

[Dark Shikari]

The explanation there is that the new AQ (when on automatic sensitivity) tries to avoid redistributing bits among frames, while older AQs didn't.

I would suggest you try a higher AQ strength, also, as the new AQ uses a different formula.

[Atak_Snajpera]

@CruNcher

...and don't forget to use RGB32 instead of YV12 !

[CruNcher]

@Sagekilla and Atak
I used the high quality YV12->RGB32 conversion but enableing/disableing doesn't anything to the picture quality with fffdshow and vmr9 windowed/renderless.
Don't know tough what Sagekilla means with the YUV conversion in hardware, never saw such an option in MPCHC Cyberlink or the VMR9 renderer.
I think the problem here arises once again from the Nvidia driver and somehow the DVI output is set on TV Level their is a registry fix for that, gonna try if this improves anything :).
It's strange that CoreAVC does compensate this somehow i think it's more then just that(also somehting with color telemetry and the decoder).
Saw a Bt709 option now in ffdshow that's something new and also the Range setting but both didn't improved anything on the visible picture quality.

@Dark Shikari
i try a higer Strength now thx for that info :)

[Atak_Snajpera]

@CruNcher
In MPC use System Default instead of vmr9

Source: HDV camcorder
Settings: PS3 profile 2-pass 6144 kbps.

AQ 1.0
http://img508.imageshack.us/img508/1593/aq1ov8.th.png (http://img508.imageshack.us/my.php?image=aq1ov8.png)


NOAQ
http://img502.imageshack.us/img502/5281/noaqca4.th.png (http://img502.imageshack.us/my.php?image=noaqca4.png)

Beautiful job Dark!

[Dark Shikari]

An extreme example. Same bitrate (3 megabit), same encoding settings, almost the same framesize.

Except the AQ one looks so much better its astounding (second is AQ):

http://i6.tinypic.com/8aksxz9.pnghttp://i17.tinypic.com/6tmoyty.png

Swap between them fast to see a huge difference.

[CruNcher]

i don't need to swap between you can allready see from the first sight less ringing :) that's amazing :D

New AQ
- less ringing
- less banding
- more details

:D that's 3 for one and with very little speed loss (and only smal complexity disadvantage) :)

yeah dark --aq-strength 1.5 enhanced it i go higher now :)

final ratefactor: 24.74 <- before that where somewhere @ 30 :D

but tough i still don't reach that optimum visual quality i reached with the old one @ 0.9
hmm and the higher i go with the new one the more the bitrate decreases and the file gets smaller in the end but still those spots are not enhanced jesus i try a strength of 5.0 now ;)
strength of 5.0 is extreme it shows all the faces basicly smashed up in blocks but the background seems to get ignored completly at least it doesn't enhance the way it did with the old AQ
i think a little more balance to the new AQ and it could be perfect for all situations you just have to find that :), but it seemes easy now comparing the Old AQ with the new AQ and the spots they actually enhance/disenhance mixing those both and voila = Super AQ :D

@Dark
here is the visual result with strength 5.0
http://mirror05.x264.nl/CruNcher/pearl-newdark-5.0.mkv

i didn't found the setting that results in the same visual quality for the whole scene yet as the oldaq did (still searching)
http://forum.doom9.org/showthread.php?p=1088356#post1088356

-aq-strength 2.0 seems to come near the visual result of the old but it's not quiet the same but i think i get closer (at least it gets harder to spot the difference)
maybe i should take the final size as the indicator for it :)

Old AQ --aq-strength 0.9 --aq-sensitivity 15 = 11.273.040 Bytes
New AQ --aq-strength 2.0 = 10.643.143 Bytes

jep it seems to be somewhere bellow the old AQ i come nearer :)
i reached now the same filesize @ --aq-strength 0.3 but the visual result is suboptimal for this scene compared with the oldAQ @ --aq-strenth 0.9 and --aq-sensitivity 15

[desta]

I'm a bit lost now - the higher the strength of the new AQ, the less overall bitrate it needs?!

[CruNcher]

jep that's how it looks to me, im almost sure the problem lies somewhere in the adaptive --aq-sensitivity in the new AQ, because that's also what is crashing the encoding process if you try to set it manualy.

here is the nearest i could get with the new AQ to the Old AQ visualy wise for this testcut (with just adjusting --aq-strength for both)

New AQ (--aq-strength 0.3) SSIM Mean Y:0.9870448 PSNR Mean Y:47.322 U:48.604 V:51.085 Avg:47.950 Global:47.431 final ratefactor: 25.26
http://mirror05.x264.nl/CruNcher/pearl-aq-newdark-0.3.mkv

Old AQ (--aq-strength 0.9) SSIM Mean Y:0.9838171 PSNR Mean Y:45.361 U:47.704 V:49.915 Avg:46.192 Global:45.801 final ratefactor: 30.14
http://mirror05.x264.nl/CruNcher/pearl-aq-olddark-0.9.mkv

Now it's up to the viewer to decide wich gives better results for this scene


I don't giva a shit about SSIM and PSNR for years now doing my visual optimization stuff keep that in mind (because SSIM isn't realizing this visual problems @ all) (im more an Artist then a Mathematician) (im absolutely thriled by HVS optimization and tricking the eye)

[Dark Shikari]

I'm a bit lost now - the higher the strength of the new AQ, the less overall bitrate it needs?!Generally, yes, because it raises quantizers on high-detail areas that don't need as many bits.

[bob0r]

@CruNcher
You deleted all your test files?
As in the last two links above?

[CruNcher]

No their are up they just werent up before i wrote it yes the --aq-strength 5.0 i deleted i wasn't sure if Dark really needs it as he can reproduce it but maybe it's interesting for the rest to watch it that's why i uploaded it again ;)

[Dark Shikari]

No their are up they just werent up before i wrote it yes the --aq-strength 5.0 i deleted i wasn't sure if Dark really needs it as he can reproduce it but maybe it's interesting for the rest to watch it that's why i uploaded it again ;)Strength 5.0 is retarded since it allows QP adjustments of up to 25 in each direction :p

[CruNcher]

Dark if i see those Metric results did you really blindly optimized for SSIM ?
I think above is the best example to showof where such Optimizations end and where you should better go the HVS (Psy Optimization way) the Ateme guys really understood that fast after i did my first beta tests of their encoder back then and really did some great HVS research since then (in the whole area Picture Sharpness, Perfect Masking and some more clever things) :)

[Dark Shikari]

Dark if i see those Metric results did you really blindly optimized for SSIM ?The reason SSIM rises is simple: SSIM is directly related to the variance of a block. One of its main advantages over PSNR is that it measures distortion relative to the detail already present in a block--which makes perfect sense visually.

My AQ has the exact same approach, and for the same reasons; X amount of lossiness in a low detail block looks far worse than X amount of lossiness in a high detail block. Therefore, low-detail blocks should have much less lossiness than high-detail blocks.

I actually learned that SSIM worked this way slightly after I wrote my first version of this AQ, but the results are not at all surprising given what the AQ does. It doesn't specifically optimize for SSIM, but by its very nature it should increase SSIM.

[desta]

Generally, yes, because it raises quantizers on high-detail areas that don't need as many bits.
So to preserve detail evenly, it's best to use moderate strength and/or lower sensitivity?

[Dark Shikari]

So to preserve detail evenly, it's best to use moderate strength and/or lower sensitivity?Sensitivity should be kept at automatic whenever possible.

Obviously too high strength will ruin quality.

The trick is that with no AQ, detail is kept far better in high-detail areas than in low detail. This is because of how quantization works. Let's say we have two blocks, each of which consist of a single frequency (for simplicity):

Block 1 frequency: 102.9

Block 2 frequency: 7.4

Let's say our quantizer allows values 0, 5, 10, etc.

Block 1 will be rounded to 105, resulting in an error of about 2%. Block 2 will be rounded to 5, resulting in an error of about 30%. Yet they both used the same quantizer!

This AQ tries to resolve this by forcing Block 1 to use a higher quantizer and Block 2 to use a lower quantizer.

[desta]

Right, yeah I understand. Thanks for the explanation.

It threw me a bit when CruNcher showed the stronger AQ using less bits, but I see now it's pretty much the same principle as your original AQ - just that pushing the AQ strength too much will tip the results in the opposite direction.

[CruNcher]

Might be Dark but that doesn't change the fact that your new AQ Visualy isn't @ the optimum as the Old one wich renders especialy the very problematic background edges that they eye imidiatly realizes (in motion even more) better.
This is even clearly viewable with a correct calibrated color and brightness representation as how CoreAVC shows it and with a bad calibration (like for sure many Windows users have them) it even looks more worse with your new aproach compared to your old aproach, you really should think about it.
Im not sure if more details are worth it to have such scenes apearing that way to the eye and the viewer get distracted and the illusion of a very clean source is gone in that moment (talking especialy for very low bitrates here).

[Dark Shikari]

Might be Dark but that doesn't change the fact that your new AQ Visualy isn't @ the optimum as the Old oneI've found it to be much better, personally; but perhaps you're looking for an AQ with a stronger QP-lowering effect at very low variances? This would act more like Haali's AQ, and have a much stronger effect on dark backgrounds.

[CruNcher]

Dark im now conducting some more improvements stuff on this source project im gonna todo 2 encodes of the complete 3 hour with your AQ and the old one @ the given settings and then do a subjective compare of most of the scenes and tell you the final results, first im doing a extreme cut of what you just saw with many different scenes put together (also used that to evaluate Atemes quality) and show you those results with both AQs and my Visual understanding of this (the result will be the Encode that has the lowest amount of scenes where you could realize this is a lossy encode wins) :) later im also doing that with a almost complete dark Movie and see how good each of those AQs does in each situation, definately will take some time.

[DeathTheSheep]

This RCRD is revolutionary... (and slow, of course, but the slower the revolution, the more exquisite the results).

Looks like it's ignoring/washing out complex detail that's only shown for a split second, keeping all the goodies that persist in the following frames. This reminds me of some beta vp6 builds of yore.

[Dark Shikari]

Dark im now conducting some more improvements stuff on this source project im gonna todo 2 encodes of the complete 3 hour with your AQ and the old one @ the given settings and then do a subjective compare of most of the scenes and tell you the final results, first im doing a extreme cut of what you just saw with many different scenes put together (also used that to evaluate Atemes quality) and show you those results with both AQs and my Visual understanding of this :)How about you encode using a source that isn't atrocious?

And try using a sane bitrate? :p

Encoding with unrealistic settings is not a way to test an AQ.

[DeathTheSheep]

Haha, maybe unrealistic settings on an atrocious source brings out the best in an unrealistic, atrocious AQ? :)

[CruNcher]

Dark you know the EBU (Broadcast) test sequence ? my stuff isn't far away from that testcut just that i completly use hollywood (film) stuff.
And we all know that X264 still has problems just run ParkRun throug it with low bitrate and you see there could be alot still done in Psy Optimizations for X264

[Dark Shikari]

Dark you know the EBU (Broadcast) test sequence ? my stuff isn't far away from that testcut just that i completly use hollywood (film) stuff.
And we all know that X264 still has problems just run ParkRun throug it with low bitrate and you see there could be alot still done in Psy Optimizations for X264Given what my AQ does, I suspect it would give extremely good results for ParkRun...

[DeathTheSheep]

Are there any plans for improving the quality algorithm further? You mentioned lambda support and such being removed "for the time being" since they "screw up" B frames on high quantizers or whatnot, but are there plans to pump the algo to the next level soon enough?

[Dark Shikari]

Are there any plans for improving the quality algorithm further? You mentioned lambda support and such being removed "for the time being" since they "screw up" B frames on high quantizers or whatnot, but are there plans to pump the algo to the next level soon enough?At this point the only real possible benefits are:

1) Adding the ability to drop QP even further in extremely flat areas, in addition to the current method.

2) Trellising the QP_deltas to save bits.

3) Lambda modification to save bits (will be a hell of an annoyance to do, and I'm not sure how good an idea it is). Lambda modification is better in Xvid or similar, where you can't move quantizers around as effectively.

At this point, for the most part, I think this AQ is ready for primetime; I have yet to see it have a negative effect on anything but a cartoon source at low bitrates. The positive effects are of course huge.

[DeathTheSheep]

Mm, good stuff. I'm curious as to the origin of this statement: "Its not particularly good at cartoons; I wouldn't use it on anime/cartoons," especially in light of its groundbreaking success on my anime test clips at high QP, and whether the improvements you mention have the potential to remedy what problems may remain in this regard.

[Danisan]

Mm, good stuff. I'm curious as to the origin of this statement: "Its not particularly good at cartoons; I wouldn't use it on anime/cartoons," especially in light of its groundbreaking success on my anime test clips at high QP, and whether the improvements you mention have the potential to remedy what problems may remain in this regard.

Do you have any comparison pictures for the anime tests you've made? :thanks:

[CruNcher]

Dark whats about the --aq-sensitivity crash in your latest build ?

[Dark Shikari]

Dark whats about the --aq-sensitivity crash in your latest build ?What crash? If someone reported a bug, I missed it...

[ditche]

Yep, I have a crash with v0.42, there's no encoding...

-[Information] Log for job1 (video, video test.avs -> video test new aq.mp4)
--[Information] [17/01/2008 20:26:02] Started handling job
--[Information] [17/01/2008 20:26:02] Preprocessing
--[NoImage] Job commandline: "C:\Program Files\megui\x264.exe" --qp 23 --ref 10 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --bime --weightb --direct auto --filter -2,-1 --trellis 2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads 3 --thread-input --sar 1:1 --progress --no-dct-decimate --no-psnr --no-ssim --output "D:\VIDEO\rip\video test new aq.mp4" "D:\VIDEO\rip\video test.avs" --aq-strength 0.6
--[Information] [17/01/2008 20:26:02] Encoding started
--[NoImage] Standard output stream
--[NoImage] Standard error stream
--[Information] [17/01/2008 20:26:02] Job completed

:helpful::)

[Dark Shikari]

Found one crash bug. Note to self: you cannot say if(floating point value == 0) before calling x264_cpu_restore(). Updated builds/patch.

[CruNcher]

Wow this really almost fliped me out of the chair :D

No AQ
http://mirror05.x264.nl/CruNcher/parkrun-noaq.mkv

New AQ
http://mirror05.x264.nl/CruNcher/parkrun-newaq-1.0.mkv

Old AQ
http://mirror05.x264.nl/CruNcher/parkrun-oldaq-0.9.mkv
http://mirror05.x264.nl/CruNcher/parkrun-oldaq-1.0.mkv

[Dark Shikari]

Wow this really almost fliped me out of the chair :D

No AQ
http://mirror05.x264.nl/CruNcher/parkrun-noaq.mkv

New AQ
http://mirror05.x264.nl/CruNcher/parkrun-newaq-1.0.mkv

Old AQ
http://mirror05.x264.nl/CruNcher/parkrun-oldaq-0.9.mkv
http://mirror05.x264.nl/CruNcher/parkrun-oldaq-1.0.mkvHoly crap! I thought it was just a ratecontrol issue until I realized the I-frames were the same size between the three encodes... holy shit!

The difference in some of the later I-frames between those is unbelievable.

[Dark Shikari]

>>>WOW<<<

http://i8.tinypic.com/6pocmcy.png

http://i2.tinypic.com/87k6ipf.png

:eek::eek::eek::eek::eek::eek::eek::eek:

I didn't realize any AQ could be this effective. Wow.

(both I-frames, same size)

[Romario]

Dark Shikari,what about Vista users, especially Vista 64 users?

Do you have a plan to compile 64-bit build?

[Dark Shikari]

Dark Shikari,what about Vista users, especially Vista 64 users?

Do you have a plan to compile 64-bit build?I don't have a 64-bit OS, so I can't make a 64-bit build--but the patch is available for someone else to.

Also, here's an animated GIF of the difference, since its so shocking:

http://i18.tinypic.com/82u8c9j.gif

[CruNcher]

just the pulseing from the closed gop that needs to be workedaround visualy, then it would be perfect :)

[Snowknight26]

The I-frames really detract from the visual asthetics when watching those samples.
Pretty impressive nonetheless.

[Dark Shikari]

The I-frames really detract from the visual asthetics when watching those samples.The main problem is that Cruncher used 1-pass ABR. The second problem being his crappy settings :p

[Razorholt]

@Darky: what build did you use?

[Dark Shikari]

Apparently (and not surprisingly) people want to see comparisons of P-frames in properly encoded clips, rather than Cruncher's subme1 atrocities, so I will post a real comparison in a bit.

[CruNcher]

i know Dark but it's realtime that way on my 2.8 GHZ Dualcore machine :D sure with subme 5 this would look much better and useing b-frames and more ref and all the stuff but complexity would go up and speed down and that's not the way i balance stuff my goal is also another one then best compression existing ;)

[DeathTheSheep]

Is 1.0 now the recommended strength? Everyone seems to scream 1.0 is the best, but I see no evidence. Did you do your tests with 1.0, DS?

[Dark Shikari]

Is 1.0 now the recommended strength? Everyone seems to scream 1.0 is the best, but I see no evidence. Did you do your tests with 1.0, DS?Yeah, I've been using 1.0.

1.0 = max QP adjustment of +/- 5

[Atak_Snajpera]

PS3 profile 2-pass 6144 kbps

frame 370
no AQ
http://img174.imageshack.us/img174/5076/noaquq8.th.jpg (http://img174.imageshack.us/my.php?image=noaquq8.jpg)

AQ1.0
http://img156.imageshack.us/img156/5189/aq1yu6.th.jpg (http://img156.imageshack.us/my.php?image=aq1yu6.jpg)

[Dark Shikari]

:eek:

I just finished my own test encodes, 5 megabits at 25 frames per second, maxed out settings.

AQ got a 37% SSIM boost.

The quality difference is mindblowing--the AQ, even in motion, looks nearly transparent, while the non-AQ looks atrocious.

Linkage to download both clips (AQ, no AQ) (http://www.mediafire.com/?62dzmttn0n4).

[Inventive Software]

And the award for next SVN entry goes to:


Dark Shikari! :D

[DeathTheSheep]

You know, I'd like a link to the clips (preferably the anime) you claim didn't benefit visually from the AQ.
I want to test if any settings combo can give a better effect, then see if I can generalize these to other problem anime samples.

[Dark Shikari]

You know, I'd like a link to the clips (preferably the anime) you claim didn't benefit visually from the AQ.
I want to test if any settings combo can give a better effect, then see if I can generalize these to other problem anime samples.I haven't tested a lot on cartoons--once I fix interlacing to work correctly with AQ, I'll go back to them.

[CruNcher]

I haven't tested a lot on cartoons--once I fix interlacing to work correctly with AQ, I'll go back to them.

Arghh don't deoptimize X264 for Anime stuff at least don't unbalance the Real Footage behaveiour that would be a disaster.

This i-Frame pulseing should be looked into even @ 6 Mbits it's visible and even with heavy encoding settings (but i think that problem is very deep inside X264 since the first days and has todo with the partitioning itself won't be easy to fix it for sure (make it more consistent), hmm playing arround with the deadzone reduces it a little but it's still their)

[Dark Shikari]

Interlacing's stream corruption appears to not be a bug in AQ bug actually a long-standing bug in x264--the CABAC context for interlaced QP_deltas seems to be wrong in interlaced mode, since it only appears with CABAC, and appears even if I entirely remove AQ and do nothing but randomely apply quantizers to different blocks.

[DeathTheSheep]

Crunch: Nobody's de-optimizing anything.

DS: If interlacing is causing you trouble, don't use it. It's more visually difficult to spot differences in screencaps anyway with the presence of two fields. Instead (just for testing purposes, if nothing else), deinterlace the source well first; it might even be advisable to discard one field entirely and interpolate the width (via spline36) to maintain the AR--this way, you'll have a great low-res, low-br test clip on your hands.

[Dark Shikari]

Crunch: Nobody's de-optimizing anything.

DS: If interlacing is causing you trouble, don't use it. It's more visually difficult to spot differences in screencaps anyway with the presence of two fields. Instead (just for testing purposes, if nothing else), deinterlace the source well first; it might even be advisable to discard one field entirely and interpolate the width (via spline36) to maintain the AR--this way, you'll have a great low-res, low-br test clip on your hands.I don't have this choice. Dakaz insists that interlacing and AQ work together :p

[DeathTheSheep]

Da...kaz...? :eek:

[Dark Shikari]

Da...kaz...? :eek:x264 is not only used by hobbyists, you know... :)

[bob0r]

I find these results stunning:

No AQ:
x264.exe --pass 2 --bitrate 3096 --threads auto --deblock 0:0 --bframes 3 --b-pyramid --bime --weightb --b-rdo --me umh --ref 5 --mixed-refs --subme 7 --trellis 1 --analyse all --8x8dct --no-fast-pskip --progress --fps=25 --output x264aq.mkv 720p50_parkrun_ter.yuv 1280x720
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 SSSE3 Cache64
x264 [info]: slice I:3 Avg QP:25.33 size:131303 PSNR Mean Y:41.13 U:42.91 V:45.56 Avg:41.78 Global:38.10
x264 [info]: slice P:233 Avg QP:34.52 size: 29549 PSNR Mean Y:29.02 U:36.95 V:39.19 Avg:30.50 Global:30.22
x264 [info]: slice B:268 Avg QP:35.70 size: 2964 PSNR Mean Y:29.10 U:37.13 V:39.35 Avg:30.56 Global:29.96
x264 [info]: mb I I16..4: 27.9% 66.2% 6.0%
x264 [info]: mb P I16..4: 0.3% 0.8% 0.3% P16..4: 46.4% 12.1% 9.8% 1.0% 0.7% skip:28.7%
x264 [info]: mb B I16..4: 0.0% 0.0% 0.0% B16..8: 17.4% 0.1% 0.4% direct: 3.1% skip:79.0%
x264 [info]: 8x8 transform intra:60.9% inter:56.9%
x264 [info]: ref P 80.2% 9.6% 6.0% 2.2% 2.0%
x264 [info]: ref B 87.2% 5.4% 3.9% 2.1% 1.3%
x264 [info]: SSIM Mean Y:0.8698542
x264 [info]: PSNR Mean Y:29.138 U:37.082 V:39.312 Avg:30.597 Global:30.099 kb/s:3203.68
encoded 504 frames, 9.37 fps, 3202.74 kb/s

AQ:
x264.exe --pass 2 --bitrate 3096 --threads auto --aq-strength 1.0 --deblock 0:0 --bframes 3 --b-pyramid --bime --weightb --b-rdo --me umh --ref 5 --mixed-refs --subme 7 --trellis 1 --analyse all --8x8dct --no-fast-pskip --progress --fps=25 --output x264aq.mkv 720p50_parkrun_ter.yuv 1280x720
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 SSSE3 Cache64
x264 [info]: slice I:3 Avg QP:24.67 size:138018 PSNR Mean Y:40.74 U:42.93 V:45.56 Avg:41.47 Global:38.03
x264 [info]: slice P:233 Avg QP:33.93 size: 30242 PSNR Mean Y:28.66 U:37.06 V:39.26 Avg:30.16 Global:29.88
x264 [info]: slice B:268 Avg QP:35.23 size: 2405 PSNR Mean Y:28.75 U:37.24 V:39.42 Avg:30.23 Global:29.63
x264 [info]: mb I I16..4: 26.9% 66.5% 6.5%
x264 [info]: mb P I16..4: 0.2% 0.9% 0.3% P16..4: 50.3% 15.7% 11.6% 0.7% 0.3% skip:20.1%
x264 [info]: mb B I16..4: 0.0% 0.0% 0.0% B16..8: 20.5% 0.1% 0.4% direct: 1.4% skip:77.6%
x264 [info]: 8x8 transform intra:66.3% inter:56.9%
x264 [info]: ref P 81.7% 8.8% 5.5% 2.1% 1.9%
x264 [info]: ref B 84.3% 7.3% 4.6% 2.3% 1.5%
x264 [info]: SSIM Mean Y:0.9032475
x264 [info]: PSNR Mean Y:28.781 U:37.189 V:39.386 Avg:30.266 Global:29.768 kb/s:3216.25
encoded 504 frames, 9.47 fps, 3215.30 kb/s

x264noaq3mbit.0.43.mkv (http://files.x264.nl/AQ/x264noaq3mbit.0.43.mkv) = 7.70 MB (8,078,599 bytes)
x264aq3mbit.0.43.mkv (http://files.x264.nl/AQ/x264aq3mbit.0.43.mkv) = 7.73 MB (8,110,271 bytes)

Screenshots may show local difference, in some parts the NOAQ is slightly better (the bits have to come from somewhere), but just look at these samples in motion!

Edit:
Here is the x264.exe with the patch: x264.720.dark.aq.rdrc.0.431.exe (http://files.x264.nl/AQ/x264.720.dark.aq.rdrc.0.431.exe) (pthreads/mp4 = yes, not made with make fprofiled)

[CruNcher]

yep that's even the same quality as i saw last time from Atemes Encoder :) it's a big step for X264.
i wonder if Mainconcepts Encoder can handle that now last time i saw it it couldn't :)


Reduced I-Frame Closed Gop Pulseing
http://mirror05.x264.nl/CruNcher/parkrun-newaq-reduced-iframe-pulseing.mkv

Still Realtime Encoded ABR 1Pass :)

[Don_Genaro]

I have done several encodings with Dark´s patch and comparint them to some without it. I have noticed a sustancial reduction in the size of the final stream.

Using CRF 21,22,23,24 the resulting files (Using the Patch) end with just 84% of the size of the files that don´t use the patched encoder and with no noticebly loss of quality (maybe even a gain in quality). Very good! :cool:

[Sagekilla]

I'll be testing the new AQ on a notoriously dark movie (Chronicles of Riddick anyone? I'll follow up on Pitch Black, which is even worse) Will post results as soon as my encoding finishes, which should be sometime around 5 PM tomorrow, for at least the AQ'd version. The non-AQ may not finish by then.

Edit: A cursory glance of the video using aq strength of 1 shows the new algo dealing with dark details very well. I'm viewing it before PC scale correction to exaggerate the blocking and I see a lot of detail in areas where normally the detail would be decimated.

[Razorholt]

Here is the x264.exe with the patch: x264.720.dark.aq.rdrc.0.431.exe (http://x264.nl/x264.720.dark.aq.rdrc.0.431.exe) (pthreads/mp4 = yes, not made with make fprofiled)

What's the difference between that build and the one on the OP?

[Dark Shikari]

What's the difference between that build and the one on the OP?Faster, statically built... minor differences though.

[Razorholt]

Thanks Darky for the previous answer. Also:

1. Since I'm using your GrainOptimizer, do you recommend using P4x4 with the new AQ?

2. How about intra? Have you fixed the bug or should I stick to Trellis?


Thanks,
- Dan

[Dark Shikari]

Thanks Darky for the previous answer. Also:

1. Since I'm using your GrainOptimizer, do you recommend using P4x4 with the new AQ?GrainOptimizer shouldn't get a specific benefit from p4x4 now, since the later versions have the blocks try to flip all at once when possible.
2. How about intra? Have you fixed the bug or should I stick to Trellis?


Thanks,
- DanThe bug was fixed a while ago--I removed the lambda-based AQ. Deadzone/trellis are both fine.

[Sagekilla]

Just thought about something.. I'd like to request a new feature for your AQ: sensitivity offset. It would only apply to automatic mode for obvious reasons. Would be nice to force AQ to be slightly more (or less) sensitive sometimes while using the same strength.

Edit: CoR is averaging 1400 kbps and looks great with your new AQ.

[Reuf Toc]

I've made a test on an animation clip mixing totaly flat parts and finely textured parts and I must say that the result is bluffing. AQ reduce considerably blocking and banding while retain a lot more details. Here are some pictures to illustrate my point :

Frame 120 :

AQ (http://reuf.toc.free.fr/AQ/120_AQ.png)
no AQ (http://reuf.toc.free.fr/AQ/120_no_AQ.png)

Frame 350 :

AQ (http://reuf.toc.free.fr/AQ/350_AQ.png)
no AQ (http://reuf.toc.free.fr/AQ/350_no_AQ.png)


x264 command line :

"C:\Program Files\megui\tools\x264\x264.exe" --pass 2 --bitrate 3000 --stats "E:\factory.stats" --level 4.1 --ref 3 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -1,-1 --subme 6 --trellis 1 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto --thread-input --sar 1:1 --progress --no-dct-decimate --output "E:\factory AQ 3M.mkv" "E:\factory.avs" --aq-strength 1.0

Log from the clip with AQ :

x264 [info]: slice I:7 Avg QP:24.43 size: 37380 PSNR Mean Y:42.64 U:44.68 V:45.30 Avg:43.23 Global:42.14
x264 [info]: slice P:986 Avg QP:26.31 size: 15043 PSNR Mean Y:40.72 U:43.05 V:43.30 Avg:41.34 Global:40.33
x264 [info]: slice B:347 Avg QP:27.21 size: 5232 PSNR Mean Y:40.16 U:42.82 V:42.35 Avg:40.76 Global:39.68

x264 [info]: mb I I16..4: 33.3% 50.7% 16.0%
x264 [info]: mb P I16..4: 14.2% 13.8% 2.0% P16..4: 40.4% 11.1% 3.4% 0.0% 0.0% skip:15.1%
x264 [info]: mb B I16..4: 0.6% 0.9% 0.2% B16..8: 36.9% 1.1% 2.4% direct: 0.6% skip:57.2%

x264 [info]: 8x8 transform intra:46.3% inter:65.3%
x264 [info]: direct mvs spatial:82.7% temporal:17.3%
x264 [info]: ref P 88.1% 8.0% 3.9%
x264 [info]: ref B 86.4% 8.8% 4.8%

x264 [info]: SSIM Mean Y:0.9667820
x264 [info]: PSNR Mean Y:40.589 U:43.001 V:43.068 Avg:41.199 Global:40.158 kb/s:3025.50

Log from the clip without AQ :

x264 [info]: slice I:7 Avg QP:25.57 size: 41877 PSNR Mean Y:43.93 U:45.58 V:46.10 Avg:44.44 Global:43.37
x264 [info]: slice P:986 Avg QP:27.68 size: 14984 PSNR Mean Y:41.56 U:43.74 V:43.95 Avg:42.14 Global:41.04
x264 [info]: slice B:347 Avg QP:28.50 size: 5379 PSNR Mean Y:40.77 U:43.42 V:42.95 Avg:41.37 Global:40.26

x264 [info]: mb I I16..4: 53.7% 29.4% 16.9%
x264 [info]: mb P I16..4: 25.3% 6.3% 2.0% P16..4: 31.7% 11.1% 3.7% 0.0% 0.0% skip:19.8%
x264 [info]: mb B I16..4: 0.8% 0.6% 0.2% B16..8: 32.1% 1.3% 2.8% direct: 0.6% skip:61.5%

x264 [info]: 8x8 transform intra:19.2% inter:63.5%
x264 [info]: direct mvs spatial:82.7% temporal:17.3%
x264 [info]: ref P 88.8% 7.5% 3.7%
x264 [info]: ref B 86.0% 9.0% 5.0%

x264 [info]: SSIM Mean Y:0.9659700
x264 [info]: PSNR Mean Y:41.370 U:43.665 V:43.700 Avg:41.953 Global:40.830 kb/s:3029.81


Here are the link for the two clip (16,1 MB) :

With AQ (http://reuf.toc.free.fr/AQ/factory_AQ.mkv)
Without AQ (http://reuf.toc.free.fr/AQ/factory_no_AQ.mkv)

Mirror on sendspace and mediafire if my FTP is too slow (33,3MB) :

Sendspace (http://www.sendspace.com/file/29ewp5)
Mediafire (http://www.mediafire.com/?499bz1j3w23)

[Dark Shikari]

And finally, the last bug... interlacing is fixed!

After a whole lot of CABAC debugging, narrowing down the bug, and finally a very good bug-spot by akupenguin, we found and fixed a bug in x264's handling of the CABAC encoding of mb_qp_delta with interlaced video.

Build and patch updated.

[Sharktooth]

@Reuf Toc: results are quite impressive. however, even if the second pic has less blocking, some details are missing (right side of the pic. look at details behind the colored smoke) but i guess the pic looks globally better.

@Dark Shikari: it seems you made CQMs a thing of the past. good job :)

[CruNcher]

Im looking forward to see Sagekillas Dark Encoding results because their it showed flaws to me with my test encode, tough it's still better then without any AQ at all, but not as good as the Old AQ in such specific situations. To enhance Dark Scenes a little more with this AQ it seems you have to lower --aq-strength from the 1.0 setting down this gives then a boost in Dark Sequences but it seemes to be not really visible @ all so 1.0 seems also for such scenes the most efficient if you don't wan't todo segmented Encoding with different AQ strengths ;)

@Dark
Did you tried your New AQ on your Vendeta Sample ?

[foxyshadis]

Hey, you didn't use 721. ;_; Like Chainmax, I'd still like to test this with prepass SATD ESA, but concentrate on what you think will make the best improvement.

I wonder how your elephant's dream running scene would look at this point.

[bob0r]

source:
ftp://ftp.ldv.e-technik.tu-muenchen.de/pub/test_sequences/720p/720p50_parkrun_ter.yuv

commandline:
aq:
start /belownormal /b /w x264.exe --pass 1 --bitrate 5096/3096 --threads auto --aq-strength 1.0 --thread-input --deblock 0:0 --bframes 3 --me dia --ref 1 --subme 1 --no-dct-decimate --partitions none --progress --fps=25 --output NUL 720p50_parkrun_ter.yuv 1280x720
start /belownormal /b /w x264.exe --pass 2 --bitrate 5096/3096 --threads auto --aq-strength 1.0 --thread-input --deblock 0:0 --bframes 3 --b-pyramid --bime --weightb --b-rdo --me umh --ref 5 --mixed-refs --subme 7 --trellis 1 --analyse all --8x8dct --no-fast-pskip --progress --fps=25 --output x264aq.mkv 720p50_parkrun_ter.yuv 1280x720

no aq:
start /belownormal /b /w x264.exe --pass 1 --bitrate 5096/3096 --threads auto --thread-input --deblock 0:0 --bframes 3 --me dia --ref 1 --subme 1 --no-dct-decimate --partitions none --progress --fps=25 --output NUL 720p50_parkrun_ter.yuv 1280x720
start /belownormal /b /w x264.exe --pass 2 --bitrate 5096/3096 --threads auto --thread-input --deblock 0:0 --bframes 3 --b-pyramid --bime --weightb --b-rdo --me umh --ref 5 --mixed-refs --subme 7 --trellis 1 --analyse all --8x8dct --no-fast-pskip --progress --fps=25 --output x264aq.mkv 720p50_parkrun_ter.yuv 1280x720

Change bitrate.

3mbit:
x264noaq3mbit.0.44.mkv (http://files.x264.nl/AQ/x264noaq3mbit.0.44.mkv)
x264aq3mbit.0.44.mkv (http://files.x264.nl/AQ/x264aq3mbit.0.44.mkv)

5mbit:
x264noaq5mbit.0.44.mkv (http://files.x264.nl/AQ/x264noaq5mbit.0.44.mkv)
x264aq5mbit.0.44.mkv (http://files.x264.nl/AQ/x264aq5mbit.0.44.mkv)

.exe:
x264.721.dark.aq.rdrc.0.44.exe (http://files.x264.nl/AQ/x264.721.dark.aq.rdrc.0.44.exe) (pthreads/mp4 = yes, not made with make fprofiled)

[CruNcher]

Hey, you didn't use 721. ;_; Like Chainmax, I'd still like to test this with prepass SATD ESA, but concentrate on what you think will make the best improvement.

I wonder how your elephant's dream running scene would look at this point.

the scene with the fine green moving plasma gradient in the background is much more problematic (visual blockfest) (low bitrate) for alot encoders then the running scene itself :D

[Dark Shikari]

Im looking forward to see Sagekillas Dark Encoding results because their it showed flaws to me with my test encode, tough it's still better then without any AQ at all, but not as good as the Old AQ in such specific situations. To enhance Dark Scenes a little more with this AQ it seems you have to lower --aq-strength from the 1.0 setting down this gives then a boost in Dark Sequences but it seemes to be not really visible @ all so 1.0 seems also for such scenes the most efficient if you don't wan't todo segmented Encoding with different AQ strengths ;)
The issue here is likely one of the automatic sensitivity; because the adaptive quantization is unwilling to move bits between frames, it chooses too low a sensitivity on such frames.

Wait, you're saying LOWERING the AQ strength gives a boost in dark sequences? That seems nonsensical.

[tetsuo55]

Is there any reason why i should not be using this (besides the slower encoding time)

[Dark Shikari]

Is there any reason why i should not be using this (besides the slower encoding time)The encoding time difference is extremely minimal at this point.

Reasons you shouldn't be using it:

1) Your source doesn't seem to benefit from it (unlikely, but some cartoons at low bitrates seemed to suffer in my early tests).

2) uh, I can't think of anything else... :p

Some people seem to be pressing to make this an x264 default, even though its not even in SVN yet :)

[DeathTheSheep]

Like Chainmax, I'd still like to test this with prepass SATD ESA...


Woah, add me to the group. I'm stuck with 681.

[tetsuo55]

The encoding time difference is extremely minimal at this point.

Reasons you shouldn't be using it:

1) Your source doesn't seem to benefit from it (unlikely, but some cartoons at low bitrates seemed to suffer in my early tests).

2) uh, I can't think of anything else... :p

Some people seem to be pressing to make this an x264 default, even though its not even in SVN yet :)

~Default~ ~default~ ~default~

[CruNcher]

Dark why does your build creates every run (small) different results? isn't your AQ deterministic?
And yes exactly that's whats happening with the scene you also have lowering the AQ-strength improves it (bitrate increase) higher SSIM, highering it lowers SSIM (bitrate decrease).

[Dark Shikari]

Dark why does your build creates every run (small) different results? isn't your AQ deterministic?It should be 100% deterministic. Is it deterministic without AQ enabled?

[ToS_Maverick]

nope it isn't, with our without AQ

[Dark Shikari]

nope it isn't, with our without AQSounds like its not my problem then :p

[CruNcher]

nope it isn't, with our without AQ

Strange for me it's only non deterministic with AQ enabled without it i get every run the same results with it every run small differences.
--aq-strength 0 (allways the same results) without --aq-strength on the commandline same results every run as with --aq-strength 0 and with parkrun starting with --aq-strength 0.4 allways different results each run.

1st run = SSIM Mean Y:0.8447759
2nd run = SSIM Mean Y:0.8439500
3rd run = SSIM Mean Y:0.8447161
4rd run = SSIM Mean Y:0.8446099

Settings = x264-aq 720p50parkrun.yuv 1280x720 --bitrate 3000 --level 4.1 --
min-keyint 1 --keyint 25 --bframes 0 --ref 1 --weightb --subme 1 --8x8dct --qpmi
n 15 --trellis 0 --nf --me dia --threads auto --partitions all --no-fast-pskip -
-no-dct-decimate --aq-strength 0.4 --progress --sar 1:1 -o parkrun-0.4.mkv

[bob0r]

And with 1 thread?

[CruNcher]

With 1 thread it's ok @ 0.4 allways SSIM Mean Y:0.8448698 @ 12.40 fps
With 2 threads @ 0.4 SSIM changes every run @ 22.62 fps

Seems not threadsafe Dark ;)

[Sagekilla]

Sorry guys, results on Chronicles of Riddick won't be up until tomorrow at the earliest now. I ended up having to standby my computer so I could sleep (Had finals today) so I'm only 20% through the first encode.

[bob0r]

its thread safe, just not non deterministic thread safe.
with or without aq, makes no difference.
pengvado is aware of this issue, and one days hopes to find the cure :)

[Snowknight26]

In the .diff, it says the --aq-strength can go to 1.4 for strong AQ, but in the 1st post you say 1.0. :x

[Dark Shikari]

In the .diff, it says the --aq-strength can go to 1.4 for strong AQ, but in the 1st post you say 1.0. :xI didn't say what it can go to, I just gave examples.

There is technically no limit on how high AQ can go, but after about 2.0-3.0 it will really start to backfire. Perhaps in the help I should say 1.0 is a good medium.

[Sagekilla]

In the .diff, it says the --aq-strength can go to 1.4 for strong AQ, but in the 1st post you say 1.0. :x

1.0 is the recommended setting to use to start with. It gives, on average, very good results. If you read the help, it says 0.7 is "medium" and 1.4 is "stronger."

[chipzoller]

DS, in your original post you mentioned it may not be good for Anime/Cartoon and that you wouldn't use it there. Do you still maintain this, even at AQ's present state?
Is --aq-strength 1.0 a safe-to-use general setting for anime/cartoon if using AQ is advisable?

[Snowknight26]

I didn't say what it can go to, I just gave examples.

Add --aq-strength X to the commandline, where X is a value between 0.0 and 1.0.

Sorry, seemed a little misleading. :p

[Dark Shikari]

DS, in your original post you mentioned it may not be good for Anime/Cartoon and that you wouldn't use it there. Do you still maintain this, even at AQ's present state?
Is --aq-strength 1.0 a safe-to-use general setting for anime/cartoon if using AQ is advisable?Try it... I'm not sure at this point. I haven't done much testing.

If you do testing, post the results.

[Razorholt]

This new AQ seems to darken the picture, am I right?

[Dark Shikari]

This new AQ seems to darken the picture, am I right?AQ has absolutely no affect on image brightness. If it does so, its because your levels are off.

[Sagekilla]

Here's a frame I grabbed from my AQ. (That's still encoding) AQ did a pretty good job at preventing blocking from occurring on the jacket, as you can see. Unfortunately, I'll have to run the encode a third time to possibly increase the strength again. In the second image, you can see some blocking on the light shining in.

http://img.photobucket.com/albums/v621/Sagekilla/Riddick.png

http://img.photobucket.com/albums/v621/Sagekilla/riddick2.png

[DeathTheSheep]

its thread safe, just not non deterministic thread safe.
with or without aq, makes no difference.
pengvado is aware of this issue, and one days hopes to find the cure :)

I have a stopgap cure: don't use threads auto. Use threads=3 if on dual core and threads=6 on quad.

[salehin]

Sagekilla, is it possible for you to share your avs script, encode cmd, & logs. thanks :)

[Sagekilla]

Sagekilla, is it possible for you to share your avs script, encode cmd, & logs. thanks :)

Yes, I'll post my script and settings for now. I have to wait for my encoding to finish before I can post my logs however.


AVS Script
SetMTMode(2,2)
MPEG2Source("source.d2v")
Crop(0,62,0,-66).Spline36Resize(848,352)

# Some light processing, helps x264 compress better.
o = last
b1vec = o.MVanalyse(pel=2,overlap=2,idx=1,delta=1,isb=true)
f1vec = o.MVanalyse(pel=2,overlap=2,idx=1,delta=1,isb=false)
return(o.MVDegrain1(b1vec,f1vec,thSAD=350,idx=2))

x264 Settings
AQ On: x264 --keyint 1000 --crf 18 --ref 4 --mixed-refs --no-fast-pskip --bframes 16 --bime --weightb --b-pyramid --b-rdo --8x8dct --subme 7 --me umh --trellis 1 --aq-strength 1 --threads auto --thread-input --progress --output "videoAQ.264" --pass 1 --stats "stats_AQ.log" "source.avs"

AQ Off: x264 --keyint 1000 --crf 18 --ref 4 --mixed-refs --no-fast-pskip --bframes 16 --bime --weightb --b-pyramid --b-rdo --8x8dct --subme 7 --me umh --trellis 1 --aq-strength 0 --threads auto --thread-input --progress --output "video_NoAQ.264" --pass 1 --stats "stats_NoAQ.log" "source.avs"

Note, I used a long keyint because I don't mind the long latency to decode and I don't seek much to begin with. Change this to something a little more sane (250) if you wish.

[bob0r]

My bad on the nondeterministic issue.
This is only the case with Dark's AQ.

I guess the fixes since 713 also solved the threads issue with different bitrates.

Still as Dark explained AQ + something within x264 code can trigger something nondeterministic.
Guess they have to figure that one out :)


So if any of you got some nondeterministic issue, please report a reproducable method and pengvado will look into it!

[CruNcher]

Yes, I'll post my script and settings for now. I have to wait for my encoding to finish before I can post my logs however.


AVS Script
SetMTMode(2,2)
MPEG2Source("source.d2v")
Crop(0,62,0,-66).Spline36Resize(848,352)

# Some light processing, helps x264 compress better.
o = last
b1vec = o.MVanalyse(pel=2,overlap=2,idx=1,delta=1,isb=true)
f1vec = o.MVanalyse(pel=2,overlap=2,idx=1,delta=1,isb=false)
return(o.MVDegrain1(b1vec,f1vec,thSAD=350,idx=2))

x264 Settings
AQ Off: x264 --keyint 1000 --crf 18 --ref 4 --mixed-refs --no-fast-pskip --bframes 16 --bime --weightb --b-pyramid --b-rdo --8x8dct --subme 7 --me umh --trellis 1 --aq-strength 1 --threads auto --thread-input --progress --output "videoAQ.264" --pass 1 --stats "stats_AQ.log" "source.avs"

AQ On: x264 --keyint 1000 --crf 18 --ref 4 --mixed-refs --no-fast-pskip --bframes 16 --bime --weightb --b-pyramid --b-rdo --8x8dct --subme 7 --me umh --trellis 1 --aq-strength 0 --threads auto --thread-input --progress --output "video_NoAQ.264" --pass 1 --stats "stats_NoAQ.log" "source.avs"

Note, I used a long keyint because I don't mind the long latency to decode and I don't seek much to begin with. Change this to something a little more sane (250) if you wish.

Ehhh you mean AQ ON/AQ OFF? and don't you think those settings are a little insane ;)

[foxyshadis]

Woah, add me to the group. I'm stuck with 681.

Sorry, meant to type DTS.

I get some non-determinism, but it's only in the least-significant two digits of the SSIM. Not something I'm going to worry about.

[chipzoller]

When using CRF mode, is there a threshold one should not go under while using AQ, i.e. what's the lowest CRF that can be used with AQ while still producing helpful results?

[bob0r]

....

I get some non-determinism, but it's only in the least-significant two digits of the SSIM. Not something I'm going to worry about.

With AQ or None AQ?
If you get it with revision 721 SVN, please report a way to reproduce.

[Dark Shikari]

When using CRF mode, is there a threshold one should not go under while using AQ, i.e. what's the lowest CRF that can be used with AQ while still producing helpful results?I don't think there's any. AQ should work well at any bitrate--the only exception to this rule is if the bitrate is so low that the qp_deltas become a considerable portion of the bit cost of the video.

[Sagekilla]

Ehhh you mean AQ ON/AQ OFF? and don't you think those settings are a little insane ;)

D'oh, yeah it should be AQ on/off. Fixed that. Those settings are perfectly sane for me :) 8 fps on my dual core Opty 170. Besides, that's partially the reason why I can get good quality @ 1400 kbps on CoR.

[jmnk]

Ehhh you mean AQ ON/AQ OFF? and don't you think those settings are a little insane ;)
@CruNcher: could you elaborate on why these settings are 'little insane'?

[burfadel]

Just a slightly irrelevent question, but its something that may be asked or have confusion about when your patch build is updated! Afterall, depending on the actual change, using the same settings from 720 to 721 may result in inaccurate comparisons?...

On the revision log (from www.x264.nl) for rev 721, its says:

- change the meaning of --ref: it now selects DPB size (including B-frames), rather than L0 size (which B-frames are added to)

Does that mean if we normally select 5 reference frames and set b frames to 16, that --ref should now be set to 21?...

How would this affect the use of Megui/Staxrip etc?

[Dark Shikari]

Just a slightly irrelevent question, but its something that may be asked or have confusion about when your patch build is updated! Afterall, depending on the actual change, using the same settings from 720 to 721 may result in inaccurate comparisons?...

On the revision log (from www.x264.nl) for rev 721, its says:

- change the meaning of --ref: it now selects DPB size (including B-frames), rather than L0 size (which B-frames are added to)

Does that mean if we normally select 5 reference frames and set b frames to 16, that --ref should now be set to 21?...

How would this affect the use of Megui/Staxrip etc?No, that's not what the patch means.

Previously, DPB size required was equal to (--ref + [0,1,2]), with 0 in the case of no b-frames, 1 in the case of b-frames, and 2 in the case of b-pyramid. Now, I think its equal to (--ref + 1), no matter what. Akupenguin, feel free to correct me if I'm not exactly correct.

[Sagekilla]

@CruNcher: could you elaborate on why these settings are 'little insane'?

It's insane because I enabled every setting short of esa using 16 reference threads, and unlimited merange. Some of these settings can easily be turned off without increasing the bitrate massively (think 1500 kbps instead of 1400 kbps) while providing a sizeable speed boost. But, I prefer the extra quality. ESA I don't use because the speed vs quality tradeoff isn't worth it yet.

[Dark Shikari]

It's insane because I enabled every setting short of esa using 16 reference threads, and unlimited merange. Some of these settings can easily be turned off without increasing the bitrate massively (think 1500 kbps instead of 1400 kbps) while providing a sizeable speed boost. But, I prefer the extra quality. ESA I don't use because the speed vs quality tradeoff isn't worth it yet.I'd say you could probably improve that commandline by adding a few more refs--I've found that 4 is generally not where the the curve begins to top out, even for live-action.

[burfadel]

No, that's not what the patch means.

Previously, DPB size required was equal to (--ref + [0,1,2]), with 0 in the case of no b-frames, 1 in the case of b-frames, and 2 in the case of b-pyramid. Now, I think its equal to (--ref + 1), no matter what. Akupenguin, feel free to correct me if I'm not exactly correct.

How does that affect b-pyramid then, if its always +1 instead of +2?

[Sagekilla]

I'd say you could probably improve that commandline by adding a few more refs--I've found that 4 is generally not where the the curve begins to top out, even for live-action.

I probably could yes, especially if you merged the fast ref search diff to this build. I use very slow preprocessing though (to remove grain, noise, etc), which manages to cut down on the bitrate needed by a huge margin compared to increasing refs. I actually was using 6 refs back when your fast ref build was being actively developed, but since the AQ on this build is vastly better than the older AQ I'd rather use this one. All a matter of tradeoffs, no?

On a side note, CoR is coming out very nicely.. 154k of 192k frames encoded and the file is a mere 1.09 GB big. Looking at the end file being around 1.36 GB, possibly more. Throw on an extra 440 MB for the audio track though.

[CruNcher]

I i finished my 3 longcut encodes (CRF 25) :)

No AQ

x264 [info]: SSIM Mean Y:0.9849869
x264 [info]: PSNR Mean Y:46.721 U:47.332 V:51.035 Avg:47.291 Global:46.857 kb/s:
4125.27

encoded 6842 frames, 10.16 fps, 4132.79 kb/s

AQ ON --aq-strength 1.0

x264 [info]: SSIM Mean Y:0.9843622
x264 [info]: PSNR Mean Y:45.762 U:46.754 V:50.071 Avg:46.394 Global:46.023 kb/s:
4228.15

encoded 6842 frames, 9.72 fps, 4235.46 kb/s

Old Aq --aq-strength 0.9

x264 [info]: SSIM Mean Y:0.9858071
x264 [info]: PSNR Mean Y:46.987 U:47.763 V:51.368 Avg:47.590 Global:47.106 kb/s:
6594.71

encoded 6842 frames, 9.64 fps, 6602.20 kb/s

[Sagekilla]

Can we some screen caps of that clip, CruNcher?

[CruNcher]

First i have todo a subjective test of all 3 to determine the problematic areas for this the 3rd encode has to finish first

[foxyshadis]

With AQ or None AQ?
If you get it with revision 721 SVN, please report a way to reproduce.

With AQ, I didn't bother to test multiple identical SVN encodes.

How does that affect b-pyramid then, if its always +1 instead of +2?

It now uses the same number of references for all frames, eliminating the need to trade off b or b-pyramid for extra references.

Cruncher, pleeeease link instead of inlining the images this time. ;)

[akupenguin]

Now, I think its equal to (--ref + 1), no matter what. Akupenguin, feel free to correct me if I'm not exactly correct.
DPB is equal to --ref.
Unless --ref is less than the minimum (1 for P only, 2 for B-frames, 3 for pyramid), in which case --ref selects the number of L0 refs used for P-frames (like it did before) and DPB is equal to said minimum.

[CruNcher]

Ok im finished and the result is the same as what i told you before from the visual side the difference in Bright Scenes between NoAQ and NewAQ is almost 0 (if their is a enhancement i couldn't see it ,most probably the bitrate for those scenes is allready perfectly chosen by the encoder), but the Black Scenes with very defined gradients (and especialy when noise was left) did improved with the NewAQ compared to noAQ a little, see bellow. And as you allready see from the Size difference the OldAQ gave the best visual results also in the Dark Scenes (but this is still unbalanced and not really useable that way most probably this behaveiour can be tweaked alot).

Im gonna post pictures from every Scene this testcut contains and the problematic scenes (that doesn't get enhanced enough by either the NewAQ or NoAQ).

The Testcut is buildup from 6 scenes 5 of them are dark 1 is very bright and detailed (the Dark scenes are also detailed and when wrong encoded band extremely and most of the visible problems are in the ROI then, so a very bad situation)

Scene 1 = http://s6.directupload.net/images/080119/xjbzbe2m.png
Scene 2 = http://s6.directupload.net/images/080119/hc5re8nm.png
Scene 3 = http://s6.directupload.net/images/080119/hc5re8nm.png
Scene 4 = http://s5.directupload.net/images/080119/bfs48jze.png
Scene 5 = http://s2.directupload.net/images/080119/8bmbnscd.png
Scene 6 = http://s2.directupload.net/images/080119/lxw5ntsx.png

Problematic Scenes (areas)

Scene 2
No AQ = http://s4.directupload.net/images/080119/b3r6jc5i.png
New AQ = http://s3.directupload.net/images/080119/j4933af4.png
Old AQ = http://s6.directupload.net/images/080119/ojc2ha8s.png

Scene 3
No AQ = http://s5.directupload.net/images/080119/ackxyl3o.png
New AQ = http://s3.directupload.net/images/080119/gfl2dcw9.png
Old AQ = http://s1.directupload.net/images/080119/dggqn965.png

Scene 5-1
No AQ = http://s3.directupload.net/images/080119/oxuq5zo3.png
New AQ = http://s1.directupload.net/images/080119/zm3m6gj3.png
Old AQ = http://s2.directupload.net/images/080119/afa45iww.png

Scene 5-2
No AQ = http://s6.directupload.net/images/080119/mvz5ahzn.png
New AQ = http://s1.directupload.net/images/080119/ebgix8it.png
Old AQ = http://s3.directupload.net/images/080119/qdocncsk.png

Scene 6
No AQ = http://s1.directupload.net/images/080119/e29e7rgw.png
New AQ = http://s2.directupload.net/images/080119/tnenotri.png
Old AQ = http://s5.directupload.net/images/080119/yno3ves7.png

All in all the NewAQ is very good compared to noAQ at all (especialy in those Dark Scenes they look better in Motion alot then with NoAQ this way) and this just for 100 kbps more bitrate usage (think about the OldAQ result and the bitrate difference yada yada think about 100 kbps more *g*), everyone should invest that for this big HVS Improvement :)
and it definately should go into SVN :D

Final Size
No AQ = 140 mb
New AQ = 144 mb
Old AQ = 224 mb

Tough in a ABR Encode @ 3 Mbit Visualy the New AQ fails in the Problematic ROI Scenes and the old AQ is supirior in those and shows no visible sign of any Visual degredation to any of the other scenes.

[Sagekilla]

Nice results there CruNcher. Very bothersome that we keep getting closer to that "great" flat area performance yet are still quite far from it.

Dark Shikari, you said increasing aq-strength affects how high and low a qp can raise for a frame. Is this all that it affects in the new build or do higher values give a greater tendency to allocate more bits? Because in my testing, it seemed like going from 0.5 to 1, and then to 2 had little effect on overall quality of flat areas.

[Dark Shikari]

Nice results there CruNcher. Very bothersome that we keep getting closer to that "great" flat area performance yet are still quite far from it.

Dark Shikari, you said increasing aq-strength affects how high and low a qp can raise for a frame. Is this all that it affects in the new build or do higher values give a greater tendency to allocate more bits? Because in my testing, it seemed like going from 0.5 to 1, and then to 2 had little effect on overall quality of flat areas.One of the problems with the existing AQ is that it makes a compromise--that it won't move bits considerably between frames. As a result, it avoids screwing up ratecontrol--but that also means that in extremely flat frames, it isn't willing to lower the framewide quantizer to compensate.

[DeathTheSheep]

Which is a darn shame if you use CQ. :)

[Dark Shikari]

Which is a darn shame if you use CQ. :)Actually, one interesting thought: AQ currently limits the amount it raises and lowers QPs. If I make an option to customize or remove that limitation on the commandline, then what you can do is use a static (non-automatic) sensitivity--and use AQ for ratecontrol with CQ as a base :devil:

[Sagekilla]

Any chance of a patch coming out to try this out? *wink wink* :)

[Dark Shikari]

Line 293, ratecontrol.c clips the QPs to (-5 * AQ strength, 5 * AQ strength). Comment that out, or change it to whatever you want.

That's all you need to do.

I *strongly* suggest you don't put the strength too high when doing this, and use a sensitivity between 15 and 30 or something like that--you'll have to fool around to find the best values.

[CruNcher]

this is so funny i did now a Encode @ 3 Mbit with the Old Aq and geez it looks stunning in those scenes even better then the CRF 25 New Aq encode with 4 mbit so the Old one is really thought for ABR or 2 Pass but not CRF :) those scenes look now as if they used the 6 mbit :D and the degredation on the other scenes is visualy not visible :) This is what i wanted (balanced distribution) :D

SSIM everything lowered as expected but the Visual Quality is Perfect :) no ROI problems anymore in any of those Problematic Scenes i posted above and the Final size is 104 Mb :)

x264 [info]: SSIM Mean Y:0.9807726
x264 [info]: PSNR Mean Y:44.941 U:46.045 V:49.483 Avg:45.600 Global:45.188 kb/s:
3043.12

encoded 6842 frames, 9.49 fps, 3050.65 kb/s

Line 293, ratecontrol.c clips the QPs to (-5 * AQ strength, 5 * AQ strength). Comment that out, or change it to whatever you want.

That's all you need to do.

I *strongly* suggest you don't put the strength too high when doing this, and use a sensitivity between 15 and 30 or something like that--you'll have to fool around to find the best values.

gonna try that and do the 3 mbit again, i think this should improve it like the Old AQ also for ABR :)

[TheRyuu]

I've built it with the newest rev. (r721) with this patch, mp4 output, pthreads, and avis input.

http://www.fileducky.com/NWvOdpST/

I've tested it to the point were it built and it didn't error.
Just thought I'd share this build.

[Dark Shikari]

Here are some things that I want people to feel free to experiment with my AQ (since the function itself is relatively easy to modify):

1. I find that the cap I've placed on the AQ adjustment seems like a bad solution to the issue of raising/lowering quantizers too much. Can anyone here come up with a better solution?

2. As mentioned earlier, I tried to avoid the problem of AQ redistributing bits with the automatic sensitivity. However, this doesn't bode well for very flat scenes, it seems. Is there a way to avoid screwing up ratecontrol while biasing the AQ a bit in the favor of flat scenes getting lower quantizers? This would most likely take the form of some sort of bias in the summing of the x264_aq_autosense() function.

3. Would the AQ formula benefit from any change in scaling method?

Etc.

[CruNcher]

Dark I tried now to max it out @ CRF 25 like the oldAQ does i did it with --aq-sensitivity 40 without comenting out that line now the dark small test scene gets the same enhancement as the OldAQ did @ --ag-strength 0.9 --aq-sensitivity 15 (even more bitwise but not visualy have to find the perfect visual max out point now) :) so the key for the new AQ seems to be as i thought at first really to be in the --aq-sensitivity :D

This is the max i get now with the new AQ
x264 [info]: SSIM Mean Y:0.9918510
x264 [info]: PSNR Mean Y:49.527 U:50.649 V:52.874 Avg:50.108 Global:49.794 kb/s:
8130.14

encoded 701 frames, 9.09 fps, 8136.86 kb/s

This is the max i got with the OldAQ @ --aq-strength 0.9 --aq-sensitivity 30
x264 [info]: SSIM Mean Y:0.9916371
x264 [info]: PSNR Mean Y:49.032 U:50.341 V:52.570 Avg:49.657 Global:49.356 kb/s:
7589.89

encoded 701 frames, 9.13 fps, 7596.56 kb/s

Btw isn't this now the same behaveiour like a VBR Ratecontroll ? (you get the best possible output, without any limitations) i think such a mode is really missing and this seems a cheap way to force this by useing AQ (Atemes Encoder has this mode and seems X264 now too) :D (and it seems to work perfectly) and by maxing it out it seems to work perfectly then in ABR for lower bitrates too (best possible visual result for the bitrate target) :) i think this is indeed a big big step. I think this is even better then CRF because now you can decide with the tools of H.264 how much compression and complexity in the end you wan't (like a Stream Complexity Ratecontrol) and how much Speed loss you wan't to invest for that (both sides Encoding/Decoding or based on Compression Efficiency) if it really works out as i think (this is what i allways dreamed of for a mode for EDP) :) as base for this CRF 19 or 21 should be used i think.

If my ideas work out i think there are 2 new Modes born for X264 in this moment :D
Variable Bitrate Mode (useing CRF or QP X as start)
Stream Complexity Mode (Encoding/Decoding Speed Mode)


It looks like this now

OldAQ
CRF/ABR Visually max out (CRF doesn't max out as good as with New AQ)

New AQ
CRF Visually maxes out (better then OldAQ)
ABR didn't found yet how to max it out Visually

gonna try that aproach now on ParkRun with the Old AQ maxed out :)

[DeathTheSheep]

Actually, one interesting thought: AQ currently limits the amount it raises and lowers QPs. If I make an option to customize or remove that limitation on the commandline, then what you can do is use a static (non-automatic) sensitivity--and use AQ for ratecontrol with CQ as a base :devil:

Wow, I was actually just thinking about doing something like that... sheesh. Well, I'm encoding with it now. Strength is at 0.7, and I'm trying both 15 and 30 sensitivity.

[CruNcher]

This is really a hard balance act definately the OldAQ can't improve Parkrun in any way, now i tried to balance the New AQ between Parkrun and my testcut and got better results then before in the ROI scenes (a little SSIM decrease in Parkrun, but still visualy better then No AQ) but still it's nowhere near the OldAQ results on my testcut (ABR).

[DeathTheSheep]

Okay, some results.
0.9741502 SSIM for original code, q29 AQ07. (4248KB)
0.9737121 SSIM for RC mod, q28 AQ07. (4258KB)

Visually, undecided.
Of course, I'll try other settings... but at least as far as sensitivity goes, 15 might just be too low.

[Dark Shikari]

Okay, some results.
0.9741502 SSIM for original code, q29 AQ07. (4248KB)
0.9737121 SSIM for RC mod, q28 AQ07. (4258KB)

Visually, undecided.
Of course, I'll try other settings... but at least as far as sensitivity goes, 15 might just be too low.It is too low. That sets the "center variance" at about 25000. 25 will set it at around 200,000.

[DeathTheSheep]

I'm using sensitivity 30 now (the "other extreme"), and things might be looking up.

[edit] Filesize is ridiculously high, I'm going to have to up my qp by at least 2 notches...

[chipzoller]

I'm finding this AQ function very handy. Do you know if it is currently used in any other encoders? And this may be a stupid question as I'm not an expert on the inner-workings of x264 and, indeed, H.264 in general, but does using AQ in any way tweak or perhaps "break" the H.264 spec to your knowledge? Is this a profile-limited tool, or can it be used in the creation of any stream?

[CruNcher]

I think i found a good balance :) have to test it on the longcut first :D

[Sharktooth]

@chipzoller: no. it doesnt break anything. other encoders use some kind of AQ too.

[DeathTheSheep]

Q31: 0.9750365 (RC, strength 1.0, size 4293KB)
Q29: 0.9732632 (std strength 1.0, size 4129KB)
Q29: 0.9741502 (std strength 0.6, size 4248KB)

Since it's quants we're dealing with here, it's hard to settle on a single, uniform bitrate. The three points here seem almost linear in bitrate/SSIM scaling, so I really can't say. (Needless to say, it takes forever to take derivatives of each one's SSIM vs bitrate curve for comparison).

Should I post the resultant clips?

[Dark Shikari]

Q31: 0.9750365 (RC, strength 1.0, size 4293KB)
Q29: 0.9732632 (std strength 1.0, size 4129KB)
Q29: 0.9741502 (std strength 0.6, size 4248KB)

Since it's quants we're dealing with here, it's hard to settle on a single, uniform bitrate. The three points here seem almost linear in bitrate/SSIM scaling, so I really can't say. (Needless to say, it takes forever to take derivatives of each one's SSIM vs bitrate curve for comparison).

Should I post the resultant clips?
Compare visual quality, not SSIM.

Eliminating blocking in very flat areas, for example, doesn't generally help SSIM.

[DeathTheSheep]

That's true, but at exactly the same bitrate (<1KB difference out of >4000), the large SSIM increase and PSNR slight increase isn't easy to count out.

As for blocking, it's a non-issue for me anyway at Q31...or is it?
And now for a complete 180 degree turnaround, I'm going to check the visual quality of the clips in dark scenes, if you'll excuse me. :p

[CruNcher]

I give up it seems impossible that way (changeing settings) to achive good results on the problematic ROI stuff in this testcut @ 3 Mbit with ABR and the New AQ (with CRF it's no problem or ABR and OldAQ)

[DeathTheSheep]

Okay, here's a test pack. The settings (and SSIM) can be inferred from the filenames. One of the files is without AQ, and one of them uses standard .431 at strength 0.7 with auto threshold, I believe. All AQ strengths are 1.0 unless specified otherwise in the filename with st_. As you can tell, I tried to achieve roughly 4201KB for each file; however, 9749861_RC_q29_s21 was blatantly oversized (lowering the sensitivity even one point makes it severely undersized).

http://gabe.ej.am/samples/

[CruNcher]

Okay, here's a test pack. The settings (and SSIM) can be inferred from the filenames. One of the files is without AQ, and one of them uses standard .431 at strength 0.7 with auto threshold, I believe. All AQ strengths are 1.0 unless specified otherwise in the filename with st_. As you can tell, I tried to achieve roughly 4201KB for each file; however, 9749861_RC_q29_s21 was blatantly oversized (lowering the sensitivity even one point makes it severely undersized).

http://gabe.ej.am/samples/

Wussa what for fast pictures (not used to this speed) very hard to realize any difference @ all, but they definitely all behind my visual understanding of quality, especialy the ringing and extreme blurienes would annoy me to death (it allready does in my stuff) and especialy with H.264 (blurrienes of X264 can for sure be enhanced tough). Im only used @ this problem from past codecs and ASP (except blurrienes) since the better partitioning, Qpel and Inloop deblocker should prevent such stuff from happening in H.264 (was the source that bad?). I would say no real visual improvement there visible that might come from the New AQ.

[DeathTheSheep]

Yes, the source wasn't exactly par excellence, if you catch my drift. I do see markedly better quality (detail preservation) in the q29 (and to some extent the q30) samples as opposed to both original and AR-std.

Also keep in mind this is baseline QVGA resolution (320x240), not even VGA, much less HD, not to mention q30+ equivalent bitrate, which might account for some "blurriness." :D

[edit] CruNcher: for your viewing pleasure, a max setting DivX6.8Pro and CruNcherEDP1.4.5 have been added to the site. Even at max settings, ASP (at least without tons of filtering) looks like sheer buttocks compared to baseline AVC at this bitrate/resolution.

[CruNcher]

(detail preservation) - ehh what details if i might allowed to ask, i see only a bunch of colors flipping around in ultra speed ;)

*hides far far away deep inside of looney tunes and hanna babera land from the wrath of the anime community that might come over him like a Dragonball @ the sunset dawn, for this statement*

[DeathTheSheep]

NAPA: Vegeta, what does the SSIM say about its quality level?

VEGETA: It's over nine thousaaaand!!

NAPA: What 9000?!! There's no WAY that can be right!

You don't watch much anime, do you, CruNcher? Take a look at how your EDP with max settings (except GMC) does on the clip, for instance... tsk, tsk. If you don't know the rubric, maybe you shouldn't judge. ;)

[Dark Shikari]

My hunch was right.

A constant AQ sensitivity with unlimited (no clipping) AQ, in a sense using AQ to hack at ratecontrol, is IMPRESSIVE, to say the least. And I mean astounding.

Before (this thread's automatic-sensing AQ):

http://i1.tinypic.com/80neaz9.png

After:

http://i16.tinypic.com/6l8ke14.png

Obviously the framesize is drastically different, but that's the point; this scene needed more bits. The effect of this is simply astounding; it may make this AQ even better, though it cannot work with CRF if you want any hope of a bitrate close to that you would get without AQ.

[DeathTheSheep]

You just took out that one line and used sensitivity X, right? That's what I did (see previous posts).

If so, what sensitivity precisely did you use?

[CruNcher]

Indeed i don't, but don't forget your sample uses inloop deblocking and ASP isn't haveing that feature it can make only use of outside Postprocessing, but you know all that. Think about the same AVC Baseline sample but with the sharpness of ASP, im sure it's doable someone just has to find the right way to achive that and nope you won't with only turning off deblocking unfortunately it's not that easy ;). Btw i watched anime back when i was a kid but todays stuff is ehh different i loved http://www.imdb.com/title/tt0185110/ , now you know my depest secret ;)

Yes Dark as i said it's like a VBR then it gives the bitrates to those scenes that need it the most, but i have problems to reproduce the same effect in ABR with the OldAQ it works in both RC modes, tough the OldAQ doesn't improve ParkRun the way the new does. :)

[Dark Shikari]

You just took out that one line and used sensitivity X, right? That's what I did (see previous posts).

If so, what sensitivity precisely did you use?Sensitivity was 30, which seemed like a good medium. And yes, I just removed that line.

[DeathTheSheep]

Hmm, I use 25. 30 seems rather excessive to me. Besides, as you can see from my previous post (or gabe.ej.am/samples/), there's many ways to get at the same filesize via QP, with SSIM and visual quality to match. I find qX without ratecontrol is approximately equal to -qX --aq-sensitivity ~25 OR -q[X+1] --aq-sensitivity ~30 for a good range of balanced sources.

[Dark Shikari]

Also, it was requested that I post this updated, working version of me-prepass (supposedly, I haven't tested it) on doom9, so I will post it here.

ME-prepass patch, working with 721, supposedly. (http://pastebin.com/f387ef1d)

[Reuf Toc]

The result of your hack is really awesome :eek:

Here is a gif of your pictures with contrast enhanced :

http://img255.imageshack.us/my.php?image=sanstitre1rl1.gif

[DeathTheSheep]

May I hope you have the updated, working satd floating around there as well? If so, here's your request... :devil:

[Dark Shikari]

Working SATD doesn't exist at the moment, peng will have to updated the patch.

Also, apparently that Prepass patch is broken and won't compile. Blegh. At least I did make it.

Also, major major major major update. Read the new instructions, and be shocked at the quality improvement.

[DeathTheSheep]

"major major major major update" = disable auto threshold (in 2pass)? Hmm... ;)

[Dark Shikari]

"major major major major" = disable auto threshold (in 2pass)? Hmm... ;)Major major = remove the limiter when not using automatic thresholding.

I could also make it so that the default sensitivity depends on which pass mode you choose, but for now, set it yourself.

This is the reason why I call it "major major":

http://img255.imageshack.us/img255/2779/sanstitre1rl1.gif

[DeathTheSheep]

Ah, I was wondering when you'd post another one of your aniGifs. Now with contrast enhancement!! But I realize you still forgot to mention what the other two "major"s were for. I have a suggestion: adaptive smexiness profile. ;)

Well, to tell you the truth, I don't see any clear benefit in keeping the limiter--even in auto mode.

[Sagekilla]

Oh curses.. I just started a new encode a few minutes ago, now I have to go check this new update out.

[Dark Shikari]

Ah, I was wondering when you'd post another one of your aniGifs. Now with contrast enhancement!! But I realize you still forgot to mention what the other two "major"s were for. I have a suggestion: adaptive smexiness profile. ;)

Well, to tell you the truth, I don't see any clear benefit in keeping the limiter--even in auto mode.Under the current system, its there for very good reason. This is because if the frame is flat, it'll try to raise the quantizers of the few complex blocks in the frame by a huge amount--causing problems, since it won't lower the blocks in the rest of the frame enough. It *correctly* calculated the *difference* in quantizers between the flat and complex blocks, except that since it wasn't willing to vastly lower the quantizer of the frame, the complex blocks would end up with far too high a quantizer.

[Sagekilla]

Under the current system, its there for very good reason. This is because if the frame is flat, it'll try to raise the quantizers of the few complex blocks in the frame by a huge amount--causing problems, since it won't lower the blocks in the rest of the frame enough. It *correctly* calculated the *difference* in quantizers between the flat and complex blocks, except that since it wasn't willing to vastly lower the quantizer of the frame, the complex blocks would end up with far too high a quantizer.

Hmm just read a few posts up. How come this AQ improvement won't work on CRF?

[lexor]

Automatic thresholding, the default sensitivity option, helps keep the bitrate sane in CRF mode, but in 2pass you can do much better, and use a static sensitivity. Using --aq-sensitivity 30 or similar will result not only in flat areas of individual frames getting more bits, but also entire flat FRAMES getting more bits.
I am a bit confused by that wording. It seems to allow possibility of the entire movie's bitrate going up (if it decides all frames need tuning up, I know unlikely, but as a theoretic possibility). Does it? Dark, you said that that "new" image above is larger, but is it true for the entire movie, or have the bits simply been re-allocated between frames?

[Dark Shikari]

I am a bit confused by that wording. It seems to allow possibility of the entire movies bitrate going up (if it decides all frames need tuning up, I know unlikely, but as a theoretic possibility). Does it? Dark, you said that that "new" image above is larger, but is it true for the entire movie, or have the bit simply been re-allocated between frames?Static sensitivity allows bits to be re-allocated between frames, but gives no guarantee that the bitrate of the movie is anywhere close to that of what you would get without CRF.

It *works* on CRF, but it will not keep the bitrate anywhere near to the original (or at least, one cannot guarantee it).

[lexor]

Static sensitivity allows bits to be re-allocated between frames, but gives no guarantee that the bitrate of the movie is anywhere close to that of what you would get without CRF.

It *works* on CRF, but it will not keep the bitrate anywhere near to the original (or at least, one cannot guarantee it).

Eh, I'm not the guy concerned about CRF, I do 2pass encodes. :)

On that note, will MeGUI's bitrate calculator still be correct (or as correct as it is for official builds)? I mean can I still rely on getting a more or less certain file size out of a 2pass?

[Atak_Snajpera]

It *works* on CRF, but it will not keep the bitrate anywhere near to the original (or at least, one cannot guarantee it).

Hehe It sounds like old Haali's AQ :)

[DeathTheSheep]

Sounds like pretty much anything that redefines the allocation of bits between frames contrary to x264 defaults. :)

[Dark Shikari]

Eh, I'm not the guy concerned about CRF, I do 2pass encodes. :)

On that note, will MeGUI's bitrate calculator still be correct (or as correct as it is for official builds)? I mean can I still rely on getting a more or less certain file size out of a 2pass?Yes, bitrate mode in general (1pass or 2pass) should probably still be pretty accurate at this point.

Slightly less than normal, but not by much.
Sounds like pretty much anything that redefines the allocation of bits between frames contrary to x264 defaults. :)Yup.

It'll be interesting to see what Dakaz thinks of this mode ;)

[Atak_Snajpera]

Dakaz ? Do we know him? :/

[Dark Shikari]

Dakaz ? Do we know him? :/Many don't, but he and the rest of Avail Media's encoding division are behind quite a bit of x264, especially interlaced mode :p

[DeathTheSheep]

He also offered a bounty for people who could produce good, novel x264 code. (http://mailman.videolan.org/pipermail/x264-devel/2007-May/003055.html) And of course by bounty I mean...you know.
Now I just have to cross my fingers and hope the satd patch will come magically rolling in. The guts of this thing are truly, in lack of a better term, mathemagical. (Not to mention easy to break beyond repair, if you don't know what you're doing).

[ToS_Maverick]

Static sensitivity allows bits to be re-allocated between frames, but gives no guarantee that the bitrate of the movie is anywhere close to that of what you would get without CRF.

It *works* on CRF, but it will not keep the bitrate anywhere near to the original (or at least, one cannot guarantee it).

who cares? with Haali's AQ it was clear, that if you enable it, the bitrate with CRF will rise.

i consider it that way: std x264 is not at an optimum.

why do you want to stick to a same frame size? if a frame needs more bits to look good, so be it... as long as the ratecontrol has no problem with it ;)

[Dark Shikari]

who cares? with Haali's AQ it was clear, that if you enable it, the bitrate with CRF will rise.

i consider it that way: std x264 is not at an optimum.

why do you want to stick to a same frame size? if a frame needs more bits to look good, so be it... as long as the ratecontrol has no problem with it ;)Remember what happened with the previous AQ?

Sometimes, video size would rise drastically, other times it would drop drastically.

The point being, that if most of the bits in your video are in blocks that are above the variance threshold you set, your bitrate will drop--and if most of the bits are in blocks that are below, your bitrate will rise.

[slavickas]

He also offered a bounty for people who could produce good, novel x264 code. (http://mailman.videolan.org/pipermail/x264-devel/2007-May/003055.html) And of course by bounty I mean...you know
...
http://upload.wikimedia.org/wikipedia/commons/3/31/BountyBars.jpg
?
edit: my 264 post :D

[Dark Shikari]

I think he means:

http://ecx.images-amazon.com/images/I/51HpkVvqKwL.jpg

[ToS_Maverick]

wait a minute... with a too low --aq-sensitivity xx the bitrate could DROP? well... that's a fact that i didn't know.

my dream of a CRF mode would be, set it at 18 and your encode comes out transparent. if we could reach this goal, we would have a "lame -V2" H264 codec :D

[Dark Shikari]

wait a minute... with a too low --aq-sensitivity xx the bitrate could DROP? well... that's a fact that i didn't know.

my dream of a CRF mode would be, set it at 18 and your encode comes out transparent. if we could reach this goal, we would have a "lame -V2" H264 codec :DThis is because AQ sensitivity sets a midpoint variance, above which the QP is raised, and below which the QP is lowered.

Technically, one could apply the entire "automatic sensitivity" aspect of my algorithm to the ENTIRE VIDEO on the first pass, and then use that sensitivity on the second pass.

[ToS_Maverick]

is my thinking correct:
your auto sensitivity redistributes the bits (lowers/raises quants) from areas that are more complex to lower complex areas.
what's bothering me is, if there is a scene, like the last .gif you posted, where there are no bits to redistribute, what are you going to do?

is there a reliable metric (like ssim), to find out where the problematic parts are, and give them more bits?

of course we could do a lot of testing and tweaking, to find out a good sensitivity that we could use for our encodes, but that would be empirical, and will surely get rejected by aku for SVN...

[Dark Shikari]

is my thinking correct:
your auto sensitivity redistributes the bits (lowers/raises quants) from areas that are more complex to lower complex areas.
what's bothering me is, if there is a scene, like the last .gif you posted, where there are no bits to redistribute, what are you going to do?

is there a reliable metric (like ssim), to find out where the problematic parts are, and give them more bits?

of course we could do a lot of testing and tweaking, to find out a good sensitivity that we could use for our encodes, but that would be empirical, and will surely get rejected by aku for SVN...Sensitivity is arbitrary, in a sense; as long as you don't put a limiter on how QPs are changed.

No matter what sensitivity is chosen, the QPs chosen in a particular frame will be exactly the same, except scaled up or down by a constant value.

[DeathTheSheep]

Hm, it's odd how .45 behaves differently with the same commandline as .431 with the line removed. .431 made the file 4204KB, 0.9747881 SSIM, while your new patch with same threshold made it 4205KB, 0.9747586 SSIM. How did the result become slightly bigger and worse in SSIM, if just the threshold was removed? Maybe change in the code for interlace fixes?

[Dark Shikari]

Hm, it's odd how .45 behaves differently with the same commandline as .431 with the line removed. .431 made the file 4204KB, 0.9747881 SSIM, while your new patch with same threshold made it 4205KB, 0.9747586 SSIM. How did the result become slightly bigger and worse in SSIM, if just the threshold was removed? Maybe change in the code for interlace fixes?There was a slight change to fix a bug when variance was equal to zero.

[DeathTheSheep]

Note to self: you cannot say if(floating point value == 0) before calling x264_cpu_restore()... ;)
http://www.biffleys.com/Strategy/pictures/bounty.jpg

[desta]

Some test results, using AQ with varying thresholds.

--pass 2 --bitrate 4100 --stats "D:\TESTS\test.aq1.sens-auto.stats" --keyint 240 --min-keyint 24 --ref 7 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,-2 --subme 7 --trellis 1 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --merange 24 --threads auto --thread-input --progress --no-dct-decimate --output "D:\TESTS\test.aq1.sens-auto.mkv" "D:\TESTS\test.aq1.sens-auto.avs" --aq-strength 1

x264 [info]: slice I:19 Avg QP:17.37 size: 53151 PSNR Mean Y:50.05 U:50.90 V:52.20 Avg:49.86 Global:47.70
x264 [info]: slice P:782 Avg QP:19.27 size: 27715 PSNR Mean Y:45.76 U:48.71 V:49.06 Avg:46.53 Global:46.00
x264 [info]: slice B:519 Avg QP:20.61 size: 10718 PSNR Mean Y:45.11 U:48.61 V:49.06 Avg:45.90 Global:45.07
x264 [info]: mb I I16..4: 21.1% 67.2% 11.7%
x264 [info]: mb P I16..4: 10.4% 30.0% 4.6% P16..4: 24.2% 12.1% 3.2% 0.1% 0.0% skip:15.4%
x264 [info]: mb B I16..4: 2.3% 5.2% 0.9% B16..8: 32.6% 1.9% 4.3% direct: 5.0% skip:47.9%
x264 [info]: 8x8 transform intra:66.2% inter:57.1%
x264 [info]: direct mvs spatial:94.8% temporal:5.2%
x264 [info]: ref P 67.3% 14.7% 6.9% 3.8% 2.9% 2.5% 1.8%
x264 [info]: ref B 76.9% 11.4% 4.6% 2.7% 1.8% 1.5% 1.1%
x264 [info]: SSIM Mean Y:0.9831273
x264 [info]: PSNR Mean Y:45.567 U:48.706 V:49.105 Avg:46.331 Global:45.626 kb/s:4104.32
----------

the following tests use the same options as above, with the only difference being...

--aq-sensitivity 30

x264 [info]: slice I:19 Avg QP:26.11 size: 55517 PSNR Mean Y:49.61 U:50.68 V:51.93 Avg:49.46 Global:46.92
x264 [info]: slice P:782 Avg QP:29.36 size: 27478 PSNR Mean Y:44.90 U:48.13 V:48.45 Avg:45.72 Global:44.85
x264 [info]: slice B:519 Avg QP:30.41 size: 10933 PSNR Mean Y:44.33 U:48.02 V:48.45 Avg:45.17 Global:44.20
x264 [info]: mb I I16..4: 22.6% 63.8% 13.7%
x264 [info]: mb P I16..4: 10.2% 30.8% 5.2% P16..4: 24.9% 11.6% 2.9% 0.1% 0.0% skip:14.4%
x264 [info]: mb B I16..4: 2.7% 5.5% 1.0% B16..8: 33.4% 1.9% 4.4% direct: 4.9% skip:46.3%
x264 [info]: 8x8 transform intra:65.9% inter:54.0%
x264 [info]: direct mvs spatial:93.3% temporal:6.7%
x264 [info]: ref P 67.0% 14.6% 7.0% 3.8% 3.1% 2.6% 1.9%
x264 [info]: ref B 76.2% 11.6% 4.8% 2.8% 1.9% 1.6% 1.2%
x264 [info]: SSIM Mean Y:0.9822634
x264 [info]: PSNR Mean Y:44.746 U:48.122 V:48.500 Avg:45.558 Global:44.605 kb/s:4100.14
----------

--aq-sensitivity 25

x264 [info]: slice I:19 Avg QP:23.89 size: 56126 PSNR Mean Y:49.75 U:50.76 V:52.03 Avg:49.60 Global:47.08
x264 [info]: slice P:782 Avg QP:27.18 size: 27493 PSNR Mean Y:44.97 U:48.15 V:48.49 Avg:45.78 Global:44.93
x264 [info]: slice B:519 Avg QP:28.25 size: 10881 PSNR Mean Y:44.43 U:48.05 V:48.48 Avg:45.21 Global:44.26
x264 [info]: mb I I16..4: 24.6% 62.4% 13.0%
x264 [info]: mb P I16..4: 10.1% 30.9% 5.1% P16..4: 24.8% 11.7% 2.9% 0.1% 0.0% skip:14.4%
x264 [info]: mb B I16..4: 2.6% 5.6% 0.9% B16..8: 33.4% 1.8% 4.4% direct: 4.8% skip:46.4%
x264 [info]: 8x8 transform intra:66.2% inter:54.5%
x264 [info]: direct mvs spatial:93.4% temporal:6.6%
x264 [info]: ref P 67.1% 14.6% 7.0% 3.8% 3.1% 2.6% 1.9%
x264 [info]: ref B 75.9% 11.7% 4.8% 2.9% 1.9% 1.6% 1.2%
x264 [info]: SSIM Mean Y:0.9823568
x264 [info]: PSNR Mean Y:44.825 U:48.149 V:48.536 Avg:45.612 Global:44.677 kb/s:4099.58
----------

--aq-sensitivity 20

x264 [info]: slice I:19 Avg QP:21.63 size: 54153 PSNR Mean Y:49.61 U:50.66 V:51.92 Avg:49.46 Global:47.00
x264 [info]: slice P:782 Avg QP:24.51 size: 27609 PSNR Mean Y:45.04 U:48.20 V:48.53 Avg:45.85 Global:45.03
x264 [info]: slice B:519 Avg QP:25.63 size: 10797 PSNR Mean Y:44.47 U:48.09 V:48.52 Avg:45.27 Global:44.34
x264 [info]: mb I I16..4: 22.9% 64.3% 12.8%
x264 [info]: mb P I16..4: 10.2% 31.0% 5.0% P16..4: 24.8% 11.6% 2.9% 0.1% 0.0% skip:14.4%
x264 [info]: mb B I16..4: 2.6% 5.5% 0.9% B16..8: 33.4% 1.8% 4.4% direct: 4.8% skip:46.7%
x264 [info]: 8x8 transform intra:66.4% inter:55.0%
x264 [info]: direct mvs spatial:93.8% temporal:6.2%
x264 [info]: ref P 67.1% 14.6% 7.0% 3.8% 3.0% 2.6% 1.9%
x264 [info]: ref B 75.9% 11.6% 4.9% 2.8% 1.9% 1.6% 1.2%
x264 [info]: SSIM Mean Y:0.9824983
x264 [info]: PSNR Mean Y:44.883 U:48.192 V:48.573 Avg:45.673 Global:44.765 kb/s:4101.04
----------


Source:
- http://img413.imageshack.us/img413/2694/source66fo1.png
- http://img526.imageshack.us/img526/551/source165am9.png
- http://img526.imageshack.us/img526/1001/source358ij5.png
- http://img237.imageshack.us/img237/3574/source490ut0.png
- http://img242.imageshack.us/img242/9720/source615hn0.png

Auto Sensitivity:
- http://img248.imageshack.us/img248/2976/x264sensauto66iy7.png
- http://img248.imageshack.us/img248/4585/x264sensauto165en5.png
- http://img301.imageshack.us/img301/4273/x264sensauto358is1.png
- http://img99.imageshack.us/img99/6886/x264sensauto490uw8.png
- http://img99.imageshack.us/img99/8848/x264sensauto615el2.png

Sensitivity 30:
- http://img99.imageshack.us/img99/3128/x264sens3066bp5.png
- http://img412.imageshack.us/img412/5105/x264sens30165lw9.png
- http://img262.imageshack.us/img262/4485/x264sens30358rc9.png
- http://img262.imageshack.us/img262/9405/x264sens30490qd9.png
- http://img237.imageshack.us/img237/8185/x264sens30615yt0.png

Sensitivity 25:
- http://img149.imageshack.us/img149/8880/x264sens2566my7.png
- http://img101.imageshack.us/img101/5821/x264sens25165eb6.png
- http://img237.imageshack.us/img237/9567/x264sens25358un8.png
- http://img266.imageshack.us/img266/4593/x264sens25490ad9.png
- http://img101.imageshack.us/img101/6435/x264sens25615rr8.png

Sensitivity 20:
- http://img293.imageshack.us/img293/9208/x264sens2066za7.png
- http://img293.imageshack.us/img293/6128/x264sens20165hv3.png
- http://img293.imageshack.us/img293/7093/x264sens20358hq7.png
- http://img207.imageshack.us/img207/346/x264sens20490ko7.png
- http://img207.imageshack.us/img207/1899/x264sens20615dl4.png

[Dark Shikari]

Interesting results, but it seems there aren't any particularly flat scenes there to test with, as there with some of the earlier sources (where the static sensitivity method did much better).

[DeathTheSheep]

A few comments:
- Perhaps the true power of the AQRC is realized with QP encoding. When it's the only QP-altering force at the helm (rather than x264's rate control, which is shifting quantizers one way as the AQ shifts them the other).
- How about trying 25 instead of either 20 (too low) or 30 (too high, IMHO, at least for my sources/bitrates)?

[Dark Shikari]

A few comments:
- Perhaps the true power of the AQRC is realized with QP encoding. When it's the only QP-altering force at the helm (rather than x264's rate control, which is shifting quantizers one way as the AQ shifts them the other)The issue here is that QP mode doesn't take account into things like the fact that higher QPs can be used right before an I-frame.

[DeathTheSheep]

I'm not so sure that's such a good idea in the first place, especially when you have a lot of I-frames packed into the same place. I frame "flickering" and such become prevalent and distracting...I always hated that in videos. Always. Wonder why I use QP? :p And in anime, such a mechanism is inherently worthless, since any such QP hike is easily noticeable on the static backgrounds/choppy cartoon animations. The purpose of keyframe boost is to raise the quality on I-frames in proportion to surrounding frames (p and especially b), and with your patch, the "smearing" and detail loss (in the form of blurring, sometimes) doesn't occur anyway.
Egh. Maybe that's a rant or something, psht.

[akupenguin]

I'm not so sure that's such a good idea in the first place, especially when you have a lot of I-frames packed into the same place.
Then don't pack lots of I-frames in the same place. There can't be any I-frame flicker if I-frames appear only at scenecuts.
And in anime, such a mechanism is inherently worthless, since any such QP hike is easily noticeable on the static backgrounds/choppy cartoon animations.
Increasing the QP just before an I-frame does absolutely nothing to static backgrounds, because the background has no changes to code at any QP.

[desta]

Interesting results, but it seems there aren't any particularly flat scenes there to test with, as there with some of the earlier sources (where the static sensitivity method did much better).
Well that was primarily why I was testing really. As Sharktooth said earlier, your new AQ really does look like a good reason to completely ditch CQM's. I just wanted to see which sensitivity would yield better overall results, as people aren't typically always going to be encoding from flat/soft/low-detail sources.

- How about trying 25 instead of either 20 (too low) or 30 (too high, IMHO, at least for my sources/bitrates)?
Added test results.

[Sharktooth]

auto still looks better to me.

[Dark Shikari]

The question is, how can we get the benefit of auto while still having good quality in scenes like the two sample images I posted earlier, and the animated GIF?

I could have the ability for auto to limit how low its threshold goes, so as to add more bits to problematic frames... then we could ideally get the best of both worlds.

[DeathTheSheep]

Oops, I meant non-static backgrounds. :) To elaborate, imagine a clip in which the screen is moving (shaking, panning, etc) wherein an animated character does something or an exaggerated motion occurs that triggers a keyframe without actually undergoing a scene change. This is bad enough, but when some aspect of the frame is changing (like the characters move around on the static background), the moving part visibly loses quality against the static, unchanging background. This is an eyesore of sorts if it happens enough, which it seems invariably to do in some anime. And here tweaking the scene change threshold just makes things worse, extending other GOPs to huge numbers of frames and missing actual scene cuts.
If this is an inaccessible example, consider in crf mode when there is an extended sequence of fast but uniform motion (more than even that in CruNcher's clip), and keyframes are inserted during the motion...

Then don't pack lots of I-frames in the same place.
And though that's the goal, those --keyint 9999 users out there are pretty rare, and for good reason.

[Sharktooth]

The question is, how can we get the benefit of auto while still having good quality in scenes like the two sample images I posted earlier, and the animated GIF?

I could have the ability for auto to limit how low its threshold goes, so as to add more bits to problematic frames... then we could ideally get the best of both worlds.
dark masking?

[lexor]

auto still looks better to me.

Actually it seems form desta's images that auto does the best at foreground (faces, etc) and 30 does the best with the backgrounds. Now, if only we could get both... compute the difference between auto calculated value and 30, and bump auto value by half that distance towards 30 (unless it's already higher)? Talking nonsense here.

Of course the problem with those test images is the relatively high bitrate (4mbps), this new AQ would be put to a better test if it was a bit more bitrate starved.

[CruNcher]

People should note that the OLd AQ results http://forum.doom9.org/showthread.php?p=1089048#post1089048 look more visual pleasing in a ABR encode @ 3 Mbit to the eyes (fine edges) then those of the New AQ, as it looses quality there (hard edges) and makes it less watchable (limited or non limited makes no difference).
New AQ in ABR looks almost exactly the same (hard edges) as like in the CRF Encode Shoots (CRF 25).

[DeathTheSheep]

Actually, that might not really be nonsense... I was thinking...

How low can you go, with dark [Shikari] masking.

A best of both worlds approach figuring in automatic sensitivity while still freely giving more bits to frames that need them most, according to the threshold.

Maybe this can be combined with the two-pass mode you mentioned earlier--the first pass would decide on what general threshold to apply to the video in its entirety, and the 2nd pass can apply that threshold in general, yet have the freedom to shuffle bits around elsewhere as well, as would best benefit the video according to the established global threshold and whatever incidental threshold may prove favorable (predictive weighted average)?

[DeathTheSheep]

Wait a minute. When exactly did you address this variance == 0 bug? After 0.431?

[Dark Shikari]

Wait a minute. When exactly did you address this variance == 0 bug? After 0.431?0.45. I have no idea what happened beforehand, but now if variance is zero, it just uses the QP from the previous MB.

[DeathTheSheep]

Simply baffling. I suppose this is testament to the axiom that SSIM, bitrate and such are truly fickle entities indeed.

[Dark Shikari]

News of the day: Not using AQ on your first pass is extremely bad, especially if you're not using adaptive sensitivity.

In other news, water does seem to be somewhat wet and slippery.

[Razorholt]

Would it make sense to use AQ auto in first pass and AQ + sensitivity 30 in second pass - or the other way around?

[Dark Shikari]

Would it make sense to use AQ auto in first pass and AQ + sensitivity 30 in second pass - or the other way around?No, same sensitivity in both passes.

Also, I'd say 30 is probably too high, after some testing.

Maybe 20 or 25.

[Dark Shikari]

Conclusion after encoding a few dozen scenes of The Matrix: static sensitivity 20-25 or so is nearly always better than automatic, but one needs to use the same sensitivity on both passes.

[ChronoCross]

News of the day: Not using AQ on your first pass is extremely bad, especially if you're not using adaptive sensitivity.

In other news, water does seem to be somewhat wet and slippery.

I think the way you put it in the chat room was better.

<Dark_Shikari> also, it turns out
<Dark_Shikari> not using AQ on your first pass
<Dark_Shikari> = EPICFAILURE
<Dark_Shikari> don't do it
<Dark_Shikari> or you might be the next one to join the epic failtrain

[Dreassica]

Is there a way it doesn't work in megui?
I copied the patched x264.exe in the tolls dir, but it wont render anything but if I use the usual settings that do work with standard x264.exe.


This is log from megui:

[Information] Log for job1
-[NoImage] Job type: video
-[Information] [20-1-2008 13:30:29] Started handling job
-[Information] [20-1-2008 13:30:32] Preprocessing
-[NoImage] Job commandline: "G:\Program Files\megui\tools\x264\x264.exe" --qp 16 --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,1 --subme 7 --trellis 2 --analyse all --8x8dct --me esa --threads auto --thread-input --sar 1:1 --progress --no-dct-decimate --no-psnr --no-ssim --output "D:\sample.mkv" "D:\\sample.avs" --aq-strength=0.7
-[Information] [20-1-2008 13:30:34] Encoding started
-[Information] [20-1-2008 13:30:34] Job completed


Maybe I'm just stupid and missing something really obvious.

[desta]

Conclusion after encoding a few dozen scenes of The Matrix: static sensitivity 20-25 or so is nearly always better than automatic, but one needs to use the same sensitivity on both passes.
Just as a fyi, those tests I did earlier used the same AQ strength & sensitivity in both passes, and the tests with static sensitivity actually had the quants go up in the 2nd pass.

[fields_g]

Dreassica,

"--aq-strength=0.7" is wrong.
Get rid of the "=" sign.
"--aq-strength 0.7" is your answer.

[Dreassica]

Dreassica,

"--aq-strength=0.7" is wrong.
Get rid of the "=" sign.
"--aq-strength 0.7" is your answer.

Tried it, still doesn't start rendering. It's certainly not aq causing this, If I take it out from custom commandline in Megui, it still does this.

[Dark Shikari]

Just as a fyi, those tests I did earlier used the same AQ strength & sensitivity in both passes, and the tests with static sensitivity actually had the quants go up in the 2nd pass.This is because x264 measures only framewide quantizers, not block quantizers, and as such with adaptive quantization the quantizer readings can be extremely misleading.

With AQ, the average quantizers will generally be much lower than x264 displays.

[fields_g]

Dreassica,

Try using RAWAVC as your container. Also let us know if there is something weird in your "Standard Error Stream" in your log.

[salehin]

Here is a set comparisons.. I hope this helps. The source was BBC HD, Power of the Planet h264 1080p- encoded around 480 frames in 720p. Please note that the clip doesn't contain too many dark sequences.

If possible, please advise other possible ways of improving it.

aq_test with DarkShikari's v.45 aq 0.9, sens 20:

Avs Scipt

SetMemoryMax(128)
clip1=dgdecode_mpeg2source("J:\temp\crf test.d2v",info=3).ColorMatrix(hints=true,interlaced=true).tfm(order=0).tdecimate(hybrid=1).ConverttoYV12()

a=clip1.crop(2, 8, -2, -2)
clip2=DeGrainMedian(a, limitY=2,limitUV=3,mode=1)

GrainOptimizer(clip2).LimitedSharpenFaster(Strength=150).Spline36Resize(1280, 720)


Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 1 --bitrate 4800 --stats "K:\planet test\aq_test.stats" --bframes 3 --b-pyramid --direct auto --filter -3,-3 --subme 1 --analyse none --me dia --threads auto --thread-input --sar 1:1 --cqmfile "C:\Program Files\megui\Custom Matrices\Prestige.cfg" --progress --no-dct-decimate --output NUL "K:\planet test\aq_test.avs" --aq-strength 0.9 --aq-sensitivity 20
--[Information] [1/20/2008 11:52:46 AM] Encoding started
Standard output stream
Standard error stream
avis [info]: 1280x720 @ 25.00 fps (489 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 Cache64
x264 [info]: slice I:6 Avg QP:21.33 size:123021 PSNR Mean Y:46.63 U:48.01 V:49.72 Avg:47.22 Global:46.50
x264 [info]: slice P:265 Avg QP:25.22 size: 35899 PSNR Mean Y:40.57 U:46.54 V:47.86 Avg:41.86 Global:41.33
x264 [info]: slice B:218 Avg QP:26.15 size: 11726 PSNR Mean Y:40.46 U:45.89 V:47.69 Avg:41.70 Global:41.32
x264 [info]: mb I I16..4: 36.1% 0.0% 63.9%
x264 [info]: mb P I16..4: 28.7% 0.0% 0.0% P16..4: 62.6% 0.0% 0.0% 0.0% 0.0% skip: 8.7%
x264 [info]: mb B I16..4: 3.4% 0.0% 0.0% B16..8: 24.3% 0.0% 0.0% direct:35.0% skip:37.3%
x264 [info]: final ratefactor: 25.61
x264 [info]: direct mvs spatial:79.8% temporal:20.2%
x264 [info]: SSIM Mean Y:0.9598064
x264 [info]: PSNR Mean Y:40.597 U:46.270 V:47.804 Avg:41.855 Global:41.367 kb/s:5238.34
encoded 489 frames, 1.18 fps, 5238.76 kb/s

-[Information] Log for job36 (video, aq_test.avs -> aq_test with DS v.45 aq 0.9, sens 20.mkv)
--[Information] [1/20/2008 11:59:52 AM] Started handling job
--[Information] [1/20/2008 11:59:52 AM] Preprocessing
Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 2 --bitrate 4800 --stats "K:\planet test\aq_test.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,-3 --subme 7 --trellis 1 --analyse all --8x8dct --me umh --threads auto --thread-input --sar 1:1 --cqmfile "C:\Program Files\megui\Custom Matrices\Prestige.cfg" --progress --no-dct-decimate --output "K:\planet test\aq_test with DS v.45 aq 0.9, sens 20.mkv" "K:\planet test\aq_test.avs" --aq-strength 0.9 --aq-sensitivity 20
--[Information] [1/20/2008 11:59:59 AM] Encoding started
Standard output stream
Standard error stream
avis [info]: 1280x720 @ 25.00 fps (489 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 Cache64
x264 [info]: slice I:6 Avg QP:20.33 size:106716 PSNR Mean Y:46.13 U:48.86 V:50.38 Avg:46.97 Global:46.73
x264 [info]: slice P:265 Avg QP:22.55 size: 34281 PSNR Mean Y:41.89 U:47.37 V:48.66 Avg:43.13 Global:42.73
x264 [info]: slice B:218 Avg QP:23.57 size: 9752 PSNR Mean Y:41.32 U:46.51 V:48.34 Avg:42.54 Global:42.35
x264 [info]: mb I I16..4: 1.8% 93.3% 4.9%
x264 [info]: mb P I16..4: 0.1% 5.2% 0.2% P16..4: 54.9% 24.2% 9.7% 0.9% 0.5% skip: 4.2%
x264 [info]: mb B I16..4: 0.0% 0.4% 0.0% B16..8: 36.8% 1.2% 3.8% direct: 2.9% skip:54.9%
x264 [info]: 8x8 transform intra:93.5% inter:82.6%
x264 [info]: direct mvs spatial:80.3% temporal:19.7%
x264 [info]: ref P 69.0% 15.2% 7.6% 4.9% 3.4%
x264 [info]: ref B 83.4% 10.2% 2.9% 2.2% 1.3%
x264 [info]: SSIM Mean Y:0.9635958
x264 [info]: PSNR Mean Y:41.687 U:47.005 V:48.535 Avg:42.914 Global:42.586 kb/s:4846.87
encoded 489 frames, 0.41 fps, 4847.54 kb/s
--[Information] Final statistics
Desired video bitrate: 4800 kbit/s
Obtained video bitrate (approximate: 4850 kbit/s

oooooooooooooooo

aq_test with x264 r719 SVN aq 0.3.mkv

Applied patches (by techhouse):
x264_2pass_vbv.diff
x264_aq-brdo.diff
x264_faster-dia.diff
x264_fp-eta.01.r680.diff
x264_hrd_pulldown.diff
x264_me-prepass.diff
x264_satd_fpel.11.diff
x264_thread_pool.r680.diff

** note: it's not a cef built- i named the file incrrectly**
Avs Scipt: same

Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 1 --bitrate 4800 --stats "K:\planet test\aq_test.stats" --bframes 3 --b-pyramid --direct auto --filter -3,-3 --subme 1 --analyse none --me dia --threads auto --thread-input --sar 1:1 --cqmfile "C:\Program Files\megui\Custom Matrices\Prestige.cfg" --progress --no-dct-decimate --output NUL "K:\planet test\aq_test.avs" --aq-strength 0.3

--[Information] [1/20/2008 12:25:38 PM] Encoding started
Standard output stream
Standard error stream
avis [info]: 1280x720 @ 25.00 fps (489 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 Cache64
x264 [info]: slice I:6 Avg QP:21.00 size:121756 PSNR Mean Y:46.78 U:48.09 V:49.83 Avg:47.35 Global:46.77
x264 [info]: slice P:265 Avg QP:24.60 size: 36377 PSNR Mean Y:40.74 U:46.70 V:48.04 Avg:42.03 Global:41.48
x264 [info]: slice B:218 Avg QP:25.56 size: 12186 PSNR Mean Y:40.62 U:46.02 V:47.85 Avg:41.86 Global:41.50
x264 [info]: mb I I16..4: 39.0% 0.0% 61.0%
x264 [info]: mb P I16..4: 27.2% 0.0% 0.0% P16..4: 62.2% 0.0% 0.0% 0.0% 0.0% skip:10.6%
x264 [info]: mb B I16..4: 3.1% 0.0% 0.0% B16..8: 23.9% 0.0% 0.0% direct:36.1% skip:36.9%
x264 [info]: final ratefactor: 25.19
x264 [info]: direct mvs spatial:80.7% temporal:19.3%
x264 [info]: SSIM Mean Y:0.9606545
x264 [info]: PSNR Mean Y:40.765 U:46.414 V:47.976 Avg:42.022 Global:41.523 kb/s:5327.96
encoded 489 frames, 1.22 fps, 5328.38 kb/s
[1/20/2008 12:32:40 PM] Job completed
--[Information] [1/20/2008 12:32:40 PM] Postprocessing
---[Information] Deleting intermediate files
-[Information] Log for job37 (video, aq_test.avs -> aq_test with cef709 aq 0.3.mkv)
--[Information] [1/20/2008 12:32:40 PM] Started handling job
--[Information] [1/20/2008 12:32:40 PM] Preprocessing

Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 2 --bitrate 4800 --stats "K:\planet test\aq_test.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,-3 --subme 7 --trellis 1 --analyse all --8x8dct --me umh --threads auto --thread-input --sar 1:1 --cqmfile "C:\Program Files\megui\Custom Matrices\Prestige.cfg" --progress --no-dct-decimate --output "K:\planet test\aq_test with cef709 aq 0.3.mkv" "K:\planet test\aq_test.avs" --aq-strength 0.3
--[Information] [1/20/2008 12:32:56 PM] Encoding started
Standard output stream
Standard error stream
avis [info]: 1280x720 @ 25.00 fps (489 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 Cache64
x264 [info]: slice I:6 Avg QP:20.17 size:102501 PSNR Mean Y:46.06 U:48.76 V:50.33 Avg:46.90 Global:46.66
x264 [info]: slice P:265 Avg QP:22.35 size: 34217 PSNR Mean Y:41.94 U:47.40 V:48.69 Avg:43.18 Global:42.79
x264 [info]: slice B:218 Avg QP:23.34 size: 9897 PSNR Mean Y:41.38 U:46.53 V:48.35 Avg:42.60 Global:42.40
x264 [info]: mb I I16..4: 2.2% 93.1% 4.6%
x264 [info]: mb P I16..4: 0.1% 5.5% 0.2% P16..4: 53.4% 23.4% 9.9% 1.0% 0.5% skip: 5.9%
x264 [info]: mb B I16..4: 0.0% 0.4% 0.0% B16..8: 35.7% 1.2% 3.9% direct: 3.1% skip:55.8%
x264 [info]: 8x8 transform intra:94.1% inter:83.1%
x264 [info]: direct mvs spatial:83.0% temporal:17.0%
x264 [info]: ref P 68.9% 15.2% 7.6% 4.9% 3.4%
x264 [info]: ref B 83.5% 10.3% 2.8% 2.1% 1.2%
x264 [info]: SSIM Mean Y:0.9639119
[info]: PSNR Mean Y:41.741 U:47.029 V:48.558 Avg:42.964 Global:42.642 kb/s:4842.55
encoded 489 frames, 0.46 fps, 4843.22 kb/s
Final statistics
Desired video bitrate: 4800 kbit/s
Obtained video bitrate (approximate: 4846 kbit/s

[Dreassica]

Dreassica,

Try using RAWAVC as your container. Also let us know if there is something weird in your "Standard Error Stream" in your log.

No on both counts.

--crf 16 --ref 16 --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,1 --subme 7 --trellis 1 --analyse all --8x8dct --me esa --threads 8 --thread-input --progress --no-psnr --no-ssim --output "output" "input"

[Dark Shikari]

Unfortunately one cannot tell much from logs in this sort of situation; its really all about how it looks.

[Yoshiyuki Blade]

Tried it, still doesn't start rendering. It's certainly not aq causing this, If I take it out from custom commandline in Megui, it still does this.

As fields_g stated, select RAWAVC as your container (it will output as a .264 file). Also make sure pthreadGC2.dll is in the same folder as Dark's build of x264. If its not there, copy pthreadGC2.dll from the "ffmpeg" folder in the MeGUI directory, which is also in the "tools" folder like "x264"

I've been running a few anime encodes with the v0.45 and it seems great. At the same settings and bitrate with --aq-strength 1.0 --aq-sesitivity 25, there was much less blockyness with v0.45, although it seems to come at the cost of overall quality. On v0.44 the scenes look nicer, but it looks rather nasty in dark areas. Simply swapping out build v0.44 with v0.45 with the exact same configuration reduced blockyness by a lot, but the overall crispness is lost. Perhaps it needs more bitrate.

[Dreassica]

Pasting the commandline in a dos prompt gives me the error that pthreadGC2.dll can't be found. Odd as standard x264 seems to work fine.

EDIT

Downloaded said dll and put it in same dir as x264.exe, NOW it works finally.

[Dark Shikari]

Pasting the commandline in a dos prompt gives me the error that pthreadGC2.dll can't be found. Odd as standard x264 seems to work fine.That's because incompetent me can't seem to get a static build working. Get PthreadGC2.dll here (http://mirror05.x264.nl/Dark/force.php?file=./pthreadGC2.dll).

[Dreassica]

That's because incompetent me can't seem to get a static build working. Get PthreadGC2.dll here (http://mirror05.x264.nl/Dark/force.php?file=./pthreadGC2.dll).

Yea found that out on my own. Rendering finally now. :)

[Yoshiyuki Blade]

EDIT

Downloaded said dll and put it in same dir as x264.exe, NOW it works finally.

Great, now double-check and make sure RAWAVC is the output container instead of mp4, or itll error on the 2nd pass. I wasted several hours of testing by not doublechecking :D.

[salehin]

Unfortunately one cannot tell much from logs in this sort of situation; its really all about how it looks.
Here they are
With DarkShikari's v.45: aq_test with DS v.45 aq 0.9, sens 20.mkv (http://www.sendspace.com/file/vdx2rn)
*Without DarkShikari: aq_test with x264 r719 SVN aq 0.3.mkv (http://www.sendspace.com/file/rke9ij)

*Applied patches in the svn:


x264_2pass_vbv.diff
x264_aq-brdo.diff
x264_faster-dia.diff
x264_fp-eta.01.r680.diff
x264_hrd_pulldown.diff
x264_me-prepass.diff
x264_satd_fpel.11.diff
x264_thread_pool.r680.diff

[Dark Shikari]

Here they are
With DarkShikari's v.45: aq_test with DS v.45 aq 0.9, sens 20.mkv (http://www.sendspace.com/file/vdx2rn)
*Without DarkShikari: aq_test with x264 r719 SVN aq 0.3.mkv (http://www.sendspace.com/file/rke9ij)

*Applied patches in the svn:Looking at the quantizer distribution its clear neither of these actually used my AQ. This isn't surprising, given that "x264_aq-brdo.diff" isn't my patch, its Haali's.

[salehin]

eek!! i'll remove the svn from the x264 folder and post new link later

Update: Here is the proper one. Only your v.45 is in the MeGUIs x264 folder- renamed as x264.exe

(Proper) aq_test with DS v.45 aq 0.9, sens 20.mkv (http://www.sendspace.com/file/jyavnu)

Log

Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 1 --bitrate 4800 --stats "K:\planet test\aq_test.stats" --bframes 3 --b-pyramid --direct auto --filter -3,-3 --subme 1 --analyse none --me dia --threads auto --thread-input --sar 1:1 --cqmfile "C:\Program Files\megui\Custom Matrices\Prestige.cfg" --progress --no-dct-decimate --output NUL "K:\planet test\aq_test.avs" --aq-strength 0.9 --aq-sensitivity 20

[1/20/2008 2:37:33 PM] Encoding started
avis [info]: 1280x720 @ 25.00 fps (489 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 Cache64
x264 [info]: slice I:6 Avg QP:24.83 size:118889 PSNR Mean Y:45.43 U:47.34 V:49.11 Avg:46.12 Global:44.64
x264 [info]: slice P:270 Avg QP:29.72 size: 35574 PSNR Mean Y:39.36 U:45.62 V:46.89 Avg:40.67 Global:39.95
x264 [info]: slice B:213 Avg QP:30.15 size: 11561 PSNR Mean Y:39.46 U:44.99 V:46.81 Avg:40.70 Global:39.96
x264 [info]: mb I I16..4: 29.2% 0.0% 70.8%
x264 [info]: mb P I16..4: 29.3% 0.0% 0.0% P16..4: 62.8% 0.0% 0.0% 0.0% 0.0% skip: 7.9%
x264 [info]: mb B I16..4: 3.9% 0.0% 0.0% B16..8: 25.1% 0.0% 0.0% direct:33.7% skip:37.3%
x264 [info]: final ratefactor: 29.88
x264 [info]: direct mvs spatial:77.9% temporal:22.1%
x264 [info]: SSIM Mean Y:0.9580282
x264 [info]: PSNR Mean Y:39.480 U:45.366 V:46.880 Avg:40.751 Global:39.990 kb/s:5227.32
encoded 489 frames, 1.36 fps, 5226.36 kb/s
[1/20/2008 2:43:45 PM] Job completed


Log for job41 (video, aq_test.avs -> aq_test with DS v.45 aq 0.9, sens 20.mkv)
[1/20/2008 2:43:47 PM] Started handling job
[1/20/2008 2:43:47 PM] Preprocessing
Job commandline: "C:\Program Files\megui\tools\x264\x264.exe" --pass 2 --bitrate 4800 --stats "K:\planet test\aq_test.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,-3 --subme 7 --trellis 1 --analyse all --8x8dct --me umh --threads auto --thread-input --sar 1:1 --cqmfile "C:\Program Files\megui\Custom Matrices\Prestige.cfg" --progress --no-dct-decimate --output "K:\planet test\aq_test with DS v.45 aq 0.9, sens 20.mkv" "K:\planet test\aq_test.avs" --aq-strength 0.9 --aq-sensitivity 20
[1/20/2008 2:44:01 PM] Encoding started
avis [info]: 1280x720 @ 25.00 fps (489 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 Cache64
x264 [info]: slice I:6 Avg QP:23.67 size:105442 PSNR Mean Y:45.08 U:48.10 V:49.69 Avg:45.97 Global:45.58
x264 [info]: slice P:270 Avg QP:26.36 size: 33967 PSNR Mean Y:40.84 U:46.60 V:47.84 Avg:42.10 Global:41.61
x264 [info]: slice B:213 Avg QP:27.25 size: 9408 PSNR Mean Y:40.55 U:45.73 V:47.58 Avg:41.77 Global:41.46
x264 [info]: mb I I16..4: 1.9% 94.1% 4.0%
x264 [info]: mb P I16..4: 0.2% 6.6% 0.1% P16..4: 50.8% 28.8% 9.2% 0.3% 0.1% skip: 3.8%
x264 [info]: mb B I16..4: 0.0% 0.4% 0.0% B16..8: 39.1% 1.2% 4.0% direct: 3.1% skip:52.2%
x264 [info]: 8x8 transform intra:95.5% inter:81.8%
x264 [info]: direct mvs spatial:71.4% temporal:28.6%
x264 [info]: ref P 70.1% 14.4% 7.4% 4.8% 3.4%
x264 [info]: ref B 82.8% 10.4% 3.1% 2.3% 1.5%
x264 [info]: SSIM Mean Y:0.9639779
x264 [info]: PSNR Mean Y:40.766 U:46.242 V:47.749 Avg:42.004 Global:41.577 kb/s:4829.26
encoded 489 frames, 0.44 fps, 4828.30 kb/s
Final statistics
Desired video bitrate: 4800 kbit/s
Obtained video bitrate (approximate: 4831 kbit/s

[DeathTheSheep]

So you're saying you managed to successfully apply the satd and me-prepass patches to svn-r719? A lot of the variables in me.c don't even refer to the quite same things anymore... casting pointers into ints and whatnot, that's what the compiler complains of.

In other news, water does seem to be somewhat wet and slippery.
No, I don't believe you. (Why would you say that of all things?) This doesn't have anything to do with an "epic failtrain," does it? :p

[Sagekilla]

Well now, this is quite an oddity.. I tested --aq-strength 1 with sensitivity of 0 and 20 on a randomly picked scene. (Which also happened to be a very dark, flat scene.) To my surprise.. Sensitivity of 20 more than doubled the bitrate used under crf mode! And this is with the 0.45 AQ.

G:\Movies\Battlestar Galactica>x264_aq --keyint 1000 --crf 18 --ref 6 --mixed-re
fs --no-fast-pskip --bframes 16 --bime --weightb --b-pyramid --b-rdo --8x8dct --
subme 7 --me umh --trellis 1 --aq-strength 1 --aq-sensitivity 20 --threads auto
--thread-input --progress --output "video.264" --pass 1 --stats "stats.log" "sou
rce.avs"
avis [info]: 864x480 @ 23.98 fps (720 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 3DNow!
x264 [info]: slice I:3 Avg QP:14.67 size: 61484 PSNR Mean Y:50.83 U:52.00
V:53.61 Avg:51.36 Global:51.34
x264 [info]: slice P:351 Avg QP:17.30 size: 28768 PSNR Mean Y:49.25 U:50.53
V:52.70 Avg:49.85 Global:49.71
x264 [info]: slice B:366 Avg QP:19.85 size: 9467 PSNR Mean Y:47.62 U:49.54
V:51.63 Avg:48.35 Global:48.30
x264 [info]: mb I I16..4: 21.5% 69.4% 9.1%
x264 [info]: mb P I16..4: 4.7% 12.5% 2.2% P16..4: 35.2% 27.5% 17.0% 0.0% 0
.0% skip: 0.9%
x264 [info]: mb B I16..4: 0.5% 1.4% 0.5% B16..8: 33.8% 3.8% 10.1% direct:
5.8% skip:44.1%
x264 [info]: 8x8 transform intra:64.0% inter:47.0%
x264 [info]: ref P 73.5% 12.2% 5.6% 3.5% 2.6% 2.6%
x264 [info]: ref B 88.4% 7.9% 1.6% 0.9% 0.6% 0.6%
x264 [info]: SSIM Mean Y:0.9915203
x264 [info]: PSNR Mean Y:48.426 U:50.032 V:52.158 Avg:49.096 Global:48.940 kb/s:
3662.19

encoded 720 frames, 2.98 fps, 3661.24 kb/s


G:\Movies\Battlestar Galactica>x264_aq --keyint 1000 --crf 18 --ref 6 --mixed-re
fs --no-fast-pskip --bframes 16 --bime --weightb --b-pyramid --b-rdo --8x8dct --
subme 7 --me umh --trellis 1 --aq-strength 1 --aq-sensitivity 0 --threads auto -
-thread-input --progress --output "video2.264" --pass 1 --stats "stats.log" "sou
rce.avs"
avis [info]: 864x480 @ 23.98 fps (720 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 3DNow!
x264 [info]: slice I:3 Avg QP:14.67 size: 45170 PSNR Mean Y:49.09 U:50.68
V:52.41 Avg:49.72 Global:49.71
x264 [info]: slice P:351 Avg QP:17.30 size: 11716 PSNR Mean Y:47.14 U:49.62
V:51.95 Avg:47.99 Global:47.89
x264 [info]: slice B:366 Avg QP:19.85 size: 3612 PSNR Mean Y:45.90 U:48.76
V:50.91 Avg:46.80 Global:46.77
x264 [info]: mb I I16..4: 23.0% 68.2% 8.8%
x264 [info]: mb P I16..4: 5.4% 7.5% 2.4% P16..4: 51.1% 20.4% 7.8% 0.0% 0
.0% skip: 5.4%
x264 [info]: mb B I16..4: 0.5% 0.9% 0.2% B16..8: 39.4% 1.4% 3.3% direct:
1.8% skip:52.5%
x264 [info]: 8x8 transform intra:50.6% inter:65.8%
x264 [info]: ref P 69.6% 14.1% 6.9% 3.8% 2.9% 2.7%
x264 [info]: ref B 86.2% 8.4% 2.4% 1.2% 1.0% 0.8%
x264 [info]: SSIM Mean Y:0.9874901
x264 [info]: PSNR Mean Y:46.517 U:49.186 V:51.424 Avg:47.392 Global:47.289 kb/s:
1483.77

encoded 720 frames, 3.59 fps, 1482.82 kb/s


Any thoughts on this?

[salehin]

So you're saying you managed to successfully apply the satd and me-prepass patches to svn-r719? A lot of the variables in me.c don't even refer to the quite same things anymore... casting pointers into ints and whatnot, that's what the compiler complains of.


No, I don't believe you. (Why would you say that of all things?) :p

Sorry DtS- i don't know what or how to reply, as I'm no expert into this sort of thing. I just got that patched svn from another place. The accompanied text file mentions that those patches are included in that x264

techhouse's x264 (http://x264.tk/)

[DeathTheSheep]

Ah, I see. Then would you direct me to this place of mystery? :) Thanks.

Sagekilla: Yes, raising the sensitivity tends to increase the bitrate quite a bit. Try sensitivity 30 and be shocked. (EPICSHOCK.) Lowering the sensitivity (not 0, that's automatic I presume) can actually lower the bitrate significantly, because the threshold acts like a centerpoint--if the input is above this centerpoint, it gets more data data, and vise versa.

[bob0r]

If you want to test AQ, i guess you should use x264 svn + AQ patch only:
x264.721.dark.aq.rdrc.0.45.exe (http://files.x264.nl/AQ/x264.721.dark.aq.rdrc.0.45.exe) (pthreads/mp4 = yes, made with make fprofiled)

[DeathTheSheep]

You said it, bob0r. Most of the patches in that build were broken anyway--even calling fpel cmp satd would crash the program, and for good reason. The old patch would increment the memory addresses of some of the variables in the new revision, obviously leading to a crash even before the first keyframe could be composed. And it's no easy fix--the mechanism has been practically redesigned, not to mention rearranged. I'm here with 681, custom built, march core2, cyborg gcc 4.3 [rev6], custom makefprofiled commandlines, tweaked function grouping priorities, etc, which is still pretty darn fast and works like a charm, save for a non-determinism or two with MT.

But like I've indicated earlier, combining some of the old patches with this new AQ leads to a synergistic effect, yielding a significantly greater quality increase than any used separately.

[Sagekilla]

Dark Shikari, I can confirm that your new AQ does work well on animated video. I'm encoding Dark Fury right now and the quality is very good across clip so far.

[Atak_Snajpera]

I've converted 1080p anime and it looks very well. Size even lower than without AQ

[jvt40]

Both clips looked fine (watching from bit outdated 17" screen :p) , difference running them side by side was Proper DS was slightly warmer (more red!!) , could be seen on jungle colour , sky and mountains , which is more truthful is matter of opinion .

eek!! i'll remove the svn from the x264 folder and post new link later

Update: Here is the proper one. Only your v.45 is in the MeGUIs x264 folder- renamed as x264.exe

(Proper) aq_test with DS v.45 aq 0.9, sens 20.mkv (http://www.sendspace.com/file/jyavnu)

[Dark Shikari]

Well now, this is quite an oddity.. I tested --aq-strength 1 with sensitivity of 0 and 20 on a randomly picked scene. (Which also happened to be a very dark, flat scene.) To my surprise.. Sensitivity of 20 more than doubled the bitrate used under crf mode! And this is with the 0.45 AQ.This is why I said nonzero sensitivity isn't for CRF mode, because it doesn't maintain bitrate.

[Morte66]

This is why I said nonzero sensitivity isn't for CRF mode, because it doesn't maintain bitrate.

But that's much the same as the AQ in the current x264 mainstream build. --crf 18 --aq-strength 0.3 --aq-sensitivity 5 gives something like double the bitrate of plain crf 18, depending on the material, and it's about enough to control blocking/banding on a revealing display.

Nobody knows what bitrate is going to come out of crf anyhow, that's the point of it. You encode to a target quality, not a target size. AQ changes the definition of "quality" from PSNR to something more concerned with subjective perception of smooth areas.

[CruNcher]

Exactly that's also my fear as my tests show exactly this behaveiour, no doub't it has much better detail preservation @ low bitrates, but it comes at a price for more unbalanced quality between the whole final thing. I think exactly that's what the goal should be with a AQ "balanced percepted visual quality". Under normal viewing conditions for this lossy stuff it's hard anyway to see slight blocking in motion, but banding is imidietly visible under any viewing condition. Sure you can try to use a Higher Compression factor but this again comes most of the times @ a heavy speed loss (Encoding/Decoding).

[Dark Shikari]

Exactly that's also my fear as my tests show exactly this behaveiour, no doub't it has much better detail preservation @ low bitrates, but it comes at a price for more unbalanced quality between the whole final thing. I think exactly that's what the goal should be with a AQ "balanced percepted visual quality". Under normal viewing conditions for this lossy stuff it's hard anyway to see slight blocking in motion, but banding is imidietly visible under any viewing condition. Sure you can try to use a Higher Compression factor but this again comes most of the times @ a heavy speed loss (Encoding/Decoding).Wait, you're criticizing my AQ for banding, even though banding is what it handles far better than any previous algorithm?

:rolleyes:

This is getting pointlessly retarded. Its as if you ran out of valid criticisms for my AQ (of which there are many), because your comments seem just completely nonsensical. Lowering quantizers in areas with banding problems is not going to make the banding worse!

[CruNcher]

http://forum.doom9.org/showthread.php?p=1089048#post1089048

look @ those scenes in the samples you've got and tell me what your New AQ does wrong then :)
Possible that i visually interpreting those results wrong as banding but they look like as this for me because of the fine gradients of noise that suddenly become blocks

[Dark Shikari]

http://forum.doom9.org/showthread.php?p=1089048#post1089048

look @ those scenes in the samples you've got and tell me what your New AQ does wrong then :)
Possible that i visually interpreting those results wrong as banding but they look like as this for me because of the fine gradients of noise that suddenly become blocksThis is because you used automatic sensitivity, which intentionally doesn't allow bits to be moved between frames. That means if a frame does genuinely need more bits, and you can't just move the bits from elsewhere in the frame to solve the quality problem, it won't get fixed.

Use a static sensitivity with twopass, and try again.

[CruNcher]

But i don't want to use twopass, because of the Speed reason i mentioned above :) and i tried every combination possible with ABR and your New AQ and it results allways in those Visual results it doesn't change for this particular spots (and those are really nasty they ruin the complete visual interpretation of this subjectively).
Psy optimizations aren't about Metrics or that you have super duper detail preservation in one part of the final result, they are about the balanced Look & Feel of the whole thing. That's why im also not really impressed @ the moment with the Parkrun results seeing this now, i mean it should be consistent improvement and not only for 1 spot and such a scene as Parkrun is a very special thing because also it's only 1 part scene and not 1 part of something much bigger around it :)

[Dark Shikari]

But i don't want to use twopass, because of the Speed reason i mentioned above :) and i tried every combination possible with ABR and your New AQ and it results allways in those Visual results it doesn't change for this particular spots.Then use CRF with static sensitivity. Just remember the filesize isn't guaranteed to stay where you want it--much like Haali's AQ, of course.

Try sensitivity 20 if you want to try to stay near the original filesize.

[Inventive Software]

Remind me... what's RCRD? And how much of a quality benefit does SATD with this patch give over a vanilla build?

[Dark Shikari]

Remind me... what's RCRD? And how much of a quality benefit does SATD with this patch give over a vanilla build?RCRD is a ratecontrol method that finds the optimal quantizer distribution, RD-wise.

SATD patch hasn't been updated to be compatible with r717+ yet. Benefit would be the same as before.

[Inventive Software]

Thanks for the info clear-up. :)

[Dark Shikari]

Link to 1080p sample encoded with the new AQ added to the original post.

[DeathTheSheep]

Could you also upload an RCRD sample? I'm sure people would be more eager to try it out if they see some results justifying the speed loss. After all, a lot of work and #x264dev chat has apparently gone into it.

In a few days' time, we'll have working satd too, and with very well-tweaked thresholds, might I add. Or shouldn't I spoil the surprise? One can hope prepass will be tackled by someone, too (heck, if nobody's up to fixing prepass by that time, I'll look into it myself, though whether or not I can possibly whip that on to x264's new code I'm obviously quite unsure about...let's hope the problem isn't too deeply rooted). This will give the community a much better reason to use RDRC--an insane option among insane options--since they'll have others to stack on top of it to mitigate their fears.

[Dark Shikari]

Could you also upload an RCRD sample? I'm sure people would be more eager to try it out if they see some results justifying the speed loss. After all, a lot of work and #x264dev chat has apparently gone into it.

In a few days' time, we'll have working satd too, and with very well-tweaked thresholds, might I add. Or shouldn't I spoil the surprise? One can hope prepass will be tackled by someone, too (heck, if nobody's up to fixing prepass by that time, I'll look into it myself, though whether or not I can possibly whip that on to x264's new code I'm obviously quite unsure about...let's hope the problem isn't too deeply rooted). This will give the community a much better reason to use RDRC--an insane option among insane options--since they'll have others to stack on top of it to mitigate their fears.RDRC sample (http://momupload.com/files/69785/touhou.mp4.html) with AQ.

[DeathTheSheep]

MOMupload? MOM?! Boy oh boy and by golly george, there's a mom upload now of all things? And what's more, I see you used them over mediafire...ouch. :\

What, a 100MB upload limit to the former?

[Dark Shikari]

MOMupload? MOM?! Boy oh boy and by golly george, there's a mom upload now of all things? And what's more, I see you used them over mediafire...ouch. :\

What, a 100MB upload limit to the former?Correct, size limit :p

[DeathTheSheep]

Well, 180KB/sec, that's nice. (Results after the break, ladies and gentlemen!) ;)

[edit]
Oh my god, is that you playing the game? I watched it two and half times already... This is MAD SKILLZ to the max. Tell me it's not you playing the game...tell me...

As for RCRD, this seems like an inordinately complex source, and all that rapid motion (and R4D D061NG sk111z) were handled...erm, well. But since I haven't made any such video game encodes before (or even seen many of them), I'm not quite sure what to expect here. Maybe a test run using CruNcher's infamous test clip is in order, eh?

But seriously. This game is 73H 1337 H4x0rz. And the soundtrack kicks Major General Asce.

[CruNcher]

Hmm could you guys please post wich of these looks better to you on the first sight and then also your GFX Card, Monitor ,Operating System, Player Application,Decoder and used Renderer as information. Thx in Advance :)

Please don't look framewise just subjectively rate them on the first sight :)

http://mirror05.x264.nl/CruNcher/force.php?file=./1.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./2.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./3.mkv

Btw: If you want to see allways correct representation results in Windows with a Directshow Player use CoreAVC as Decoder :)

[Dark Shikari]

Oh my god, is that you playing the game? I watched it two and half times already... This is MAD SKILLZ to the max. Tell me it's not you playing the game...tell me...Hell no, I got that off Stage6 (download link, for your own test encodes--source = 12.5 megabit DivX (http://www.stage6.com/user/SeaWeeze/video/1652397/Touhou-10-Mountain-of-Faith---Extra-Reimu-B)). I can't even beat Perfect Cherry Blossom on Normal difficulty let alone Ludicrous or Extra!

Note that its not as bad as it looks, as the hitbox in the game (like all similar games in the genre) is extremely small.
As for RCRD, this seems like an inordinately complex source, and all that rapid motion (and R4D D061NG sk111z) were handled...erm, well. But since I haven't made any such video game encodes before (or even seen many of them), I'm not quite sure what to expect here. Maybe a test run using CruNcher's infamous test clip is in order, eh?

But seriously. This game is 73H 1337 H4x0rz. And the soundtrack kicks Major General Asce.The game series is called Touhou Project, and you can get it (with the English patch) and play it on non-insane difficulty :p There's 10 games in the series now, Perfect Cherry Blossom is the easiest to find of them.

I love the Touhou clip because its absolute murder for any video encoder.

The main benefit of AQ in that clip is in the backgrounds (which were blurred to death without it). The main benefit of RCRD was more intelligent I/B-frame offset decision.

[CruNcher]

Holly Bible :D
http://s3.directupload.net/images/080123/qon9i85i.png <- oldaq
http://s6.directupload.net/images/080123/bzj6hm86.png <- newaq

There is still Darkness also with it (in the real meaning) :) (in motion it can hurt sometimes especialy if it's directly in the ROI as here)
http://s5.directupload.net/images/080123/4fpl8zs5.png <- oldaq
http://s5.directupload.net/images/080123/omgt9f7o.png <- newaq

When this blocking problems gonna get fixed i jump around all night and day (when it's possible to balance it out efficiently without destroying it's purpose (takeing bits from one area allocateing to another)) :D
http://s5.directupload.net/images/080123/hgbfhnta.png <- oldaq
http://s1.directupload.net/images/080123/jnbkrecp.png <- newaq

Let's love it again ;)
http://s5.directupload.net/images/080123/m5aava2r.png <- oldaq
http://s5.directupload.net/images/080123/xh6fd92z.png <- newaq

Old AQ ( --aq-strength 1.0 --aq-sensitivity 15)

x264 [info]: final ratefactor: 32.15
x264 [info]: 8x8 transform intra:19.1% inter:49.5%
x264 [info]: direct mvs spatial:98.5% temporal:1.5%
x264 [info]: ref P 82.8% 11.2% 6.0%
x264 [info]: ref B 91.0% 9.0%
x264 [info]: SSIM Mean Y:0.9780284
x264 [info]: PSNR Mean Y:44.630 U:45.375 V:48.696 Avg:45.190 Global:44.796 kb/s:
2944.99

encoded 9336 frames, 8.66 fps, 2952.16 kb/s

New AQ (--aq-strength 1.0 --aq-sensitivity 20)

x264 [info]: final ratefactor: 34.11
x264 [info]: 8x8 transform intra:20.9% inter:50.6%
x264 [info]: direct mvs spatial:98.2% temporal:1.8%
x264 [info]: ref P 84.3% 10.2% 5.6%
x264 [info]: ref B 91.8% 8.2%
x264 [info]: SSIM Mean Y:0.9787061
x264 [info]: PSNR Mean Y:43.929 U:45.177 V:48.162 Avg:44.574 Global:43.988 kb/s:
3014.86

encoded 9336 frames, 8.53 fps, 3022.31 kb/s

New AQ (--aq-strength 0.5 --aq-sensitivity 20)

x264 [info]: final ratefactor: 30.68
x264 [info]: 8x8 transform intra:18.1% inter:51.1%
x264 [info]: direct mvs spatial:98.7% temporal:1.3%
x264 [info]: ref P 84.3% 10.1% 5.6%
x264 [info]: ref B 93.1% 6.9%
x264 [info]: SSIM Mean Y:0.9802665
x264 [info]: PSNR Mean Y:45.205 U:45.745 V:49.185 Avg:45.725 Global:45.209 kb/s:
3020.03

encoded 9336 frames, 8.72 fps, 3027.55 kb/s

New AQ (--aq-strength 0.1 --aq-sensitivity 20)

x264 [info]: final ratefactor: 28.51
x264 [info]: 8x8 transform intra:15.4% inter:51.4%
x264 [info]: direct mvs spatial:98.9% temporal:1.1%
x264 [info]: ref P 83.8% 10.4% 5.8%
x264 [info]: ref B 93.8% 6.2%
x264 [info]: SSIM Mean Y:0.9804187
x264 [info]: PSNR Mean Y:45.724 U:45.997 V:49.670 Avg:46.197 Global:45.673 kb/s:
3023.08

encoded 9336 frames, 8.72 fps, 3030.47 kb/s

New AQ (--aq-strength 0.1)

x264 [info]: final ratefactor: 28.40
x264 [info]: 8x8 transform intra:15.4% inter:51.4%
x264 [info]: direct mvs spatial:98.9% temporal:1.1%
x264 [info]: ref P 83.8% 10.4% 5.8%
x264 [info]: ref B 93.8% 6.2%
x264 [info]: SSIM Mean Y:0.9804481
x264 [info]: PSNR Mean Y:45.729 U:45.999 V:49.670 Avg:46.200 Global:45.682 kb/s:
3026.98

encoded 9336 frames, 8.67 fps, 3034.38 kb/s

New AQ (--strength 0.5)

x264 [info]: final ratefactor: 28.24
x264 [info]: 8x8 transform intra:16.8% inter:51.3%
x264 [info]: direct mvs spatial:98.7% temporal:1.3%
x264 [info]: ref P 84.1% 10.2% 5.7%
x264 [info]: ref B 93.4% 6.6%
x264 [info]: SSIM Mean Y:0.9804409
x264 [info]: PSNR Mean Y:45.506 U:45.891 V:49.456 Avg:45.998 Global:45.485 kb/s:
3033.67

encoded 9336 frames, 8.64 fps, 3041.13 kb/s

New AQ (--strength 1.0)

x264 [info]: final ratefactor: 28.58
x264 [info]: 8x8 transform intra:19.0% inter:50.8%
x264 [info]: direct mvs spatial:98.5% temporal:1.5%
x264 [info]: ref P 84.2% 10.1% 5.6%
x264 [info]: ref B 92.6% 7.4%
x264 [info]: SSIM Mean Y:0.9798093
x264 [info]: PSNR Mean Y:44.864 U:45.574 V:48.843 Avg:45.410 Global:44.888 kb/s:
3044.61

encoded 9336 frames, 8.60 fps, 3052.14 kb/s

All in all this is unbeliveable Detail Preservation (my wildest dreams since Grasprite come true) (ignoring it's side effects, for sure also partly coused by the insane bitrate) :)

[nm]

Hmm could you guys please post wich of these looks better to you on the first sight and then also your GFX Card, Monitor ,Operating System, Player Application,Decoder and used Renderer as information. Thx in Advance :)

Please don't look framewise just subjectively rate them on the first sight :)

http://mirror05.x264.nl/CruNcher/force.php?file=./1.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./2.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./3.mkv

Order from best to worst: 1, 2, 3
The first sample is noticeably better than the others. I saw difference between 2 and 3 only by scaling up the video to see the details.

Gfx card: NVIDIA GeForce4 Ti4200
Monitor: 21" CRT (resolution 1600x1200)
OS: Linux (NVIDIA binary blob X.org drivers)
Player/Decoder: MPlayer/libavcodec
Renderer: OpenGL output (same results with XVideo)

[Boardlord]

Hi all!

Has anyone tried the AQ on a concert footage? I tried to backup David Gilmour's "Remember that night" concert. I thought it would be a good test for Dark's AQ. I am sorry in advance that I have no pictures, but I am writing from my workplace, as I won't have net at home till March.
On the problems: the picture looked good, *very* little blocking, I was amazed! But, there were randomly appearing little black dot-like artifacts in the picture for a few secs, which then vanished, reappeared and so on. Tried it with Trellis, on-off (although Dark said Trellis doesn't matter). Aq strength was 0.8, sensitivity was tried at auto and 20 - same result. I used bob0r's compile. Setting weren't extreme (ref:5, bframes:3 (spatial), subme: 6, threads: 3, b-pyramid, no fast pskip, the rest were megui defaults)If needed, I'll post some pictures tomorrow, along with an x264 log. Of course the black dots were not present when AQ was not used... but blocks were :(

[salehin]

Hi all!

Has anyone tried the AQ on a concert footage? I tried to backup David Gilmour's "Remember that night" concert. I thought it would be a good test for Dark's AQ. I am sorry in advance that I have no pictures, but I am writing from my workplace, as I won't have net at home till March.
On the problems: the picture looked good, *very* little blocking, I was amazed! But, there were randomly appearing little black dot-like artifacts in the picture for a few secs, which then vanished, reappeared and so on. Tried it with Trellis, on-off (although Dark said Trellis doesn't matter). Aq strength was 0.8, sensitivity was tried at auto and 20 - same result. I used bob0r's compile. Setting weren't extreme (ref:5, bframes:3 (spatial), subme: 6, threads: 3, b-pyramid, no fast pskip, the rest were megui defaults)If needed, I'll post some pictures tomorrow, along with an x264 log. Of course the black dots were not present when AQ was not used... but blocks were :(

Try with bobr's again by using weak aq (not more than 0.50) and sensitivity of 20-25. Compare to Dark's build with aq 1.0 and sens .25 (posted in the 1st post), it gave me better results. However, note that my source contains very little dark sequence (see sample tests with other settings- links posted above)

update: perhaps you can use one of the CQMs (see the custom matrix bit (http://forum.doom9.org/showpost.php?p=1056555&postcount=3)) to see if it improves the transparency

[Morte66]

@DS

I thought you might be interested in this clip from 3 Colours White (http://www.mediafire.com/?dm1gmc1d2fl) (ffv1 in avi). It's been cropped/levelled/deblocked/denoised/debanded from DVD. It's my "worst reasonable case for AQ" material. I expect an encoder to handle it gracefully, if it's to get used for sight-unseen DVD backup.

It's assumed that it will be played back with Deband or GradFunkMirror() in ffdshow, which means it just has to avoid aggregating small blocks into bigger blocks.

[Dark Shikari]

@DS

I thought you might be interested in this clip from 3 Colours White (http://www.mediafire.com/?dm1gmc1d2fl) (ffv1 in avi). It's been cropped/levelled/deblocked/denoised/debanded from DVD. It's my "worst reasonable case for AQ" material. I expect an encoder to handle it gracefully, if it's to get used for sight-unseen DVD backup.

It's assumed that it will be played back with Deband or GradFunkMirror() in ffdshow, which means it just has to avoid aggregating small blocks into bigger blocks.
That was easy (http://www.mediafire.com/?5d1fywox22w). Just 600 kilobits, CRF mode, was plenty.

On this topic, it seems CRF mode seems to work fine with AQ if you use a reasonable sensitivity (20).

[Blue_MiSfit]

Hmm could you guys please post wich of these looks better to you on the first sight and then also your GFX Card, Monitor ,Operating System, Player Application,Decoder and used Renderer as information. Thx in Advance :)

Please don't look framewise just subjectively rate them on the first sight :)

http://mirror05.x264.nl/CruNcher/force.php?file=./1.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./2.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./3.mkv

Btw: If you want to see allways correct representation results in Windows with a Directshow Player use CoreAVC as Decoder :)


I like them in order. 1 is best followed by 2 and 3. Double checked.

8800GT, BenQ 24" LCD, Windows XP, Media Player Classic, CoreAVC, Haali Renderer.

Did the comparison at full screen (Scaled to 1920x1200). While 2 and 3 seemed to have more details in high frequency areas, the flat areas lost detail, and had irregular noise.

1 seemed to have the most even picture, and if there was any detail loss in the high frequency areas, it didn't distract me at all. The loss of detail in flat areas in the other versions WAS very distracting.

Excellent work! Let's see those settings!

~MiSfit

[ToS_Maverick]

That was easy (http://www.mediafire.com/?5d1fywox22w). Just 600 kilobits, CRF mode, was plenty.

On this topic, it seems CRF mode seems to work fine with AQ if you use a reasonable sensitivity (20).

i can second that, i use --aq-strength 1.0 --aq-sensitivity 15 in CRF mode, works great!

[DeathTheSheep]

15? Didn't he say that was too low? If 20-25 works better, with bias towards the lower end, wouldn't that imply 21-22?

Just beating some weighted averages around. Hoohoo!

[Dark Shikari]

15? Didn't he say that was too low? If 20-25 works better, with bias towards the lower end, wouldn't that imply 21-22?

Just beating some weighted averages around. Hoohoo!I have no idea. You'll have to test.

Ideally, a good "middle value" would result in true Constant Quality, since it would appropriately adjust CRF based on the variance of the video, which defines what bitrate it "really" needs. Technically any decent value of sensitivity would do this--its just that the bitrate would not necessarily be comparable to non-AQ CRF.

[Atak_Snajpera]

http://mirror05.x264.nl/CruNcher/force.php?file=./1.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./2.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./3.mkv

1 looks a lot better than 2 or 3.

MPC + FFDShow HQ-RGB32 forced

i can second that, i use --aq-strength 1.0 --aq-sensitivity 15 in CRF mode, works great!
I fully agree with you. 1.0 and 15 seems to be well balanced. However I still prefer auto sensitivity. Better quality (compared to no AQ) plus smaller files in CRF.

[Sagekilla]

Dark Shikari, would it be possible to hack in AQ to be used for RC under CRF mode? (Acronym city!) I'm sure if you did that, you could get even closer to a truer "constant quality"

[Dark Shikari]

Dark Shikari, would it be possible to hack in AQ to be used for RC under CRF mode? (Acronym city!) I'm sure if you did that, you could get even closer to a truer "constant quality"Actually, thinking about it... it should be even better than CRF mode, because it properly takes into account the quantizer needed to reach a constant quality.

Try sensitivity 20 on CRF mode.

[lexor]

hey guys, is anyone else having problems getting the files from x264.nl? I can't get there for a few days now. I thought there was a site hiccup, but seeing how a new video was added there (the one in OP) not everyone is having these issues. Or is it intermittent and I always try it when it's down?

[DeathTheSheep]

Wouldn't a "truer" constant quality be reached using QP and threshold? (To answer my own question, yeah, pretty much. No qcomp hiccups to worry about here, or artificial degradation near keyframes -.-).
:)

[CruNcher]

@Dark Shikari
A strange idea came to my mind :D wouldn't it be possible to implement both AQs @ the same time? yours to enhance the current picture (distribution on the frame) and the old one todo the rest (distribution between the frames)? hmm something like a Mixed-Mode AQ (the most efficient of both worlds combined) :D

[Dark Shikari]

@Dark Shikari
A strange idea came to my mind :D wouldn't it be possible to implement both AQs @ the same time? yours to enhance the current picture (distribution on the frame) and the old one todo the rest (distribution between the frames)? hmm something like a Mixed-Mode AQ :DWouldn't work. The old AQ didn't change framewide QP, it just changed it (in an extremely strong manner) on a few blocks in the frame.

Also, 0.46 is out, see original post for changes.

[fields_g]

Actually, thinking about it... it should be even better than CRF mode, because it properly takes into account the quantizer needed to reach a constant quality.

Try sensitivity 20 on CRF mode.

I WAS trying out the most current version (.45) until SOMEONE wanted to update it! :) Anyway... I'm using my favorite AQ testing material (POTC2 first 34 seconds.. aka the new Disney Logo) @ 1080p. Here's my base command line:

--crf 18 --ref 16 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 7 --trellis 1 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me esa --merange 12 --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "output" "input" --aq-strength 1.0

Here are some results:
Sensitivity ....Filesize
-------------------------------
Auto.............20.0 MB
10................18.5 MB
11................23.8 MB
15................50.6 MB
25................79.5 MB
NoAQ............20.5 MB


As you can see, on this clip, the sensitivity really blows up the bitrate. Auto does a really good job of following the NoAQ bitrate. When you guys suggest 20, does this produce a filesize that is approximately equal to auto/NoAQ on your clips?

Visually inspecting the results, it is obvious that Auto needs more help. I much prefer the 15 than the Auto or 11 results. This leads me to believe that your suggestion of 20 might not be so bad! Just hoping this doesn't always mean a 3x bitrate at a given CRF!

[CruNcher]

fields_g see it's almost a real VBR mode you get the best possible quality but with it CRF becomes non predictable anymore DABR i think isn't possible this way (or actually has to be new ploted). :) At least the whole CRF shifts the higher you set the Sensitivity CRF 19 with for example --aq-sensitivity 40 would get very very big ;) i would suggest for a Real VBR that reaches Visual Losslesness (Transparency) 50dB here as a good target to test alot of source (the most complex you can find) and see how it comes out CRF 19 and @ wich Sensitivity :)

[fields_g]

I guess there is a reason that it is an AQ test sample for me. It isn't a cakewalk to fix. The bitrate is showing that!

[CruNcher]

25................79.5 MB
how much dB did you reached with it with your settings?

[fields_g]

Playing around earlier I made sure to hit the ssim / pnsr boxes, but this run I didn't. I'll rerun some of these and get those.


EDIT: sensitivity 25 - SSIM .9931239 - PSNR Y55.808 U57.585 V57.585 Avg56.259 Global50.669
Though AQ has to be judged subjectively.

[Dark Shikari]

OK folks, here's a challenge for you!

In order to put the AQ in SVN, we're going to need to decide on a good general sensitivity to use that would keep bitrate relatively constant on a large sample of different types of video. That is, a flat video might get higher bitrate, and a complex one lower, but in general, on a large sample of a bunch of random stuff, it shouldn't change the bitrate at all in CRF mode.

So, grab yourself a ton of selecteveryrange() and find me a sensitivity that doesn't change bitrate at all on a combination of many sources. Once we get that sensitivity, I will likely remove the sensitivity option completely and simply have two modes--automatic and static.

[fields_g]

OK folks, here's a challenge for you!

In order to put the AQ in SVN, we're going to need to decide on a good general sensitivity to use that would keep bitrate relatively constant on a large sample of different types of video. That is, a flat video might get higher bitrate, and a complex one lower, but in general, on a large sample of a bunch of random stuff, it shouldn't change the bitrate at all in CRF mode.

Is the idea to also get rid of --aq-strength also? Should we be testing at 1.0 or a variety of strengths and comparing to NoAQ?

[Dark Shikari]

Is the idea to also get rid of --aq-strength also? Should we be testing at 1.0 or a variety of strengths and comparing to NoAQ?Nah, just compare strength 1.0 to strength 0, since 1.0 is meant to be a good default.

AQ strength should always be adjustable.

[CruNcher]

does someone has the EBU testsequence they test broadcast codecs with i think that would be perfect a mix of the schumacher and alot of other testclips that is :)

http://www.ebu.ch/en/technical/hdtv/test_sequences.php

[desta]

I'm assuming DS means finding a sensitivity that gives the same final bitrate as using no AQ. The sensitivity being the same or pretty close when applied to various samples.

Edit: too late.

[burfadel]

Who else thinks it would be better, in general, to have aq-strength 1.0 set as default (that is, without the need to actually specify it at all), but still have it adjustable by the addition of the --aq-strength x.x command?

[Dark Shikari]

Who else thinks it would be better, in general, to have aq-strength 1.0 set as default (that is, without the need to actually specify it at all), but still have it adjustable by the addition of the --aq-strength x.x command?I'm going to leave in AQ strength. Having it on by default might definitely be useful, if akupenguin would allow it.

[CruNcher]

Btw Dark i fixed all this problems in those scenes by combining your aq with the prestige matrix :) now i have both of best worlds your enhanced detail preservation + a fix for those ROI problems im uber happy (no banding situations anymore) :)

http://mirror05.x264.nl/CruNcher/force.php?file=./roi-fixed-newaq.mkv <- combined result still 0.45 :)

This is the most advanced lossy encode i ever did :)

[Morte66]

OK folks, here's a challenge for you!

In order to put the AQ in SVN, we're going to need to decide on a good general sensitivity to use that would keep bitrate relatively constant on a large sample of different types of video. That is, a flat video might get higher bitrate, and a complex one lower, but in general, on a large sample of a bunch of random stuff, it shouldn't change the bitrate at all in CRF mode.

Let me see if I understand this...

Current CRF implies a certain level of quality (crispness, fine detail retention etc), but the judgement of quality places more weight on complex material relative to flat areas than humans do. Current CRF (without AQ) is not a constant quality mode as judged by humans.

With current CRF/AQ, CRF achieves the same level of quality on complex stuff and AQ throws extra bitrate at flat areas. It maintains the quality of plain CRF on complex material, and improves flat material with extra bitrate. Overall bitrate rises. It gets closer to constant quality as perceived by humans. [I like this -- for me the point of CRF is to give constant quality by spending whatever bitrate is necessary.]

VAQ, however, nabs bitrate from complex areas to improve flat areas (both within and optionally between frames). So if CRFVAQ (hey, a new acronym) maintains the bitrate of plain CRF, will it look worse on complex stuff and better on smooth? Will a particularly shadowy episode of the X-Files look less crisp and detailed in its complex areas than one which is visually busy? Will "CRF 18" no longer imply a certain level of crispness and fine detail on the complex stuff? Or have I completely misunderstood it?

[DarkZell666]

From what I gathererd, an important change is rather the method used to distinguish/define what's complex and what isn't.

You'd think the new AQ munches fine details since it supposedly takes bitrate out of complex scenes, but CruNcher's screenshots actually show some extra detail on people's faces and in flying particles :)


Just my 0.02€ ^^ :helpful:

[akupenguin]

@Morte66
Current CRF is constant quality as measured by one particular metric (not psnr or ssim or any of the normal metrics, but an ad-hoc one implicit in the CRF algorithm). It's similar bitrate on average as the same value of CQP, because I multiplied the CRF scale by a magic number to make that true, not by anything inherent in the algorithm.
CRF+HaaliAQ is constant quality as measured by another metric. It's always at least a little higher bitrate per CRF value than plain CRF because HaaliAQ has a negative average QP bias, but that's just a UI tuning issue and could be fixed if desired.
CRF+VAQ(static) is constant quality as measured by another metric. If you pick some random value of aq-sensitivity, it won't be equal on average, but will have some bias. We're trying to find a sensitivity value such that that bias averages to 0, thus it will hopefully be about the same bitrate on average as a given value of plain CRF if we do the tuning right (unless we choose to keep psy quality the same instead of bitrate).
CRF+VAQ(auto) does the frame-wise bit allocation using the same metric as plain CRF, but reallocates bits within the frame using the VAQ metric.

In short, there's no substantive difference between "improves smooth stuff at the cost of complex stuff" and "improves smooth stuff while keeping complex stuff the same". The only criterion AQ can be judged by is the relative distribution of quality. Any bias in the total quality is just a user interface issue separate from the algorithm.

[Morte66]

In short, there's no substantive difference between "improves smooth stuff at the cost of complex stuff" and "improves smooth stuff while keeping complex stuff the same". The only criterion AQ can be judged by is the relative distribution of quality. Any bias in the total quality is just a user interface issue separate from the algorithm.

*digests*

OK, I see what you mean. That makes good sense to me. Thank you.

So if I found that...

- plain crf 18 creates a 350MB file with satisfactory complex stuff but unsatisfactory smooth stuff

- crf 18 with Haali --aq-strength 0.3 --aq-sensitivity 5 creates a 600MB file which is satisfactory on complex and flat

- crf 18 with variance --aq-strength 1.0 --aq-sensitivity 15 creates a 350MB file which is better overall than either of the above, but occasionally loses a little on the complex stuff

... then I ought to try using variance AQ and changing crf to 17, hoping to get quality that is equal or better in every respect to the 600MB Haali AQ encode in perhaps 400MB?

[LoRd_MuldeR]

@Morte66
Current CRF is constant quality as measured by one particular metric (not psnr or ssim or any of the normal metrics, but an ad-hoc one implicit in the CRF algorithm). It's similar bitrate on average as the same value of CQP, because I multiplied the CRF scale by a magic number to make that true, not by anything inherent in the algorithm.
CRF+HaaliAQ is constant quality as measured by another metric. It's always at least a little higher bitrate per CRF value than plain CRF because HaaliAQ has a negative average QP bias, but that's just a UI tuning issue and could be fixed if desired.
CRF+VAQ(static) is constant quality as measured by another metric. If you pick some random value of aq-sensitivity, it won't be equal on average, but will have some bias. We're trying to find a sensitivity value such that that bias averages to 0, thus it will hopefully be about the same bitrate on average as a given value of plain CRF if we do the tuning right (unless we choose to keep psy quality the same instead of bitrate).
CRF+VAQ(auto) does the frame-wise bit allocation using the same metric as plain CRF, but reallocates bits within the frame using the VAQ metric.

In short, there's no substantive difference between "improves smooth stuff at the cost of complex stuff" and "improves smooth stuff while keeping complex stuff the same". The only criterion AQ can be judged by is the relative distribution of quality. Any bias in the total quality is just a user interface issue separate from the algorithm.

Thanks for the explanation. Very interesting post :)

[ToS_Maverick]

OK folks, here's a challenge for you!

In order to put the AQ in SVN, we're going to need to decide on a good general sensitivity to use that would keep bitrate relatively constant on a large sample of different types of video. That is, a flat video might get higher bitrate, and a complex one lower, but in general, on a large sample of a bunch of random stuff, it shouldn't change the bitrate at all in CRF mode.

So, grab yourself a ton of selecteveryrange() and find me a sensitivity that doesn't change bitrate at all on a combination of many sources. Once we get that sensitivity, I will likely remove the sensitivity option completely and simply have two modes--automatic and static.

hey i got some results for you!

aq10 means --aq-strength 1.0
sens15 means --aq-sensitivity 15
and so on

BlackPearlSample (3622 frames):

Black.Pearl.Sample test crf 18 aq00.mkv 35,3 MB
Black.Pearl.Sample test crf 18 aq10 sens13.mkv 35 MB
Black.Pearl.Sample test crf 20 aq10 sens15.mkv 32,6 MB
Black.Pearl.Sample test crf 22 aq10 sens18.mkv 33,3 MB
Black.Pearl.Sample test crf 24 aq10 sens21.mkv 32,1 MB
Black.Pearl.Sample test crf 26 aq10 sens25.mkv 32,3 MB


SAW 1 DVD comp check (1488 frames):

SAW test crf 18 aq00.mkv 15,5 MB
SAW test crf 18 aq10 sens12.mkv 15,7 MB
SAW test crf 18 aq10 sens13.mkv 19,4 MB


Klick from HDTV comp check (1548 frames):

klick comp test crf 18 aq00.mkv 49,7 MB
klick comp test crf 18 aq10 sens12.mkv 42,5 MB
klick comp test crf 18 aq10 sens13.mkv 50,9 MB


what i found out:
- 3 steps in sensitivity roughly equals 2 steps in CRF
- sensitivity 13 seems to be a good pick (for now) if you want the same CRF filesize with and without AQ

[Chabb]

Would it be right to use AQ with lowered deadzones and custom quantization matrixes? Please express any pro et contra.

[Morte66]

Entourage Season 2 Episode 11 with AviSynth levels/deblock/denoise/deband -> huffYUV, then x264 crf 18 encode. FWIW I've put what I thought of the quality in brackets.

Variance AQ build 46 but no AQ set: 372MB (unsatisfactory)
Variance AQ strength 1.0 sensitivity 15: 326MB (about the same quality but smaller)
Variance AQ strength 1.0 sensitivity 17: 378MB (barely satisfactory, only saw problems if I looked for them)
Variance AQ strength 1.0 sensitivity 20: 516MB (comfortable, maybe diminished returns)

Well, based on that completely inadequate pool of data I'd make --aq-sensitivity a parameter that defaults to 17.

@TOS_Maverick: you got a much lower number. Did you pre-process in any way? Maybe because I deblocked/denoised/debanded my video has more "smooth/flat" in it.

[DeathTheSheep]

Would it be right to use AQ with lowered deadzones and custom quantization matrixes? Please express any pro et contra.

Pro:
Btw Dark i fixed all this problems in those scenes by combining your aq with the prestige matrix :) now i have both of best worlds your enhanced detail preservation + a fix for those ROI problems im uber happy (no banding situations anymore) :)

http://mirror05.x264.nl/CruNcher/force.php?file=./roi-fixed-newaq.mkv <- combined result still 0.44 :)

This is the most advanced lossy encode i ever did :)

Contra: Not much has been said about deadzones, so more testing has to be done.

[CruNcher]

It seems with useing a custom matrix you can have an influence on how Darks NewAQ behaves on the Frame for example what you will see here is that both combined are like a AI AQ for this testcut (it seems almost to decide magicaly were the scene could band and then allocates more bits into this area)
This is perfect Visually as it enhances allways the right area :)
So if there is no banding possibility it will enhance (Facial Details) if their could banding happen (it does enhance the background with the gradients).
I know it's not AI but the balance that i shifted this way by useing a custom matrix, tough still 1 scene gives me major headaches Visually ;)

Noaq
http://s5.directupload.net/images/080124/9u87knfx.png
http://s6.directupload.net/images/080124/g7owp2vh.png (this scene is the Visual killer, especialy when the viewing system is wrong calibrated)
http://s2.directupload.net/images/080124/rqsiveqs.png
http://s5.directupload.net/images/080124/qj6gkjzb.png
http://s1.directupload.net/images/080124/fi6pqiwd.png
http://s3.directupload.net/images/080124/9xfz7l2t.png
http://s3.directupload.net/images/080124/bf8kl9uw.png

Newaq
http://s2.directupload.net/images/080124/idkvrpus.png
http://s5.directupload.net/images/080124/ge4gmtph.png
http://s6.directupload.net/images/080124/8apvaqwh.png
http://s2.directupload.net/images/080124/j32kt8zb.png
http://s3.directupload.net/images/080124/pe7f62ej.png
http://s5.directupload.net/images/080124/sf3up4fl.png
http://s4.directupload.net/images/080124/4qr7pot6.png

But i think i still need to balance it out a little as you can see ringing (to sharp) appearing, btw Prestige Matrix does the shifting in this case very precise dunno what some have against this matrix it seems visually optimized entirely for grain preservation and it does a good job i can also see no artifacts @ all, coused by it (except the sligth ringing effect coused by overoptimizing the source visually with this) :)

[Morte66]

Maybe because I deblocked/denoised/debanded my video has more "smooth/flat" in it.

So I ran it again with all the crud left in, just doing levels. This time:

crf 18 no AQ: 623MB
crf 18 VAQ strength 1.0 sensitivity 17: 932MB (sensitivity 17 matched sizes when cleaned up)
crf 18 VAQ strength 1.0 sensitivity 13: 559MB (13 seemed closest for TOS_Maverick's tests)
crf 18 VAQ strength 1.0 sensitivity 14: 648MB (and 14 is closest for my uncleaned DVD)

So if I feed in a middle of the road DVD with no cleanup, I get results fairly similar to TOS_Maverick (sensitivity ~13 matches no AQ). For a cleaned up DVD, which doesn't have crud all over the smooth/dark content making it un-smooth, it needed to be 3 or 4 higher.

[bob0r]

AQ patch 0.47: http://akuvian.org/src/x264/x264_aq_var.47.diff

x264.721.dark.aq.0.47.exe (http://files.x264.nl/AQ/x264.721.dark.aq.0.47.exe) (pthreads/mp4 = yes, made with make fprofiled)

[Terranigma]

Thanks for that bob0r. :)

[Dark Shikari]

AQ patch 0.47: http://akuvian.org/src/x264/x264_aq_var.47.diff

x264.721.dark.aq.0.47.exe (http://files.x264.nl/AQ/x264.721.dark.aq.0.47.exe) (pthreads/mp4 = yes, made with make fprofiled)Can you guys do your testing again with this patch? It vastly cleaned up the code, and more importantly, it fixed a rounding error with low QPs which should somewhat improve visual quality in flat areas. However, this may change the results of your sensitivity tests.

[DeathTheSheep]

The fact that this patch has been worked on by akupenguin (and is now hosted on his site) of no small significance or import, and perhaps signals the coming of something momentous.

A rounding error was fixed? What's the likelihood of any quality difference occurring at high QP?

[Dark Shikari]

The fact that this patch has been worked on by akupenguin (and is now hosted on his site) of no small significance or import, and perhaps signals the coming of something momentous.

A rounding error was fixed? What's the likelihood of any quality difference occurring at high QP?I'm referring to the lower QPs *in the video*, i.e. the flat areas that get lowered a lot. Areas with low variance.

Not low QPs as a general statement.

Previously, variance was calculated as (SSD - (SAD / 16) ^2). Now its calculated as (SSD - SAD^2 / 256).

[DeathTheSheep]

Ouch. In that case, there's almost guaranteed to be a [somewhat] positive difference on the very places that count most.

[Morte66]

Can you guys do your testing again with this patch?

Arrgh! Just as I've nearly finished the 1080p test. ;)

I did the Taurus media samples, which are uncompressed footage from a digital camera. They're very busy, not much smooth/flat stuff, so they're not really a good surrogate for film/TV. But I've never been able to resist testing them since I spent three whole days downloading them.

crf 18 no AQ: 228MB
crf 18 VAQ strength 1.0 sensitivity 17: 381MB
crf 18 VAQ strength 1.0 sensitivity 14: 203MB

That's with 0.46.

[DeathTheSheep]

.47 makes the files balloon in size at the same threshold I was using before (22 in this particular test case).

Seems quite a bit slower, but maybe this can be explained by the fact that my "old" settings apparently correspond to different things now.

[edit]
[I said edit, where's the edit notifier at the bottom? ...that's better.]
SSIM just...went...through...the...roof...... (I'm going to have to re-shingle now, damn it).

[Dark Shikari]

.47 makes the files balloon in size at the same threshold I was using before (22 in this particular test case).

Seems quite a bit slower, but maybe this can be explained by the fact that my "old" settings apparently correspond to different things now.Yeah, due to rounding, 0.46 wasn't giving flat blocks low enough quantizers.

[ditche]

I don't understand...

I've encoded a movie (duration : 1h52) with theses settings :
--qp 22 --ref 10 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --bime --weightb --direct auto --filter -2,-1 --trellis 2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads 3 --thread-input --sar 1:1 --progress --no-dct-decimate --no-psnr --no-ssim --output "D:\VIDEO\rip\video.mp4" "D:\VIDEO\rip\video.avs" --aq-strength 1.0
CRF : 22, resolution : 688x304, x264 721 0.45

Results : 339 Mo @ CRF22 for 1h52.
Fun, tiny size, but...

Nop, because the result is very bad...

http://img134.imageshack.us/img134/7029/snapshot20080124230054ib5.th.jpg (http://img134.imageshack.us/my.php?image=snapshot20080124230054ib5.jpg)

http://img134.imageshack.us/img134/94/snapshot20080124230249cm7.th.jpg (http://img134.imageshack.us/my.php?image=snapshot20080124230249cm7.jpg)

My other encodings with same settings give me larger files but a good quality...
:confused:


Have you an idea ? Thanks. :)

:helpful:

[Dark Shikari]

Those are somewhat odd settings you're using, plus, you're using QP mode, not CRF mode. Trellis 2 doesn't help detail retention, either.

I suspect you're just not giving it enough bits.

[Atak_Snajpera]

Since when --qp 22 means --crf 22 ?

[ditche]

Since when --qp 22 means --crf 22 ?
Arf !! :)
I make a little confusion between qp & crf... :p

[LoRd_MuldeR]

Since when --qp 22 means --crf 22 ?

ditche, according to your command-line you were using --qp 22, but you said "CRF : 22" :p

[ditche]

Yeah, I'm confused...

It's almost midnight here, tomorrow i'll try with theses settings, is it OK for you (± Shartooth settings :p) ?

--crf 22.0 --ref 3 --mixed-refs --bframes 16 --b-pyramid --bime --weightb --filter -2,-1 --subme 1 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto --thread-input --progress --no-psnr --no-ssim --output "output" "input" --aq-strength 1.0 --aq-sensitivity 15

:)


Edit :
Ah, it's looks so much better on my last sample test. :)

[Dark Shikari]

Those are really bizarre settings--UMH/8x8dct/many refs, but subme 1?!

[Don_Genaro]

Those are really bizarre settings--UMH/8x8dct/many refs, but subme 1?!

I think they´re taken from the megui CRF profile wich has --subme 1

[nurbs]

Aq version 0.47

avs:
SelectRangeEvery(1000, 50)

Commandline:
--crf 20.0 --level 3.1 --ref 3 --bframes 3 --b-pyramid --weightb --filter -1,-1 --subme 6 --trellis 1 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-maxrate 12000 --me umh --threads auto --thread-input --progress --no-psnr --no-ssim

--aq-strength 1.0 when AQ is used

Coffee and Cigarettes:
No AQ : 1387
aq-sensitivity 9 : 1079
aq-sensitivity 10 : 1516

Clerks 2:
No AQ : 3123
aq-sensitivity 12 : 2677
aq-sensitivity 13 : 3161

A Farscape Episode(4x22):
No AQ : 1755
aq-sensitivity 8 : 1676
aq-sensitivity 9 : 2170

Severance:
No AQ : 1362
aq-sensitivity 8 : 1071
aq-sensitivity 9 : 1413

The Great Dictator:
No AQ : 1627
aq-sensitivity 12 : 1552
aq-sensitivity 13 : 1857

Babylon 5 - Lost Tales:
No AQ : 1108
aq-sensitivity 7 : 946
aq-sensitivity 8 : 1709

[ToS_Maverick]

Dark Shikari, what have you done, it's even more insane now :D

Black.Pearl.Sample test crf 18 aq00.mkv 35,3 MB
Black.Pearl.Sample test crf 18 aq10 sens10.mkv 33,3 MB
Black.Pearl.Sample test crf 18 aq10 sens11.mkv 40,2 MB !!!
Black.Pearl.Sample test crf 20 aq10 sens12.mkv 34,2 MB

SAW comp test crf 18 aq00.mkv 15,4 MB
SAW comp test crf 18 aq10 sens09.mkv 17,3 MB
SAW comp test crf 18 aq10 sens10.mkv 21,7 MB

Casablanca test crf 18 aq00.mkv 13,5 MB
Casablanca test crf 18 aq10 sens10.mkv 11,2 MB
Casablanca test crf 18 aq10 sens11.mkv 13,3 MB

quality for BP crf 18@10 and crf 20@12 seems to be quite identical, bot got SSIM 0.973xx and look good. i just had a quick look over them, but they seem fine.

so my vote at the moment is sens for default at 10. nice number and seems to be quite ok :cool:

[salehin]

Thanks a lot, guys. :)

...It vastly cleaned up the code, and more importantly, it fixed a rounding error with low QPs which should somewhat improve visual quality in flat areas. However, this may change the results of your sensitivity tests.

I may be completely wrong, but does it mean that now the flat areas gets more balanced bits and thus making the QP in those sections (and consequently the overall avg QP) better. It would be great if you can kindly explain in laymen's terms.

Regarding: I noticed that in very clean source (like bbchd power of the planet h264 1080p) that contains very small amount of dark sequences, the encode quality improved a signficantly with weak quantization (0.30 - 0.50) and sensitivity of 20-25 (based on bob0r's v .45). Did anyone had similar experience?

Also, what if the source is quite dark, or full of dirts due to old age, 60's handheld cameras etc? Thanks :)

[Morte66]

Here we go again at the Blue Balls Lagoon (Entourage season 2 episode 11, about 0.94GB of MPEG2), with v0.47.

Levelled/deblocked/denoised/debanded:

crf 18 no AQ: 372MB
crf 18 VAQ strength 1.0 sensitivity 17: 538MB
crf 18 VAQ strength 1.0 sensitivity 14: 414MB
crf 18 VAQ strength 1.0 sensitivity 13: 370MB
crf 18 VAQ strength 1.0 sensitivity 12: 325MB

I would say sensitivity 13 (which matched the bitrate of non-AQ) produced a well-balanced encode. No one imperfection stood out ahead of the others. Blocking/banding was there if I looked for it, but it did not leap out as the dominant issue. It also seemed a little soft compared to no AQ, it could use another crf level or two to get back the crispness.


Same material just levelled, no other cleanup:

crf 18 no AQ: 623MB
crf 18 VAQ strength 1.0 sensitivity 17: 1.17GB (25% bigger than source)
crf 18 VAQ strength 1.0 sensitivity 14: 900MB
crf 18 VAQ strength 1.0 sensitivity 13: 792MB
crf 18 VAQ strength 1.0 sensitivity 12: 685MB
crf 18 VAQ strength 1.0 sensitivity 11: 580MB


Well, call it 13 for denoised and 11.5 for noisy. It's getting tighter, but I don't think you can remove the sensitivity parameter yet.

[Dark Shikari]

Well, call it 13 for denoised and 11.5 for noisy. It's getting tighter, but I don't think you can remove the sensitivity parameter yet.The idea is that some videos do indeed need more bits than others--and therefore noisy videos will get more bits at the same "quality level." Therefore, sensitivity isn't needed, because all that matters is that two videos at the same AQ strength and CRF are "the same quality."

[CruNcher]

Nice with the latest patch i don't need the Prestige Matrix anymore Aku's 0.47 :) and save some details --aq-strength 1.0 alone is now able to handle all of the scenes accordingly (except the very dark scene when he gets out of the watter (killer scene) that's still a mess but that was expected, just to less bitrate would need a 2pass for that or very strong oldaq) and doesn't lose anything of it's visual detail enhancement the SSIM lowered a little again (steady with every patch) but the visual balance is much better now :)

[Yoshiyuki Blade]

Hmmm so how do you guys determine the best AQ sensitivity (at a given clip)? The one that produces a file size approximately equal to the same clip without AQ?

EDIT: I didnt mean to use the word "best" but rather "recommended" or something.

[akupenguin]

It doesn't matter. Any sensitivity is equally good. What we're tuning is the mapping of CRF value to quality level, so that hopefully CRF will mean the same thing with and without AQ.

[Dark Shikari]

Hmmm so how do you guys determine the best AQ sensitivity (at a given clip)? The one that produces a file size approximately equal to the same clip without AQ?

EDIT: I didnt mean to use the word "best" but rather "recommended" or something.It seems to be roughly around 10-12 for most clips.

[EuropeanMan]

Perhaps a stupid question...

My custom parametre in Command line: --b-pyramid doesn't work...it errors out in MeGUI...any suggestion on how to fix it? For now, I eliminated this...and the encode went through...

thanks in advance.

I'm using the latest version of MeGUI on vista ultimate. all plugins updated...and i used ONLY the build version of AQ (don't know what this means) & installed...

And yes I'm doing my very very first x264 encode. and PS...since noone has answered this yet in my other thread...is it possible to use DTS audio in my mp4? :)

[DeathTheSheep]

In the interest of being honest, I don't think this new AQ with "fixed" rounding does as well on low-bitrate (anime). Here's what's happening qualitatively as a result of the new AQ:

1. Sensitivity must be 3 or 4 lower at the same QP for comparable bitrate.
2. SSIM at similar bitrate is decreased (tried with many QP/sensitivity combinations).
3. Dark, crappy scenes (pardon my slang) are allocated *less* bitrate in the video stream, and bright, high-contrast ones are allotted more than with previous AQ. Increasing AQ strength seems to take bits away from both.

[edit]To actually see them, cut from the original which is the same (total) size under both AQs, http://gabext.com/samples/

Maybe there should be an anime-lowpass. Blocks that are too flat shouldn't be given quantizers so low, in order to improve what we actually see.

[akupenguin]

An effect of the rounding (before I fixed it) was to add a small constant to all the variances, thus reducing the difference in QPs (but not exactly the same as reducing aq-strength). If that is desirable, we can explicitly add such a constant, and still benefit from the higher arithmetic precision.

[CruNcher]

@Death
Aku shifted it very nicely :) at least for mixed Quality stuff it makes alot of sense (Flat and Non Flat) look @ this

http://mirror05.x264.nl/CruNcher/force.php?file=./noaq.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./newaq-dark.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./roi-fixed-newaq-dark-cruncher.mkv (my aproach shifting Darks AQ with a custom matrix)
http://mirror05.x264.nl/CruNcher/force.php?file=./newaq-aku.mkv akus work :)

i know people rated the newaq-dark scene before allready as ok, but there were clearly problems visible to me (tough they are hard to realize with a correctly calibrated viewing device, especialy as there aren't really in the ROI in this example (hidden in the luminance @ the left side )), but they are in some situations in middle of the ROI and sometimes for a long time and so the whole thing became painfull to watch before.
Now the AQ does good with every of those situation (very balanced, and it still enhances the Details even at such insane low bitrates) and don't forget PSY is not about SSIM or PSNR it is about the most efficient Visual Perception @ best even at such low bitrates you could put it on a floppy ;) (actually im not far away from putting this HD 400 frame on a floppy ;))

[DeathTheSheep]

I agree with you CruNcher in that it produces considerably better visual quality on live footage. But this seems to be at the direct expense of quality in other sources.
If that is desirable, we can explicitly add such a constant, and still benefit from the higher arithmetic precision.
Yes, I think that this is a good idea with much possible benefit. Perhaps this constant can be made user-adjustable as well, for even further potential gain.

It's somewhat counter-intuitive to me that the recently instated increase in QP differences would yield such significantly lower quality (and bitrate) on such a flat scene with very low contrast.

[Razorholt]

So, is there a significant vidual improvement between 0.46 and 0.47 @ low rates ?

[CruNcher]

@Razorholt
should be yes :)

http://mirror05.x264.nl/CruNcher/force.php?file=./noaq-crf40.mkv
http://mirror05.x264.nl/CruNcher/force.php?file=./newaq-aku-crf40.mkv (--aq-strength 0.1 --aq-sensitivity 11)

[ditche]

I think they´re taken from the megui CRF profile wich has --subme 1
Yeah, I tried to edit my post (for subme 5) but the forum was down...

[Morte66]

Therefore, sensitivity isn't needed, because all that matters is that two videos at the same AQ strength and CRF are "the same quality."

So would AQ-strength remain as a parameter users can override?

Putting it another way: if AQ shifts the definition of "quality" from regular encoding to something more concerned with smooth/dark areas, can users choose how far they want to shift it?

[Dark Shikari]

So would AQ-strength remain as a parameter users can override?

Putting it another way: if AQ shifts the definition of "quality" from regular encoding to something more concerned with smooth/dark areas, can users choose how far they want to shift it?Yes. AQ strength will stay.

[Morte66]

Yes. AQ strength will stay.

Phew. Thanks, that's what I was missing. I got so caught up in testing sensitivity that I forgot that was there. Sorry for the distraction.

[CruNcher]

No AQ CRF 29

x264 [info]: SSIM Mean Y:0.9797418
x264 [info]: PSNR Mean Y:45.548 U:45.836 V:49.471 Avg:46.021 Global:45.525 kb/s:
2972.16

encoded 9336 frames, 7.90 fps, 2975.00 kb/s


New AQ CRF 29 (--aq-strength 1.0 --aq-sensitivity 11)

x264 [info]: SSIM Mean Y:0.9791818
x264 [info]: PSNR Mean Y:43.834 U:45.212 V:47.979 Avg:44.493 Global:43.912 kb/s:
5078.05

encoded 9336 frames, 7.57 fps, 5080.97 kb/s

That's way off :( might have to be lower then --aq-sensitivity 10 to reach the same size

[ToS_Maverick]

this seems to be a very strange video... CRF29 with SSIM 0.98
my guess would be sens 6 or 7 for the same bitrate

i don't get 0.98 even at CRF18 with BlackPearl...

[Atak_Snajpera]

I found something weird in 0.47.

x264 settings
--crf 20 --level 4.1 --sar 1:1 --filter 0,0 --aq-strength 1 --ref 3 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --subme 6 --trellis 1 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto --thread-input --progress --no-psnr --no-ssim --output

1280x544
0.45 version gave size 5.12 MB , NOAQ gave me 4.63 but 0.47 gives me 19.4 MB !!!!

source
http://rapidshare.com/files/59949596/Logo.zip.html

[nurbs]

@Atak_Snajpera:
Filesize explosion and implosion depending on aq-sensitivity happened to me with both 0.45 and 0.47. Maybe you could try to lower the sensitivity to somewhere around 10 and see if that helps.

Also while I don't know if this is important I made a graph with the bitrate for a sample clip at a given crf and sensitivity with varying strenghts.

http://img246.imageshack.us/img246/3500/strvariation1qy5.th.png (http://img246.imageshack.us/my.php?image=strvariation1qy5.png)

[Atak_Snajpera]

Maybe you could try to lower the sensitivity to somewhere around 10 and see if that helps.

but I want to use automatic sensitivity.

[nurbs]

If I understand the first post correctly the default sensitivity is now 20. What you want is 0.

[Atak_Snajpera]

ok thanks. Sensitivity 12 now gives 4.11 MB compared to 4.63 MB. (0 = 5.12 MB)

default sensitivity is now 20
Default value is way to high.

[Jawed]

If qcomp isn't at default 0.6 (I presume that's still the default) will VAQ still deliver the same behaviour?

Does anyone care about qcomp?

It strikes me that qcomp could become more useful (or more meaningful) with VAQ than with old AQ and I'm wondering if qcomp might even be a good way of tweaking VAQ, particularly as VAQ apparently isn't the wild horse that AQ seems to be.

Jawed

[Morte66]

DVD...

Deadwood season 3 episode 7, cleaned up, SelectEvery(1000,50):
crf 16 no AQ: 17.4MB
crf 16 VAQ strength 1.0 sensitivity 8: 15.3MB
crf 16 VAQ strength 1.0 sensitivity 9: 17.8MB

Battlestar Galactica Razor, cleaned up, SelectEvery(1000,50):
crf 16 no AQ: 96.3MB
crf 16 VAQ strength 1.0 sensitivity 8: 75.6MB
crf 16 VAQ strength 1.0 sensitivity 9: 93.5MB
crf 16 VAQ strength 1.0 sensitivity 10: 110MB

Trois Couleus Blanc, cleaned up, SelectEvery(1000,50):
crf 16 no AQ: 49.2MB
crf 16 VAQ strength 1.0 sensitivity 8: 42.3
crf 16 VAQ strength 1.0 sensitivity 9: 51.8

Irreversible, cleaned up, all of it:
crf 16 no AQ: 1.11GB
crf 16 VAQ strength 1.0 sensitivity 13: 1.86GB
crf 16 VAQ strength 1.0 sensitivity 10: 1.37GB
crf 16 VAQ strength 1.0 sensitivity 9: 1.18GB


Whole lotta nines...

[Jawed]

DVD...

Deadwood season 3 episode 7, cleaned up, SelectEvery(1000,50):
Battlestar Galactica Razor, cleaned up, SelectEvery(1000,50):
Trois Couleus Blanc, cleaned up, SelectEvery(1000,50):

What am I missing?: those clips are going to have a way way higher proportion of I frames than a "normal" encode, so how is the crf algorithm going to be acting anything like it would when encoding the real film?

Jawed

[Dark Shikari]

What am I missing?: those clips are going to have a way way higher proportion of I frames than a "normal" encode, so how is the crf algorithm going to be acting anything like it would when encoding the real film?

JawedIn this case, that doesn't actually really matter for comparison.

[akupenguin]

A typical film has one scenecut per 50-100 frames. SelectEvery adds an additional one I-frame per 50 frames, i.e. 2-3x the total number of I-frames. That's not enough to significantly affect this test.

[burfadel]

I tried a small clip (24 sec) of 'That 70's show', season 3, that just had Eric sitting on the couch in the basement eating pizza, then Hyde comes in and there's some panning. The background was completely still until Hyde comes through the door. I resized the image to 512x384, quite low I know but it was just for this test. Anyways, I found a problem with the AQ (0.47), especially with a sensitivity of 9 such as what Morte66 used.

The settings I used (apart from AQ), where all the same:
--crf 24 --keyint 450 --ref 5 --bframes 16 --b-pyramid --weightb --b-bias 40 --b-rdo --bime --direct auto --analyse all --8x8dct --subme 7 --me umh --trellis 1 --mixed-refs --progress

With the default AQ sensitivity (that is, not specified), and strength 1.0 the image looked quite good and very close to the original, but slightly larger (685kb compared to 571kb).

At AQ 9, the filesize was a tiny 201kb, and at 8 it was 170kb! This is with using the same settings. The quality at these sensitivities was shocking, both the background and the foreground were extremely blocky, and the slight moving parts of Eric in the first part caused severe artifacting. Even at a CRF of 17 it did not resolve all of this artifacting.

Therefore, in the case of a standard quality encode, the results with low sensitivities are absolutely disgusting!

I wouldn't go any lower than say, 17... even then the filesize was larger than without AQ, and the picture quality in the bright areas was still slightly reduced!

With these tests and a few others I have done, I am not fussed at all with the slightly larger files, its worth it for the quality increase. That said, I would like to suggest one thing in regards to sensitivity.

Having automatic sensitivity seems to be the best as no one sensitivity seems to be good for all situations. What you could have is sensitivity ratings of say, low, medium, and normal, where normal is the default (naturally).

For low and medium settings, you could still have it as auto, but subtract say, 2 for low, and 1 for medium (or something along those lines). It just seems too much of a disadvantage having a fixed sensitivity for every encoding scenario from bright to dark, low res to full HD, almost no motion to high motion.

[Dark Shikari]

I tried a small clip (24 sec) of 'That 70's show', season 3, that just had Eric sitting on the couch in the basement eating pizza, then Hyde comes in and there's some panning. The background was completely still until Hyde comes through the door. I resized the image to 512x384, quite low I know but it was just for this test. Anyways, I found a problem with the AQ (0.47), especially with a sensitivity of 9 such as what Morte66 used.

The settings I used (apart from AQ), where all the same:
--crf 24 --keyint 450 --ref 5 --bframes 16 --b-pyramid --weightb --b-bias 40 --b-rdo --bime --direct auto --analyse all --8x8dct --subme 7 --me umh --trellis 1 --mixed-refs --progress

With the default AQ sensitivity (that is, not specified), and strength 1.0 the image looked quite good and very close to the original, but slightly larger (685kb compared to 571kb).

At AQ 9, the filesize was a tiny 201kb, and at 8 it was 170kb! This is with using the same settings. The quality at these sensitivities was shocking, both the background and the foreground were extremely blocky, and the slight moving parts of Eric in the first part caused severe artifacting. Even at a CRF of 17 it did not resolve all of this artifacting.

Therefore, in the case of a standard quality encode, the results with low sensitivities are absolutely disgusting!

I wouldn't go any lower than say, 17... even then the filesize was larger than without AQ, and the picture quality in the bright areas was still slightly reduced!

With these tests and a few others I have done, I am not fussed at all with the slightly larger files, its worth it for the quality increase. That said, I would like to suggest one thing in regards to sensitivity.

Having automatic sensitivity seems to be the best as no one sensitivity seems to be good for all situations. What you could have is sensitivity ratings of say, low, medium, and normal, where normal is the default (naturally).

For low and medium settings, you could still have it as auto, but subtract say, 2 for low, and 1 for medium (or something along those lines). It just seems too much of a disadvantage having a fixed sensitivity for every encoding scenario from bright to dark, low res to full HD, almost no motion to high motion.I'm thinking of --aq-strength 0.7 --aq-sensitivity 14 as a medium... I would think this should generate a relatively decent encode with ordinary CRF settings and be comparable between different clips.

[burfadel]

Having only a 24 second clip at 512x384 makes for some quick testing!

At --aq-strength 0.7 and --aq-sensitivity 14, the image was still blocky where without it its no problem...! The file size is still smaller than that of with no AQ.

With a senstivity of 16 (strength 0.7), the file size is only 20kb smaller but the area of blocking is still not as good as without AQ... This may not be a problem with 1920x1080 encodes, but for dvd encodes (720x576 aka PAL) the problem is still there!

[CruNcher]

(--aq-strength 1.0 --aq-sensitivity 7)

x264 [info]: SSIM Mean Y:0.9687704
x264 [info]: PSNR Mean Y:41.195 U:43.895 V:46.262 Avg:42.077 Global:41.401 kb/s:
1607.20

encoded 9336 frames, 8.24 fps, 1610.42 kb/s


eh hehe

[Jawed]

A typical film has one scenecut per 50-100 frames. SelectEvery adds an additional one I-frame per 50 frames, i.e. 2-3x the total number of I-frames. That's not enough to significantly affect this test.
Well, it seems to me that an I frame will have a significantly lower quantiser, so the test is being biased towards "low quantisers" where VAQ is going to have relatively little effect.

Anyway, who am I to argue...

Also, why are people posting SSIM and PSNR data when it's the fact that these measures don't match human perception that calls for the VAQ algorithm?

Jawed

[Yoshiyuki Blade]

I've noticed many clips jump significantly in file size when AQ sensitivity gets bumped up a notch. Does AQ strength have anything to do with this?

[CruNcher]

The Size is important here and it's flipping like crazy :P

[lexor]

Having only a 24 second clip at 512x384 makes for some quick testing!

At --aq-strength 0.7 and --aq-sensitivity 14, the image was still blocky where without it its no problem...! The file size is still smaller than that of with no AQ.


Well, since the testing is so quick, how about trying strength 1 and going with sensitivity from say 7 to 20 (maybe step by 2, to limit the number of encodes) and telling us which one works on that particular clip? We really can't tell you what to do to improve the scene since we aren't working with the same material.

[akupenguin]

I've noticed many clips jump significantly in file size when AQ sensitivity gets bumped up a notch. Does AQ strength have anything to do with this?
That's every clip, and it's perfectly predictable and intended. In static sensitivity mode, --aq-sensitivity simply determines a constant that gets added to the QP. Because if we didn't add anything, then VAQ would introduce a huge bias in quality-per-CRF-value, much more than HaaliAQ. Come to think of it, it should be in units of QP; dunno why Dark Shikari didn't do that.

[Yoshiyuki Blade]

That's every clip, and it's perfectly predictable and intended. In static sensitivity mode, --aq-sensitivity simply determines a constant that gets added to the QP. Because if we didn't add anything, then VAQ would introduce a huge bias in quality-per-CRF-value, much more than HaaliAQ. Come to think of it, it should be in units of QP; dunno why Dark Shikari didn't do that.

Ah ok. I used the word "many" for being uncertain of its predictability.

[CruNcher]

(--aq-strength 1.0 --aq-sensitivity 8)

x264 [info]: SSIM Mean Y:0.9721481
x264 [info]: PSNR Mean Y:42.009 U:44.243 V:46.741 Avg:42.817 Global:42.172 kb/s:
2204.62

encoded 9336 frames, 7.83 fps, 2207.73 kb/s

Seems like i endup with a --aq-sensitivity of 9 like Morte66 before me :)

[nurbs]

With my clips I also ended up at --aq-sensitivity 9 with --aq-strength 1.0 to get similar bitrates to pure crf. But the settings don't work well on all clips. If I use them on Clerks 2 for instance I end up with a small really bad looking file.

[Inventive Software]

@Dark_Shikari, akupenguin: Can you leave the sensitivity option in for those "unusual" sources if the patch gets committed, but conceal it as an "advanced" option, i.e only list it in --longhelp?

[akupenguin]

No. Sensitivity is a property of the algorithm, it's useless as an option. If your source needs more bitrate, lower the value of CRF.

[CruNcher]

yep i would also say so, because it's really like sitting infront of a Roulette Table makeing your bets this way better to use CRF ;)
Reja ne va plue

9 Black wins

x264 [info]: SSIM Mean Y:0.9748461
x264 [info]: PSNR Mean Y:42.691 U:44.574 V:47.179 Avg:43.439 Global:42.820 kb/s:
2989.76

encoded 9336 frames, 8.71 fps, 2992.77 kb/s

So for me this would mean most probably --aq-sensitivity 20 and a CRF somewhere @ 32 i guess instead of currently without AQ 29 :), the result above also looks far from Optimal compared to the No AQ (hard edges, looks oversharped in motion, tough fixes the problematic scenes but the Detailed Scenes now look horrible)

[Dark Shikari]

Especially since moving sensitivity is equivalent to changing CRF :)

[Inventive Software]

Damn... who knew AQ could be so complicated eh? :D

[burfadel]

With my clips I also ended up at --aq-sensitivity 9 with --aq-strength 1.0 to get similar bitrates to pure crf. But the settings don't work well on all clips. If I use them on Clerks 2 for instance I end up with a small really bad looking file.

Thats exactly what I found with some small tests I did as well! Looking at the resultant file is much better than relying on the numbers alone :)

Lowering the CRF from 24 to even 18 didn't completely help the situation, and you certainly don't want to keep finding the 'sweet spot' every time you encode something.

The lower the sensitivity the lower the quality of the resultant file, I thought this AQ was an 'additive' function (add more bits where needed, subtract from where not needed), and not a 'base' function (every block has a higher quality with a higher AQ).

For instance, if you select a sensitivity of 1, the result from the test I did earlier was a file of 45kb and just a complete mess.

Just for clarification, what frames does AQ operate on? does it operate on keyframes alone to have a higher quality base for the p and b frames to be based on, or does it operate on all frames?

[CruNcher]

MAMA (--aq-strength 1.0 --aq-sensitivity 20) CRF 32

x264 [info]: SSIM Mean Y:0.9858374
x264 [info]: PSNR Mean Y:45.823 U:46.553 V:49.433 Avg:46.350 Global:45.818 kb/s:
11161.54

encoded 9336 frames, 7.81 fps, 11164.45 kb/s

CRF 38 maybe?

[Dark Shikari]

The lower the sensitivity the lower the quality of the resultant file, I thought this AQ was an 'additive' function (add more bits where needed, subtract from where not needed), and not a 'base' function (every block has a higher quality with a higher AQ).

For instance, if you select a sensitivity of 1, the result from the test I did earlier was a file of 45kb and just a complete mess.That's because in order to add or subtract, it needs to know where its baseline is.
Just for clarification, what frames does AQ operate on? does it operate on keyframes alone to have a higher quality base for the p and b frames to be based on, or does it operate on all frames?All frames.

[burfadel]

Ah ok! I found that a sensitivity of around 17 is the lowest I'd want to go with anything, anything lower than that and for standard definition encodes the visual quality dropped too much in certain scenes. A lower strength also seems to be a good idea! Once the strength is lowered to around 0.6 at sensitivity 17 the file size is very close to without AQ without quality loss. That seems to be the best combination for this particular clip.

[Yoshiyuki Blade]

nurbs posted a graph that shows the bitrate across different AQ strengths, which seems to be helpful here (post #525). Strength of 0.5 produces the smallest filesize at a given CRF and sensitivity. Other than strength 0.1, its a nice parabolic pattern (above and below 0.5, the bitrate starts going up).

EDIT: Is high strength/low sensitivity any different (visually) than low strength/high sensitivity?

[DeathTheSheep]

Actually, I've started to use sensitivity as a means by which to fine-tune the desired QP, and since x264 doesn't accept fraction QP, it is of use indeed for comparison purposes; that is, in order to reach a desired filesize to some degree of exactitude, one can simply input a fractional sensitivity instead of worrying over the effects of differences in rate control mechanism behavior.

However, I can't quite mirror DS's statement when it comes to QP. Lowering the nominal QP while lowering threshold produces strikingly different results than a higher QP and sensitivity (at nearly identical resultant filesizes). Again, these findings pertain to nominal QP mode.

[Dark Shikari]

Actually, I've started to use sensitivity as a means by which to fine-tune the desired QP, and since x264 doesn't accept fraction QP, it is of use indeed for comparison purposes; that is, in order to reach a desired filesize to some degree of exactitude, one can simply input a fractional sensitivity instead of worrying over the effects of differences in rate control mechanism behavior.

However, I can't quite mirror DS's statement when it comes to QP. Lowering the nominal QP while lowering threshold produces strikingly different results than a higher QP and sensitivity (at nearly identical resultant filesizes). Again, these findings pertain to nominal QP mode.This shouldn't be the case unless there is QP clipping going on; i.e. --qpmin is limiting some QPs.

[MasterNobody]

As I understand the main purpose of ac_energy_mb function is calculation of (X - Xa)^2 where X is pixel value and Xa is average value of pixel. Now it is calculated as X^2 - Xa^2 (ssd - sad^2). But mathematically correct formula would be (X - Xa)^2 = X^2 + Xa^2 - 2*X*Xa (ssd + sad^2 - 2*sqrt(ssd)*sad). So I made modification of AQ which use this formula for ac_energy_mb calculation (also it has other cosmetic changes, and by default it use automatic sensitivity because in my opinion every source need its own sensitivity and you never find value which would be good for any source). I don't know would it have better or worse quality so it needs thorough testing and comparison with not modified AQ.

patch: http://stashbox.org/75410/x264_aq_var.47.mod.diff
exe (pthread = yes, mp4 = no, fprofiled = no):http://stashbox.org/75411/x264.721.dark.aq.0.47.mod.exe

[burfadel]

As I understand the main purpose of ac_energy_mb function is calculation of (X - Xa)^2 where X is pixel value and Xa is average value of pixel. Now it is calculated as X^2 - Xa^2 (ssd - sad^2). But mathematically correct formula would be (X - Xa)^2 = X^2 + Xa^2 - 2*X*Xa (ssd + sad^2 - 2*sqrt(ssd)*sad). So I made modification of AQ which use this formula for ac_energy_mb calculation (also it has other cosmetic changes, and by default it use automatic sensitivity because in my opinion every source need its own sensitivity and you never find value which would be good for any source). I don't know would it have better or worse quality so it needs thorough testing and comparison with not modified AQ.

patch: http://stashbox.org/75410/x264_aq_var.47.mod.diff
exe (pthread = yes, mp4 = no, fprofiled = no):http://stashbox.org/75411/x264.721.dark.aq.0.47.mod.exe

I agree with the automatic sensitivity, especially with your modified patch :) It seems to work much better for the clip I was using before, and the filesize is very close to without AQ! I tried strengths 0.5 and 1, at least for lower resolution clips, including DVD resolution, your modified AQ seems to work better!
(I hope that doesn't sound too rude to Dark_Shikari and Akupenguin, I'm just giving credit to someone giving very worthwhile input into the codec!)

[Dark Shikari]

I agree with the automatic sensitivity, especially with your modified patch :) It seems to work much better for the clip I was using before, and the filesize is very close to without AQ! I tried strengths 0.5 and 1, at least for lower resolution clips, including DVD resolution, your modified AQ seems to work better!
(I hope that doesn't sound too rude to Dark_Shikari and Akupenguin, I'm just giving credit to someone giving very worthwhile input into the codec!)The inherent problem with automatic sensitivity is that it avoids giving more bits to frames that really need it, such as flat, dark frames; this leads to, in many cases, this AQ performing worse than the old (Haali's) AQ.

One could possibly use automatic sensitivity with a min-sensitivity feature to ensure it never drops below a certain value.

[burfadel]

The min sensitivity feature would be a good idea, it would help resolve the problem I had with the other clip! What do you think of Masternobody's modification? It seemed to work okay with me, but then again its trying to find an adequate balance amongst all clips :)

[radius]

Hi, from what I tested the default sensitivity at 20 is a bit aggressive, compared with automatic sensitivity there's a small yet noticeable compression loss, stuff like text suffers at 20 or more, min-sensitivity thing looks like a good idea :)

[Terranigma]

...and automatic sensitivity would be 0 like with strength?

[akupenguin]

As I understand the main purpose of ac_energy_mb function is calculation of (X - Xa)^2 where X is pixel value and Xa is average value of pixel. Now it is calculated as X^2 - Xa^2 (ssd - sad^2). But mathematically correct formula would be (X - Xa)^2 = X^2 + Xa^2 - 2*X*Xa (ssd + sad^2 - 2*sqrt(ssd)*sad).
No, the current formula is correct.
http://en.wikipedia.org/wiki/Variance

[Yoshiyuki Blade]

Heres my results from an anime test encode:

Rurouni Kenshin, episode 48 (aggressively filtered/resized)

CRF 25 (no AQ): 157 MB
CRF 25 --aq-strength 1.0 --aq-sensitivity 8: 95.6 MB
CRF 25 --aq-strength 1.0 --aq-sensitivity 9: 124 MB
CRF 25 --aq-strength 1.0 --aq-sensitivity 10: 161 MB
CRF 25 --aq-strength 1.0 --aq-sensitivity 12: 259 MB

Looking purely by numbers, sensitivity of 10 is the closest to the original bitrate. Quality is significantly lower across the entire episode, but dark areas look much better.

[DeathTheSheep]

The inherent problem with automatic sensitivity is that it avoids giving more bits to frames that really need it, such as flat, dark frames; this leads to, in many cases, this AQ performing worse than the old (Haali's) AQ.
The same is true of .47 with sensitivity, compared to .45. (I hope the reintroduction of the constant will make it into the next release). Really, it's odd how the correct rounding was so stingy on the frames that needed it most in my samples.

[burfadel]

No, the current formula is correct.
http://en.wikipedia.org/wiki/Variance

So the results I got with some quick testing weren't 'accurate'? I didn't and don't have time today for more thorough tests, on the outset it just seems to work better (then again, it may have just like the particular clips I threw at it)!

[Dark Shikari]

So the results I got with some quick testing weren't 'accurate'? I didn't and don't have time today for more thorough tests, on the outset it just seems to work better (then again, it may have just like the particular clips I threw at it)!That's odd, as in my experience fixing the rounding made low QPs even lower.

[akupenguin]

So the results I got with some quick testing weren't 'accurate'? I didn't and don't have time today for more thorough tests, on the outset it just seems to work better (then again, it may have just like the particular clips I threw at it)!

I'm not saying that AC energy aka variance is the optimal heuristic for AQ. I'm saying that what I calculated is variance, and what you calculated isn't. In particular, the X in the 3rd term of "X^2 + Xa^2 - 2*X*Xa" is avg(X), not sqrt(avg(X^2)).

[Morte66]

Especially since moving sensitivity is equivalent to changing CRF :)

Hmm. That raises a concern. I'm not 100% sure about this yet, still gnawing at it.

I've been doing some actual encoding (not just tests). I used --aq-sensitivity 10 which seems like a good weighted average -- I got lots of 9 and the occasional 13 when matching sizes.

A lot of the results are excellent. All the results are excellent for the bitrate. But there's an issue...

The stuff that needed sensitivity 13 to match sizes in the testing is coming out quite small at sensitivity 10, and and although it's admirable in terms of blocking/banding and dark/smooth detail it's rather soft and lacking in texture. Woven cotton shirts become smooth polyester. It almost looks like I did an encode at crf 20/21 and chucked in loads of Haali AQ control the blocks/bands. That would need at least 50% more bitrate, which is obviously a score for Varaince AQ.

But the thing is, with Haali's AQ you could say --crf 18 and know it would be sharp enough, and you could say --aq-strength 0.3 --aq-sensitivity 5 and know that whatever blocking/banding came out would be controllable by ffdshow DeBand. You didn't have to inspect the video or run test encodes to find a set of parameters that would give satisfactory quality. I am concerned that this might not be the case with variance AQ.

[Dark Shikari]

We may have found a problem underlying some of the problems with VAQ.

In particular, the qcomp setting already varies frame quantizers based on the SATD score of the frame. So its basically doing what AQ is doing also, in a sense overweighting and underweighting some frames.

I would suggest trying qcomp=1 in the meantime, and seeing whether it improves it or not. We're working on a fix.

In particular, qcomp = 0.6 + AQ strength 0 should have a similar QP-moving effect (on frame level, not block level) as qcomp = 1.0 + AQ strength 0.28 (according to akupenguin's math).

Edit: More information. Qcomp = 1 is equivalent to CQP mode. AQ in a sense does the same job as qcomp, so it would make sense to disable qcomp (set it to 1) when using decently strong AQ. For now, try settings like qcomp=1,aq strength=0.6, and experiment with AQ sensitivity.

[ToS_Maverick]

here you go again, some results:

Black.Pearl.Sample test crf 18 aq00 qcomp10.mkv 48,8 MB
Black.Pearl.Sample test crf 18 aq06 sens12 qcomp10.mkv 53,3 MB
Black.Pearl.Sample test crf 18 aq10 sens11 qcomp10.mkv 53,6 MB

Black.Pearl.Sample test crf 18 aq00 qcomp06.mkv 35,3 MB
Black.Pearl.Sample test crf 18 aq06 sens09 qcomp10.mkv 37,1 MB
Black.Pearl.Sample test crf 18 aq10 sens09 qcomp10.mkv 36,9 MB

with the 37 MB encodes, 0.6 is better in the foreground, 1.0 is better in the background, visually. 0.6 delivers a better SSIM

[Hellworm]

So Qcomp = 1 does constant quant and AQ with sensivity = high then does constant quality, completely ignoring the bitrate? Is this correct? I sort of followed the thread, but didn't have much time to test and I'd find a "real" constant quality very useful, as crf often doesn't produce a constant quality, even with Qcomp = 1

Edit: read again and high sensivity seems to mean ignore bitrate, so whats the maximum sane sensivity?

[DeathTheSheep]

And I was right all along, practically. QP is best for VAQ, is what I said. :)

Anyway, I got satd/tesa and me-prepass working with 721. But I have a question about prepass (optimization) I'd like to discuss with you, Dark Shikari (read: Good Morning! You've got mail!). :D

And how goes adding the so-called erroneous "rounding constant" back in? Like I've said, for anime, 0.47 doesn't help where it should, instead throwing bits at frames which clearly don't need them. See gabext.com/samples/.

[Dark Shikari]

So Qcomp = 1 does constant quant and AQ with sensivity = high then does constant quality, completely ignoring the bitrate? Is this correct? I sort of followed the thread, but didn't have much time to test and I'd find a "real" constant quality very useful, as crf often doesn't produce a constant quality, even with Qcomp = 1

Edit: read again and high sensivity seems to mean ignore bitrate, so whats the maximum sane sensivity?Ignoring clipping of QPs, sensitivity rising is equivalent to CRF dropping. As I said, try AQ strength 0.5-0.6, AQ sensitivity 15, qcomp 1.

[Morte66]

All range tests, qcomp 1.0 with levels/deblock/denoise/deband...

Entourage 211 DVD

crf 18 no AQ: 17.7MB
crf 18 VAQ strength 1.0 sensitivity 9: 12.5 MB
crf 18 VAQ strength 1.0 sensitivity 13: 23.2 MB
crf 18 VAQ strength 1.0 sensitivity 11: 18.1 MB
crf 18 VAQ strength 0.6 sensitivity 11: 15.8 MB
{crf 18 qcomp 0.6 no AQ: 14.4 MB}

Deadwood 307 DVD (unusually dark and shadowy episode)

crf 18 no AQ: 8.81MB
crf 18 VAQ strength 1.0 sensitivity 11: 14.1 MB
crf 18 VAQ strength 1.0 sensitivity 9: 10.3 MB
crf 18 VAQ strength 1.0 sensitivity 8: 8.52 MB
crf 18 VAQ strength 0.6 sensitivity 8: 7.75 MB
{crf 18 qcomp 0.6 no AQ: 12.7 MB}


Trois Couleurs Blanc DVD

crf 18 no AQ: 37.2 MB
crf 18 VAQ strength 1.0 sensitivity 8: 28.0 MB
crf 18 VAQ strength 1.0 sensitivity 10: 45.9 MB
crf 18 VAQ strength 1.0 sensitivity 9: 37.0 MB
crf 18 VAQ strength 0.6 sensitivity 9: 29.5 MB
{crf 18 qcomp 0.6 no AQ: 35.7 MB}


Irreversible DVD

crf 18 no AQ: 44.1 MB
crf 18 VAQ strength 1.0 sensitivity 9: 47.0 MB
crf 18 VAQ strength 1.0 sensitivity 8: 36.6 MB
crf 18 VAQ strength 0.6 sensitivity 9: 39.1 MB
{crf 18 qcomp 0.6 no AQ: 42.6 MB}


Battlestar Galactica Razor DVD

crf 18 no AQ: 70.0 MB
crf 18 VAQ strength 1.0 sensitivity 9: 66.5 MB
crf 18 VAQ strength 1.0 sensitivity 10: 82.4 MB
crf 18 VAQ strength 0.6 sensitivity 9: 58.8 MB
{crf 18 qcomp 0.6 no AQ: 68.0 MB}


The Magic Mile MPEG2 SDTV

crf 18 no AQ: 22.2 MB
crf 18 VAQ strength 1.0 sensitivity 9: 18.4 MB
crf 18 VAQ strength 1.0 sensitivity 10: 23.1 MB
crf 18 VAQ strength 0.6 sensitivity 10: 19.6 MB
{crf 18 qcomp 0.6 no AQ: 18.5 MB}


Nine with outliers again.

[Dark Shikari]

How about comparing to no-AQ CRF *with* qcomp on default? (i.e. qcomp = 0.6)

[Morte66]

How about comparing to no-AQ CRF *with* qcomp?

{added}

{busy for next ~20 hours}

[Jawed]

Sigh, ignore, comprehension...

Jawed

[Yoshiyuki Blade]

I have a n00b question. What does lowerering the AQ strength do besides create a smaller file size at a given sensitivity? lol. Say from 1.0 to the (now suggested) 0.5-0.6. There's not much I know beyond that.

[Dark Shikari]

I have a n00b question. What does lowerering the AQ strength do besides create a smaller file size at a given sensitivity? lol. Say from 1.0 to the (now suggested) 0.5-0.6. There's not much I know beyond that.Because the new VAQ redistributes quantizers; unlike the old AQ, it doesn't just lower them. On many sources, this means a decrease in file size.

[DeathTheSheep]

Strength 0.5 to 0.6 is now recommended for 0.47? Looks like I missed something here. ;)

[Yoshiyuki Blade]

Strength 0.5 to 0.6 is now recommended for 0.47? Looks like I missed something here. ;)

I wouldn't say recommended, but DS has suggested it a few posts ago, as well as setting qcomp to 1 for the time being. Unfortunately, it's gonna be hours till I get some results O_O!

[Sagekilla]

Because the new VAQ redistributes quantizers; unlike the old AQ, it doesn't just lower them. On many sources, this means a decrease in file size.

See, in my eyes AQ was an algorithm that rearranged quants in a frame to get the best quality, not one that would just "lower quants" like Haali's AQ does.


On a side note, if qcomp is set to 1 and aq-strength is varied, isn't that just making AQ the rate control algorithm?

[akupenguin]

Yes. Static-mode AQ is very similar algorithm-wise to CRF, it just uses a different complexity metric and applies to MBs.

[fields_g]

I have a n00b question. What does lowerering the AQ strength do besides create a smaller file size at a given sensitivity? lol. Say from 1.0 to the (now suggested) 0.5-0.6. There's not much I know beyond that.

I want anyone to feel free to correct me.

FIRST think of Sensitivity.... What level of flatness should be considered necessary adjustment. THEN.. knowing what needs to be changed, Strength determines how radical the adjustment is.

Higher Sensitivity, more should be fixed.
Higher Strength, more radically the attempt to fix it.
This is why while lowering one of these and raising the other, you can get a file with the same bitrate.

[Yoshiyuki Blade]

From what I've tested so far, AQ on anime look much better with strong settings (such as --aq-strength 1.0 --aq-sensitivity 20). Even though it makes file sizes large at a given crf, it still looks nice at smaller file sizes too, though doing that will compromise overall image quality.

Setting qcomp 1, aq str 0.5, and sensitivity 20 looks nice, and the file size is small, but dark areas don't get improved much in comparison. I'm going to re-run the same settings but change strength to 1.0 and see how it turns out. Sorry that I can't post numbers, I'm unorganized and everything's a mess :D. I'll try and get a consistent order going.

[Morte66]

Slight diversion from crf: any thoughts about qcomp and sensitivity for 2-pass encode to a target size with 0.47?

[Dark Shikari]

Slight diversion from crf: any thoughts about qcomp and sensitivity for 2-pass encode to a target size with 0.47?qcomp = 1, sensitivity = 10-15? Sensitivity is much less meaningful with 2pass.

[burfadel]

I did some tests on 'A knight's tale' and 'the 6th day', on both occasions I found the best aq strength and sensitivity, in regards to these two clips as strength 0.6 and senstivity 17.6 to maintain the same file size (very slightly larger, I mean by about 0.5 percent) and be of good quality. I didn't set qcomp.

Again, setting the sensitivity lower than that, even with qcomp being 1, the filesize was way undersized. The quality was also noticeable lower, but was about right in terms of its file size.

Setting aq strength to 1 (auto sens.) with the same clips ended up with doubling the file size.

I keep ending up with a sensitivity of around 17 for strengths of say, 0.6, and slightly lower at around 15 or 16 for strength 1

[Dark Shikari]

Before I head out for the day, let me throw up a concept to be shot down (in the full knowledge that I've no idea what I'm talking about). This describes functionality, not necessarily algorithm:

The encoder chugs along, frame after frame. Within each individual frame, it calculates a spatial bitrate distribution according to the new Variance AQ's definition of quality. Then at the whole frame level, it measures the quality according to the vanilla CRF metric, the one that's not PSNR or SSIM but "implicit in the algorithm", and varies the whole-frame bit budget to hit a quality level measured that way. So VAQ does spatial bitrate distribution within frames, and non-AQ does temporally varying bitrate allocation across frames.

I wonder if that would give the excellent smooth/dark area handling by using VAQ spatially within frames, then make sure the frames are crisp enough by using the current CRF metric (which seems more dependable re crispness than VAQCRF 0.47).Using CRF for frame QP distribution and VAQ for within-frame bit distribution can be done using --aq-sensitivity 0 and adjusting qcomp to however strong you want CRF to be (lower = stronger) ;)

[Morte66]

Using CRF for frame QP distribution and VAQ for within-frame bit distribution can be done using --aq-sensitivity 0 and adjusting qcomp to however strong you want CRF to be (lower = stronger) ;)

I thought that approach had a problem on frames with a small amount of sharp detail that gets clobbered by variance AQ to feed the large amount of smooth stuff?

I was thinking particularly of CRF measuring quality after VAQ has done spatial bitrate distribution, so it would feed extra bitrate in from the top to resharpen the sharp bits of frames like that. Or is that what it does anyhow?

{I seem to have deleted my original post by accident (!), but DS quoted it whole.}

[Jawed]

Yes. Static-mode AQ is very similar algorithm-wise to CRF, it just uses a different complexity metric and applies to MBs.
Should VAQ constitute an x264 mode alongside crf, qp, bitrate, pass ?

Jawed

[Dark Shikari]

Should VAQ constitute an x264 mode alongside crf, qp, bitrate, pass ?

JawedWell, VAQ works with all of those modes (though, technically, qcomp=1 and CRF is equivalent to QP mode)... so not really, I think.

[Jawed]

A few months back I experimented with qcomp in conjunction with HAQ (pre-processed to PC Levels + MVDegrain2). It certainly produced interesting results:


AQ 0.3 5 qcomp 0 qcomp 1 source bitrate
24 1827.9 2272.1 1621.0 4762.7
Black Pearl 2270.7 1671.9 2924.5 3639.8
Carnivale 2485.2 2115.3 2942.0 6623.8
Hana & Alice 1 2102.7 2557.1 1958.2 4015.5
Monk 3207.5 2132.0 4358.8 8354.6
War of the Worlds 6829.3 3047.2 11542.4 7059.3
X-Files 1951.0 2275.5 1693.9 4821.1
Average 2953.5 2295.9 3863.0 5611.0


Note all three test encodes are with the same HAQ setting, --aq-strength 0.3 --aq-sensitivity 5. The other x264 settings were:

--crf 16 --ref 5 --mixed-refs --no-fast-pskip --bframes 8 --b-pyramid --b-rdo --bime --weightb --subme 6 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 32 --no-dct-decimate

qcomp is a bit surprising. Maybe you guys are used to it.

I've attached the spreadsheet containing a bit more detail.

Jawed

[bob0r]

AQ patch 0.47: http://akuvian.org/src/x264/x264_aq_var.47.diff

x264.735.dark.aq.0.47.exe (http://files.x264.nl/AQ/x264.735.dark.aq.0.47.exe) (pthreads/mp4 = yes, made with make fprofiled)

Should work i guess, patch did apply.

[nurbs]

I did some encodes with qcomp = 1. With aq-strenght 0.5 and sensitivity 10 it gives me similar filesizes to my previous encodes with normal qcomp and without aq. The videos look very good.

[Yoshiyuki Blade]

After some more testing, older anime with lots of noise and details look better (in my opinion) with high CRF and high AQ strength vs low CRF and low AQ strength (at a fixed sensitivity). Both results have their ups and downs.

-Low CRF and Low AQ strength gives a cleaner overall picture as expected, AQ but hardly seems to address ugly blocking in dark areas.

-High CRF and High AQ strength results in a worse overall image quality (as expected), but dark areas look much better.

This trend happens at lower bitrates. At high bitrates, the results look very similar to each other (very good quality across the board).

I'm going to edit this post in a couple hours with some examples. In the meantime, this is what I have so far:

--crf 28.0 --qcomp 1 aq-strength 0.5 aq-sensitivity 20: 188 MB (Sample 1 (http://www.mediafire.com/?2axiwywmdzm)) (Sample 2 (http://www.mediafire.com/?49ne1jndn61))
--crf 33.0 --qcomp 1 aq-strength 1.0 aq-sensitivity 20: 199 MB (Sample 1 (http://www.mediafire.com/?djnj12he71x)) (Sample 2 (http://www.mediafire.com/?e6yj4kjy1jz))
--crf 34.0 --qcomp 1 aq-strength 1.0 aq-sensitivity 20: 161 MB

Update: Uploaded the samples. To be honest, none of them are satisfactory lol, but I expect them to look much better in a 2-pass encode. I did a 2-pass encode a while ago (before patch .47) at lower bitrates than those, and it still looks better than both.

I'm currently re-running tests with 2-pass at a higher bitrate not only to keep the final filesizes very close to each other (there's an 11 MB bias in the tests above), but to narrow the gap on the differences between high CRF/high AQ strength vs low CRF/low AQ strength. The next set of results will look much easier on the eyes.

[Raere]

Are low AQ values recommended as a general psychovisual improvement at this point, or is it still just a tradeoff for lower overall quality for better dark areas?

[Dark Shikari]

Are low AQ values recommended as a general psychovisual improvement at this point, or is it still just a tradeoff for lower overall quality for better dark areas?VAQ is supposed to be a general psychovisual improvement, period. Too high AQ strengths may backfire though.

[DeathTheSheep]

Does anyone know where I can find 0.46? It seems to have vanished, and I'd like to upgrade from 0.45.

[edit] w007

- var += ssd - (sad * sad >> (i?6:8));
+ var += ssd - ((sad/16) * (sad/16));


I win. :p

Y4yz 4 err0r-ous (0[)3 |=+\X/

[burfadel]

Anyone else notice that in general, the higher the input resolution he lower the sensitivity required? So a sensitivity of say, 9 is good for 1920x1080, whereas for much lower resolutions a 16 or 17 etc is required? Also the strength affects whether a higher or lower sensitivity is more beneficial.

That makes a fixed sensitivity almost impossible, maybe an automatic sensitivity that can adjust to the input resolution and AQ strength?

[Dark Shikari]

Does anyone know where I can find 0.46? It seems to have vanished, and I'd like to upgrade from 0.45.

[edit] w007

- var += ssd - (sad * sad >> (i?6:8));
+ var += ssd - ((sad/16) * (sad/16));


I win. :p

Y4yz 4 err0r-ous (0[)3 |=+\X/That will fail because i?6:8 compensates for the different sizes of the SSDs in different planes, while your code does not.
Anyone else notice that in general, the higher the input resolution he lower the sensitivity required? So a sensitivity of say, 9 is good for 1920x1080, whereas for much lower resolutions a 16 or 17 etc is required?

That makes a fixed sensitivity almost impossible, maybe an automatic sensitivity that can adjust to the input resolution?This reflects the fact that lower resolutions generally have higher average block variances, since there are more edges and fewer large areas of flat detail.

[DeathTheSheep]

Do you have a link to 0.46? :)

If not, how do I add this compensation?

[Dark Shikari]

Do you have a link to 0.46? :)

If not, how do I add this compensation?Divide by 8 for chroma planes, 16 for luma planes.

[DeathTheSheep]

Divide which part? The whole expression, the (sad/16)^2 chunk only, or...? And how can I test with a neat little "colon question mark" if (like something below)?

var += ssd - ((sad/16) * (sad/16)) / (i?8:16);


Is this even the right syntax for such a trick? And is there a neat bitshift >> or whatnot for "divide by 16"? Admittedly, I know very little when it comes to syntax/speed notation.

[burfadel]

This reflects the fact that lower resolutions generally have higher average block variances, since there are more edges and fewer large areas of flat detail.

I found that:
--aq-strength 0.3 --aq-sensitivity 16.5
Seems to maintain an almost identical file size to not having AQ on for both very low res and Full HD encodes, although strength 0.3 is very little. With even a strength of 0.6 the difference is too great between low and high resolutions to the normal, non AQ builds. - that is, what works for high res doesn't work for low res and vice versa.

[akupenguin]

var += ssd - ((sad/16) * (sad/16)) / (i?8:16);

No, and even if you corrected it it would just bring back the imprecision.
The compensation would be:
- unsigned int var=0
+ unsigned int var=42
for some value of 42. Or better yet, add it after the computation, so that the satd shortcut still works.

[CruNcher]

hmm is it possible that the Visual Quality decreased? i see ringing now @ Face Edges (something i saw last time with ASP but not used to see with H.264) aq-strength of 1.0 only lower strength avoid this now.

[burfadel]

hmm is it possible that the Visual Quality decreased? i see ringing now @ Face Edges (something i saw last time with ASP but not used to see with H.264) aq-strength of 1.0 only lower strength avoid this now.

Have you checked the file size with and without AQ? I found at a strength of 1 NO senstivity suited all scenarios. There was a very large difference in terms of comparing it with its non-AQ counterpart between HD and resolution such as 512x384. What sensitivity at that strength works for one DOES NOT work for the other.

Even at a strength of 0.6, the sensitivities made a large difference with different resolutions. The only figure that worked perfectly for low and high resolutions for me, was strength 0.3 and sensitivity 16.5. (16 was giving a slightly lower file size, 16.5 fractionally larger, which is ok!).

It did help in the dark areas still, although its very weak at those settings, although I'd have to agree with you about strength 1, it did make the faces etc a bit less in terms of quality. That doesn't occur with strength 0.3. It was as if I raised the CRF in a way... Problem is to have an ideal quality in the dark areas, taking bits away from bright areas will lower its quality, and it seems having a strong AQ takes too much away from the bright areas. I'm not even sure that it maintains a CRF of say 24, with aq enabled at strength 1, it lowers the bright area quality and the filesize can become way undersized - or oversized!

Try strength 0.3 and sensitivity 16.5 and see whether that helps you like it did with me :) - and its ideal for both very low res and full HD

[DeathTheSheep]

Since the one posted here earlier wasn't at all valid, here's my updated version:
Updated me-prepass (unoptimized):
--- common/common.c Sun Jan 27 13:39:08 2008
+++ common/common.c Sun Jan 27 13:39:08 2008
@@ -441,6 +441,8 @@
p->analyse.i_mv_range_thread = atoi(value);
OPT2("subme", "subq")
p->analyse.i_subpel_refine = atoi(value);
+ OPT2("me-prepass", "meprepass")
+ p->analyse.i_me_prepass = atobool(value);
OPT("bime")
p->analyse.b_bidir_me = atobool(value);
OPT("chroma-me")
@@ -879,6 +881,7 @@
s += sprintf( s, " analyse=%#x:%#x", p->analyse.intra, p->analyse.inter );
s += sprintf( s, " me=%s", x264_motion_est_names[ p->analyse.i_me_method ] );
s += sprintf( s, " subme=%d", p->analyse.i_subpel_refine );
+ s += sprintf( s, " me-prepass=%d", p->analyse.i_me_prepass );
s += sprintf( s, " brdo=%d", p->analyse.b_bframe_rdo );
s += sprintf( s, " mixed_ref=%d", p->analyse.b_mixed_references );
s += sprintf( s, " me_range=%d", p->analyse.i_me_range );
--- x264.c Sun Jan 27 13:39:08 2008
+++ x264.c Sun Jan 27 13:39:08 2008
@@ -232,7 +232,8 @@
H1( " --mvrange-thread <int> Minimum buffer between threads [-1 (auto)]\n" );
H0( " -m, --subme <integer> Subpixel motion estimation and partition\n"
" decision quality: 1=fast, 7=best. [%d]\n", defaults->analyse.i_subpel_refine );
- H0( " --b-rdo RD based mode decision for B-frames. Requires subme 6.\n" );
+ H0( " --me-prepass Run an ME prepass on predictors. Requires subme 3 or higher.\n");
+ H0( " --b-rdo RD based mode decision for B-frames. Requires subme 6 or higher.\n" );
H0( " --mixed-refs Decide references on a per partition basis\n" );
H1( " --no-chroma-me Ignore chroma in motion estimation\n" );
H1( " --bime Jointly optimize both MVs in B-frames\n" );
@@ -398,6 +399,7 @@
{ "mvrange", required_argument, NULL, 0 },
{ "mvrange-thread", required_argument, NULL, 0 },
{ "subme", required_argument, NULL, 'm' },
+ { "me-prepass", no_argument, NULL, 0 },
{ "b-rdo", no_argument, NULL, 0 },
{ "mixed-refs", no_argument, NULL, 0 },
{ "no-chroma-me", no_argument, NULL, 0 },
--- x264.h Sun Jan 27 13:39:08 2008
+++ x264.h Sun Jan 27 13:39:08 2008
@@ -220,6 +220,7 @@
int i_mv_range; /* maximum length of a mv (in pixels). -1 = auto, based on level */
int i_mv_range_thread; /* minimum space between threads. -1 = auto, based on number of threads. */
int i_subpel_refine; /* subpixel motion estimation quality */
+ int i_me_prepass; /* run an ME prepass on predictors */
int b_bidir_me; /* jointly optimize both MVs in B-frames */
int b_chroma_me; /* chroma ME for subpel and mode decision in P-frames */
int b_bframe_rdo; /* RD based mode decision for B-frames */
--- encoder/me.c Sun Jan 27 13:39:08 2008
+++ encoder/me.c Sun Jan 27 20:25:50 2008
@@ -69,6 +69,23 @@
COPY3_IF_LT( bpred_cost, cost, bpred_mx, mx, bpred_my, my ); \
}

+#define COST_MV_HPEL2( mx, my, cost ) \
+{ \
+ int stride = 16; \
+ uint8_t *src = h->mc.get_ref( pix, &stride, m->p_fref, m->i_stride[0], mx, my, bw, bh ); \
+ cost = h->pixf.fpelcmp[i_pixel]( m->p_fenc[0], FENC_STRIDE, src, stride ) \
+ + p_cost_mvx[ mx ] + p_cost_mvy[ my ]; \
+}
+
+#define COST_MV_HPEL3( mx, my) \
+{ \
+ int stride = 16; \
+ uint8_t *src = h->mc.get_ref( pix, &stride, m->p_fref, m->i_stride[0], mx, my, bw, bh ); \
+ int cost = h->pixf.fpelcmp[i_pixel]( m->p_fenc[0], FENC_STRIDE, src, stride ) \
+ + p_cost_mvx[ mx ] + p_cost_mvy[ my ]; \
+ COPY3_IF_LT( bestcost, cost, bestx, mx, besty, my ); \
+}
+
#define COST_MV_X3_DIR( m0x, m0y, m1x, m1y, m2x, m2y, costs )\
{\
uint8_t *pix_base = p_fref + bmx + bmy*m->i_stride[0];\
@@ -171,8 +188,13 @@
int mv_y_min = h->mb.mv_min_fpel[1];
int mv_x_max = h->mb.mv_max_fpel[0];
int mv_y_max = h->mb.mv_max_fpel[1];
+ int mv_x_min4 = h->mb.mv_min_fpel[0]<<2;
+ int mv_y_min4 = h->mb.mv_min_fpel[1]<<2;
+ int mv_x_max4 = h->mb.mv_max_fpel[0]<<2;
+ int mv_y_max4 = h->mb.mv_max_fpel[1]<<2;

#define CHECK_MVRANGE(mx,my) ( mx >= mv_x_min && mx <= mv_x_max && my >= mv_y_min && my <= mv_y_max )
+#define CHECK_MVRANGE4(mx,my) ( mx >= mv_x_min4 && mx <= mv_x_max4 && my >= mv_y_min4 && my <= mv_y_max4 )

const int16_t *p_cost_mvx = m->p_cost_mv - m->mvp[0];
const int16_t *p_cost_mvy = m->p_cost_mv - m->mvp[1];
@@ -183,19 +205,88 @@
pmy = ( bmy + 2 ) >> 2;
bcost = COST_MAX;

- /* try extra predictors if provided */
- if( h->mb.i_subpel_refine >= 3 )
- {
- COST_MV_HPEL( bmx, bmy );
- for( i = 0; i < i_mvc; i++ )
+ /* try extra predictors if provided */
+ if( h->mb.i_subpel_refine >= 3 )
+ {
+ COST_MV_HPEL( bmx, bmy );
+ if(!h->param.analyse.i_me_prepass)
+ {
+ for( i = 0; i < i_mvc; i++ )
+ {
+ const int mx = x264_clip3( mvc[i][0], mv_x_min*4, mv_x_max*4 );
+ const int my = x264_clip3( mvc[i][1], mv_y_min*4, mv_y_max*4 );
+ if( mx != bpred_mx || my != bpred_my )
+ COST_MV_HPEL( mx, my );
+ }
+ }
+ else
{
- int mx = mvc[i][0];
- int my = mvc[i][1];
- if( (mx | my) && ((mx-bmx) | (my-bmy)) )
+ for( i = 0; i < i_mvc; i++ )
{
- mx = x264_clip3( mx, mv_x_min*4, mv_x_max*4 );
- my = x264_clip3( my, mv_y_min*4, mv_y_max*4 );
- COST_MV_HPEL( mx, my );
+ const int mx = x264_clip3( mvc[i][0], mv_x_min*4, mv_x_max*4 );
+ const int my = x264_clip3( mvc[i][1], mv_y_min*4, mv_y_max*4 );
+ int doSearch = 1;
+ int j;
+ for(j = 0; j < i; j++)
+ {
+ if(mvc[i][0] == mvc[j][0] && mvc[i][1] == mvc[j][1]) doSearch = 0;
+ }
+ if( ( mx != bpred_mx || my != bpred_my ) && doSearch)
+ {
+ int bestcost;
+ int bestx = mx;
+ int besty = my;
+ COST_MV_HPEL2( mx, my, bestcost );
+ COPY3_IF_LT( bpred_cost, bestcost, bpred_mx, bestx, bpred_my, besty );
+ if(bestcost < 2*bpred_cost)
+ {
+ int n;
+ int dir = -2;
+ COST_MV_HPEL2(bestx-4,besty,costs[0]);
+ COST_MV_HPEL2(bestx-2,besty+4,costs[1]);
+ COST_MV_HPEL2(bestx+2,besty+4,costs[2]);
+ COST_MV_HPEL2(bestx+4,besty,costs[3]);
+ COST_MV_HPEL2(bestx+2,besty-4,costs[4]);
+ COST_MV_HPEL2(bestx-2,besty-4,costs[5]);
+ COPY2_IF_LT( bestcost, costs[0], dir, 0 );
+ COPY2_IF_LT( bestcost, costs[1], dir, 1 );
+ COPY2_IF_LT( bestcost, costs[2], dir, 2 );
+ COPY2_IF_LT( bestcost, costs[3], dir, 3 );
+ COPY2_IF_LT( bestcost, costs[4], dir, 4 );
+ COPY2_IF_LT( bestcost, costs[5], dir, 5 );
+ if( dir != -2 )
+ {
+ static const int hex2[8][2] = {{-2,-4}, {-4,0}, {-2,4}, {2,4}, {4,0}, {2,-4}, {-2,-4}, {-4,0}};
+ bestx += hex2[dir+1][0];
+ besty += hex2[dir+1][1];
+ for( n = 1; n < i_me_range && CHECK_MVRANGE4(bestx, besty); n++ )
+ {
+ static const int mod6[8] = {5,0,1,2,3,4,5,0};
+ const int odir = mod6[dir+1];
+ COST_MV_HPEL2(hex2[odir+0][0]+bestx,hex2[odir+0][1]+besty,costs[0]);
+ COST_MV_HPEL2(hex2[odir+1][0]+bestx,hex2[odir+1][1]+besty,costs[1]);
+ COST_MV_HPEL2(hex2[odir+2][0]+bestx,hex2[odir+2][1]+besty,costs[2]);
+ dir = -2;
+ COPY2_IF_LT( bestcost, costs[0], dir, odir-1 );
+ COPY2_IF_LT( bestcost, costs[1], dir, odir );
+ COPY2_IF_LT( bestcost, costs[2], dir, odir+1 );
+ if( dir == -2 )
+ break;
+ bestx += hex2[dir+1][0];
+ besty += hex2[dir+1][1];
+ }
+ }
+ COST_MV_HPEL3(bestx+2,besty-2);
+ COST_MV_HPEL3(bestx+2,besty);
+ COST_MV_HPEL3(bestx+2,besty+2);
+ COST_MV_HPEL3(bestx,besty-2);
+ COST_MV_HPEL3(bestx,besty+2);
+ COST_MV_HPEL3(bestx-2,besty-2);
+ COST_MV_HPEL3(bestx-2,besty);
+ COST_MV_HPEL3(bestx-2,besty+2);
+ COPY3_IF_LT(bpred_cost,bestcost,bpred_mx,bestx,bpred_my,besty);
+ }
+ }
}
}
bmx = ( bpred_mx + 2 ) >> 2;

Or better yet, add it after the computation, so that the satd shortcut still works.
After the computation in line 69, like this?
var += ssd - (sad * sad >> (i?6:8)) + k;

or just:

var += ssd - (sad * sad >> (i?6:8));
// SATD to represent the block's overall complexity (bit cost) for intra encoding.
// exclude the DC coef, because nothing short of an actual intra prediction will estimate DC cost.
if( var && satd )
*satd += h->pixf.satd[pix](flat, 0, h->fenc->plane[i]+offset, stride) - sad/2;
+ var += 42;
}
return var;

where k=42, an arbitrary constant?

[edit]
Your code above (var=42) made almost no difference in quality, SSIM, or QP distribution, and certainly didn't reinstate the behavior of .45, for better or for worse mathematically, you might say, though visually worse in my case. At constant QP, the bitrate/threshold ratio also remains the same; the algorithm still requires a vastly lower threshold to reach the same bitrate as .45 at less SSIM and quality--same as normal .47--a sure sign it's not doing what it's intended to.

At the risk of reintroducing imprecision (and what tremendously vast imprecision it must be to produce what in some cases amounts to such polar opposite bit distributions), I still seek to recreate a rounding/distribution similar to that present in .45, but perhaps, if possible, more mathematically accurate behavior.

[sysKin]

Lol, as much as 42 is the answer to everything, it's not the answer here. It's in the order of several thousands I'm guessing.

[zbutsam]

hmm is it possible that the Visual Quality decreased? i see ringing now @ Face Edges (something i saw last time with ASP but not used to see with H.264) aq-strength of 1.0 only lower strength avoid this now.

I have to report the same thing. I did a 2-pass encode with aq-strength at 1.0 and default sensitivity and in the video with AQ enabled there was a lot more ringing around people's faces than the one without AQ. Perhaps the bitrate was too low for my source and where the No-AQ version became blurred, the AQ version produced ringing.

[Dark Shikari]

I have to report the same thing. I did a 2-pass encode with aq-strength at 1.0 and default sensitivity and in the video with AQ enabled there was a lot more ringing around people's faces than the one without AQ. Perhaps the bitrate was too low for my source and where the No-AQ version became blurred, the AQ version produced ringing.AQ 1.0 is too strong. Use the recommended settings: strength 0.5, sensitivity 13, qcomp 1.

[DeathTheSheep]

Lol, as much as 42 is the answer to everything, it's not the answer here. It's in the order of several thousands I'm guessing.

Hmm, true that. :) But after all, I've tried on the order of up to 50000 already to limited success. Very high values do skew QP distribution indeed (throwing in any sizable constant into any rate control mechanism would almost have to), but trying to guess at some random, arbitrary constant isn't quite optimal unless I know exactly what constant .45 did introduce, and how exactly that supposed "constant" actually varies per frame/decision.

[zbutsam]

AQ 1.0 is too strong. Use the recommended settings: strength 0.5, sensitivity 13, qcomp 1.

:thanks:

I tried again with the settings you recommended and everything looks in order now.
Could you please add these new recommended settings on the first post of this thread?

[Dark Shikari]

:thanks:

I tried again with the settings you recommended and everything looks in order now.
Could you please add these new recommended settings on the first post of this thread?Done.

[Poopoo]

Can you please tell me where I can get this SSIM thingy ?
Thanks !

[Sagekilla]

SSIM calculation is built into x264. In fact, x264 calculates it by default. You have to use --no-ssim to disable it. When your encode finishes, it'll display the SSIM along with a bunch of other data.

[Yoshiyuki Blade]

AQ 1.0 is too strong. Use the recommended settings: strength 0.5, sensitivity 13, qcomp 1.

I've tried several different levels of AQ strength and sensitivity, but I still can't seem to reproduce the same quality as I did with v0.45 whether its in CRF mode or 2-pass (anime). High strength and high sensitivity is the closest I've gotten, but the overall visual quality turns out so bad, it's not a worthwhile tradeoff anymore. With v0.45 I used strength 1.0, sensitivity 20, and qcomp 0.6, and the improvement in dark areas was definitely worth the cost of some overall quality. It didn't look nearly as bad as it does now.

Using the current recommended settings (strength 0.5, sensitivity 13-15, qcomp 1), the quality is good, but dark areas don't get addressed very well if at all.

At the moment, nothing looks remotely satisfying unless I really up the bitrates (about 500 megabytes for 25-minute footage), in which case it will look good across the board. Perhaps this version of AQ is more optimized for real-life footage?

However, before I feel confident on my experiences, I'm gonna have to do another encode with v0.45. Be back in like 10 hours with my experiences lol :D.

[Dark Shikari]

I've tried several different levels of AQ strength and sensitivity, but I still can't seem to reproduce the same quality as I did with v0.45 whether its in CRF mode or 2-pass (anime). High strength and high sensitivity is the closest I've gotten, but the overall visual quality turns out so bad, it's not a worthwhile tradeoff anymore. With v0.45 I used strength 1.0, sensitivity 20, and qcomp 0.6, and the improvement in dark areas was definitely worth the cost of some overall quality. It didn't look nearly as bad as it does now.

Using the current recommended settings (strength 0.5, sensitivity 13-15, qcomp 1), the quality is good, but dark areas don't get addressed very well if at all.

At the moment, nothing looks remotely satisfying unless I really up the bitrates (about 500 megabytes for 25-minute footage), in which case it will look good across the board. Perhaps this version of AQ is more optimized for real-life footage?

However, before I feel confident on my experiences, I'm gonna have to do another encode with v0.45. Be back in like 10 hours with my experiences lol :D.That's very odd.

Can you encode one clip with 0.45 at settings you like, and at 0.47 at the same bitrate with settings you like, and post both .h264 files here for me to look at? Upload them to Mediafire or something.

[DeathTheSheep]

No, not really odd at all. Why do you think I'm trying to "get back" the old behavior? Nostalgic reasons? ;)

I still can't seem to reproduce the same quality as I did with v0.45

Yep, that's exactly my point. Good to see that people are reaching the same conclusion on wider scales. :) That's why I'm back to 0.46, which works like a dream. If you're interested, adding a constant 17,000 after the SATD "trick" in the ac_energy_mb function seems to bring things back to the bitrate/threshold ballpark, but still doesn't allocate bits where they're needed in anime, so the quality, long story short, will still suck.

VAQ < 0.47 was a freak occurrence of an erroneous miracle-rounding, which, for some yet-unknown reason, worked amazingly well on anime. Odd, isn't it?

[Yoshiyuki Blade]

That's very odd.

Can you encode one clip with 0.45 at settings you like, and at 0.47 at the same bitrate with settings you like, and post both .h264 files here for me to look at? Upload them to Mediafire or something.

Will do. Is this (http://files.x264.nl/AQ/force.php?file=./x264.721.dark.aq.rdrc.0.45.exe) the old v0.45 patch? I hope it wasn't ninja-edited over time or anything. ;)

[Dark Shikari]

Will do. Is this (http://files.x264.nl/AQ/force.php?file=./x264.721.dark.aq.rdrc.0.45.exe) the old v0.45 patch? I hope it wasn't ninja-edited over time or anything. ;)Probably not, since that's what it says.

[DeathTheSheep]

Ninja-edited? :eek:

You could use .46, which is more cleaned up, streamlined, and compatible with latest x264 revisions: http://pastebin.com/f22a03c76

I found this baby myself, by the way. It was a long, painful search. :D

[Yoshiyuki Blade]

In the meantime, I've uploaded clips of an older encode I had lying around as reference, which I'm confident used v0.45 of the patch.

To roughly illustrate my impressions, I'll compare the older clip to the results I posted earlier in this thread here (http://forum.doom9.org/showthread.php?p=1092791#post1092791).

(Patch v0.45) --qcomp 0.6 aq-strength 1.0 aq-sensitivity 20: 179 MB (Sample 1 (http://www.mediafire.com/?dfumwzjee9z)) (Sample 2 (http://www.mediafire.com/?fmnbtj1yuhw))

Keep in mind the descrepancy in this comparison:

- For one, this older sample is the result of a 2-pass encode, while the others were done in CRF mode. I think the average CRF for this sample is about 32.53.
- Besides the settings I listed in the description, there may be other differences in the settings used. If there is, it's probably minor. EDIT: One other major difference I spotted is that Sharktooth's AVC CQM was used on the older encode while the others didn't use any (flat), though I doubt it could make this drastic of a difference.
- The file sizes between the 3 tests are very different from each other. However, this older sample happens to be the smallest in file size and still looks subjectively better (in my opinion at least).

The overal impression of the older clip seems more well-balanced. Low strength in v0.47 looks better where the details are prominent, but the background areas dont look very good. Higher strength (1.0), looks similar to the v0.45 version, but the overall impression is worse, and this is considering its 20 MB larger than the v0.45 sample. Pay special attention to Yumi's face 11 seconds into the 2nd sample. It looks best with v0.47 strength 0.5, but the dark scenes look like crap. Second best is the v0.45 sample, which also addressed blockyness in the dark scene very well.

Anyhow, I'm currently re-encoding with v0.45 and will do the same with v0.47 to keep everything more consistent. By the way, do you really need a .264 file? For some stupid reason, I like to encode at 1280x720 even on 4:3 material so the video will be stretched unless I resize them (while putting them into an mkv container) :D

Ninja-edited? :eek:

You could use .46, which is more cleaned up, streamlined, and compatible with latest x264 revisions: http://pastebin.com/f22a03c76

I found this baby myself, by the way. It was a long, painful search. :D

I should check it out. I actually never used v0.46 because it was updated to v0.47 soon afterwards.

[Dark Shikari]

Well, if you give me an MKV, I'll just demux it anyways since Streameye doesn't accept MKVs.

[bob0r]

source:
720p50_parkrun_ter.yuv (ftp://ftp.ldv.e-technik.tu-muenchen.de/pub/test_sequences/720p/720p50_parkrun_ter.yuv)

commandlines:
start /belownormal /b /w x264aq0.48.exe --pass 1 --aq-strength 0.0 --threads auto --bitrate 5000 --deblock 0:0 --bframes 16 --direct auto --me dia --ref 1 --subme 1 --no-dct-decimate --partitions none --progress --fps=50 --output NUL 720p50_parkrun_ter.yuv 1280x720
start /belownormal /b /w x264aq0.48.exe --pass 2 --aq-strength 0.0 --threads auto --bitrate 5000 --deblock 0:0 --bframes 16 --direct auto --b-pyramid --bime --weightb --b-rdo --me umh --ref 5 --mixed-refs --subme 7 --trellis 1 --analyse all --8x8dct --no-fast-pskip --progress --fps=50 --output x264noaq.parkrun.mkv 720p50_parkrun_ter.yuv 1280x720

start /belownormal /b /w x264aq0.48.exe --pass 1 --threads auto --bitrate 5000 --deblock 0:0 --bframes 16 --direct auto --me dia --ref 1 --subme 1 --no-dct-decimate --partitions none --progress --fps=50 --output NUL 720p50_parkrun_ter.yuv 1280x720
start /belownormal /b /w x264aq0.48.exe --pass 2 --threads auto --bitrate 5000 --deblock 0:0 --bframes 16 --direct auto --b-pyramid --bime --weightb --b-rdo --me umh --ref 5 --mixed-refs --subme 7 --trellis 1 --analyse all --8x8dct --no-fast-pskip --progress --fps=50 --output x264aq.parkrun.mkv 720p50_parkrun_ter.yuv 1280x720

result:
AQ strength 0.0(disabled) versus 0.5(default), 50fps 5000kbit:
x264.736.aq.0.48.strenght.0.0.parkrun.5000kbit.50fps.mkv (http://files.x264.nl/AQ/x264.736.aq.0.48.strenght.0.0.parkrun.5000kbit.50fps.mkv)
x264.736.aq.0.48.strenght.0.5.parkrun.5000kbit.50fps.mkv (http://files.x264.nl/AQ/x264.736.aq.0.48.strenght.0.5.parkrun.5000kbit.50fps.mkv)

AQ 0.48 .exe in first post.
changelog: 0.48: AQ strength 0.5, sensitivity 13 made the defaults. Updated to r736. Qcomp is now scaled based on AQ strength automatically.

[buzzqw]

@bob0r

is possibile to have a new build (0.48) with nal/hrd/pulldown ?

thanks!

BHH

[bob0r]

@bob0r

is possibile to have a new build (0.48) with nal/hrd/pulldown ?

thanks!

BHH

- x264 revision 736
- AQ_0.48.diff
- x264_2pass_vbv.0.diff
- x264_hrd_pulldown.04.diff

x264.736.dark.aq.0.48-2pass_vbv.0-hrd_pulldown.04.exe (http://files.x264.nl/AQ/x264.736.dark.aq.0.48-2pass_vbv.0-hrd_pulldown.04.exe) pthreads/gpac/fprofiled = yes

[buzzqw]

thanks bob0r!

BHH

[Atak_Snajpera]

AQ strength 0.5, sensitivity 13 made the defaults

Does it mean that AQ is always on?

[bob0r]

Does it mean that AQ is always on?

Yes, with the AQ 0.48 patch.
To disable it: --aq-strength 0

[buzzqw]

Yes, with the AQ 0.48 patch.

:eek: wow

is this to be definitive (aka in x264 svn) ?

BHH

[vpupkind]

- x264 revision 736
- AQ_0.48.diff
- x264_2pass_vbv.0.diff
- x264_hrd_pulldown.04.diff

Can you please share the sources?

[ToS_Maverick]

why is 0.5 now the default? did you change something with the strength-scale?

[Dark Shikari]

why is 0.5 now the default? did you change something with the strength-scale?No, its just because I'd rather be conservative and choose something that will always work and never be over the top than to choose something that is too strong in many cases.

Especially with 0.47, 1.0 seems too strong for most ordinary sources.

[Inventive Software]

0.5 compared to 1.0's a bit like using subme 5 instead of subme 7. (Something that appeals to all and can be changed if it isn't suitable.)

That's an analogy BTW, not a direct comparison. ;)

[DeathTheSheep]

- The file sizes between the 3 tests are very different from each other. However, this older sample happens to be the smallest in file size and still looks subjectively better (in my opinion at least).

The overal impression of the older clip seems more well-balanced. Low strength in v0.47 looks better where the details are prominent, but the background areas dont look very good. Higher strength (1.0), looks similar to the v0.45 version, but the overall impression is worse, and this is considering its 20 MB larger than the v0.45 sample. Pay special attention to Yumi's face 11 seconds into the 2nd sample. It looks best with v0.47 strength 0.5, but the dark scenes look like crap.

Yep. As I described it, the problem is threefold:
1. Bad bit/QP distribution in regard to scenes that need it.
2. Huge filesize at same crf/cqp/threshold--at significantly less SSIM.
3. Far more erratic visual quality, favoring parts of frames that don't visibly benefit from AQ.

In a number of cases, entire clips look visibly better (or even more commonly, not visibly worse) with this new AQ turned off! Why waste more bits (read: larger files) on something that doesn't help much? Actually, believe it or not, your clip is actually better off under 0.47 than some of the 0.47 clips I've made.

Here's something interesting: If you play around with 0.46 (the "official" test build is on mirror05.x264.nl/dark), you'll get even better results than those you're probably getting with 0.45. :)

[Dark Shikari]

In a huge number of cases, entire clips look visibly better (or even more commonly, not visibly worse) with this new AQ turned off! Why waste more bits (read: larger files) on something that doesn't help much?I have never ever seen this ever except in the case of low-bitrate animation.

Please give proof and examples.

Also, I need a 0.47 version of these animated clips to compare to, so I can see what the problem is. Old versions alone just won't do.

[Yoshiyuki Blade]

I just reviewed an encode (same anime episode as the other samples) with AQ v0.48, and it actually looks very good at default settings. I compared the following:

AQ v0.45 --qcomp 1.0 --aq-strength 1.0 --aq-sensitivity 20 average CRF 30.24: 279 MB
AQ v0.48 --qcomp 1.0 --aq-strength 0.5 --aq-sensitivity 13 average CRF 26.64: 279 MB

Everything else constant.

The results are really comparable to each other, and the current patch may provide better quality. I'm not quite sure about that yet, but if the lower average CRF is any indication, it should.

Did anything major happen between 0.47 and 0.48 besides changing the defaults? It makes me wonder if Sharktooth's AVC CQM contributed to the quality difference (it's the only major difference in my earlier comparisons). The two tests above both have the CQM.

I'm currently re-testing at lower bitrates to see how well the current patch scales at lower bitrates.

[Sagekilla]

Between 0.47 and 0.48, according to the main post anyway, the biggest change seems to be qcomp being based off aq-strength.

[Dark Shikari]

Between 0.47 and 0.48, according to the main post anyway, the biggest change seems to be qcomp being based off aq-strength.But qcomp=1 will be qcomp=1 regardless of AQ strength, so that's not a change in the case of his settings.

[Morte66]

@DS
So what qcomp are you recommending now, for crf and two pass?

[Dark Shikari]

@DS
So what qcomp are you recommending now, for crf and two pass?Leave it at default, AQ will automatically raise it.

For AQ > 0.28, qcomp will be 1 automatically.
For clarity, I was only referring to low bitrate-animation.

And in pertinence to using 0.47, does 0.48 use the same algorithm? If so, an upgrade could well be in order anyway. :DYes, 0.48 is the same as 0.47 except with changed defaults and the automatic qcomp scaling.

[DeathTheSheep]

Holy crap, that was a fast response! You know, I'm actually going to repeat the bulk of my above post here, since I didn't have time to edit it:

For clarity, I was only referring to low bitrate-animation in my previous assertions.

And in pertinence to using 0.47, does 0.48 use the same algorithm? If so, an upgrade could well be in order anyway. (Answered, yes).

I'm not quite sure about that yet, but if the lower average CRF is any indication, it should.
No, a program that reports something like "lower average CRF" is useless in this case, because "lower average" says almost nothing by itself when considering both the statistical and visual entirety of the results. It's the distribution of lower (and higher) QPs across frames/scenes that matters. For all we know, the QPs could be lowering in areas where they shouldn't be, and the higher ones are relegated to areas where they harm quality. I was just saying that actually seems to be taking place for me in the context of said anime sources.

For live footage, results seem to be roughly the same across patch versions when tweaked properly.

[vpupkind]

After trying 0.48 I noticed that the bitrate constraints are no longer obeyed (which might be explained by qcomp=1)

[Dark Shikari]

After trying 0.48 I noticed that the bitrate constraints are no longer obeyed (which might be explained by qp=1)Bitrate constraints?

[burfadel]

On the help it says --aq-strength is defaulted to 0.5, but you still have to actually enter a value for the strength for AQ to be turned on...

To run default settings, wouldn't it be easier to either:
- Have it on by default :D (I think many would agree), and turn it off by '--aq off' or
- Be able to turn it on by just '--aq'? (on by default when set, but still allow '--aq on', and when set use the default settings.

The other options can remain, it just seems silly having it as a default 0.5 when you have to state it anyway!

[Dark Shikari]

On the help it says --aq-strength is defaulted to 0.5, but you still have to actually enter a value for the strength for AQ to be turned on...

To run default settings, wouldn't it be easier to either:
- Have it on by default :D (I think many would agree), and turn it off by '--aq off' or
- Be able to turn it on by just '--aq'? (on by default when set, but still allow '--aq on', and when set use the default settings.

The other options can remain, it just seems silly having it as a default 0.5 when you have to state it anyway!Uh, it works just fine here without specifying it...

[vpupkind]

Bitrate constraints?
ABR, with VBV of 1-2 sec.

[Dark Shikari]

ABR, with VBV of 1-2 sec.What do you mean--it violates VBV maxrate? 2pass ABR doesn't currently obey VBV maxrate properly anyways.

[vpupkind]

What do you mean--it violates VBV maxrate? 2pass ABR doesn't currently obey VBV maxrate properly anyways.
When I am measuring the maximum VBV fullness on a 2-sec buffer, I see that with low qcomp the buffer stays reasonably close to the limit.
When 0.48 is used, the buffer can reach >200% of the VBV size

[Dark Shikari]

When I am measuring the maximum VBV fullness on a 2-sec buffer, I see that with low qcomp the buffer stays reasonably close to the limit.
When 0.48 is used, the buffer can reach >200% of the VBV sizeHmm... somehow this doesn't surprise me.

We need VBV lookahead :p

[TheRyuu]

I built this last minute for anyone who wanted to try it (I haven't tested it, the most tested I did was it patched/builded without error).

http://www.fileducky.com/rIhWCgXL/

Has the older AQ 0.46 that is supposedly better (for animation) in a way (I haven't tested this yet) and also contains DeathTheSheep's updated me-prepass patch as well since that option was a personal favorite of mine.

Enjoy.

P.S. On a side note, is using -funroll-loops (unrolling the loops) particularly useful in building x264?

[desta]

On the help it says --aq-strength is defaulted to 0.5, but you still have to actually enter a value for the strength for AQ to be turned on...

To run default settings, wouldn't it be easier to either:
- Have it on by default :D (I think many would agree), and turn it off by '--aq off' or
- Be able to turn it on by just '--aq'? (on by default when set, but still allow '--aq on', and when set use the default settings.

The other options can remain, it just seems silly having it as a default 0.5 when you have to state it anyway!
What it actually says is..

How to use AQ:

1. AQ is on by default at strength 0.5. Change --aq-strength to make it stronger or weaker.

[burfadel]

Ah ok! Haven't done any encoding since then, I didn't realise it was on by default. Earlier it was stated AQ would not be on by default, and that it will only be available as an option unless otherwise stated.

Its a great feature, it definitely makes sense to have it on by default.

[Yoshiyuki Blade]

Yeah, at lower bitrates, v0.48 does not seem to deal with dark/flat scenes quite as well as 0.45 (and probably 0.46, though I haven't tested it myself) on this particular anime test. Although using the recommended default settings combined with Sharktooth's CQM looks much better than any other tests I've done with the recent patches, the blockyness in some areas are still quite distracting.

AQ v0.48 --qcomp 1.0 --aq-strength 0.5 --aq-sensitivity 13: 179 MB (Sample 2 (http://www.mediafire.com/?49jwuzgnf1e))

Compare it to Sample 2 posted here (http://forum.doom9.org/showthread.php?p=1094123#post1094123).

All settings and bitrate pretty much identical except for the obvious ones (qcomp and aq).

[burfadel]

I agree, at lower bitrates it does block. I mentioned that earlier, although it is different now, a lower strength at 0.3 and higher sensitivity 16.5 as mentioned earlier fixes that blocking and still gives good results without changing the final file size.

I have a suggestion that at high resolution (the cutoff point which will need to be determined) to use the current default settings, and at lower resolution have different default settings as I outlined above. Maybe have it on a linear formulated scale for strength and sensitivity or something?

[Dark Shikari]

Yeah, at lower bitrates, v0.48 does not seem to deal with dark/flat scenes quite as well as 0.45 (and probably 0.46, though I haven't tested it myself) on this particular anime test. Although using the recommended default settings combined with Sharktooth's CQM looks much better than any other tests I've done with the recent patches, the blockyness in some areas are still quite distracting.

AQ v0.48 --qcomp 1.0 --aq-strength 0.5 --aq-sensitivity 13: 179 MB (Sample 2 (http://www.mediafire.com/?49jwuzgnf1e))

Compare it to Sample 2 posted here (http://forum.doom9.org/showthread.php?p=1094123#post1094123).

All settings and bitrate pretty much identical except for the obvious ones (qcomp and aq).I don't see any real difference between your two samples. There's a bit of different bit distribution (some frames look better in one, some in the other, but mainly due to different frame size), but the main thing I see is the 0.45 has terribly uneven quantizer distribution, wasting bits, while the 0.48 has much smoother distribution.

I cannot see any real problem visually in either encode.

[DeathTheSheep]

Both of these clips look pretty much fine to me, too. That's my $0.02... Just out of curiosity, what's the SSIM for each of them?

[Yoshiyuki Blade]

I don't see any real difference between your two samples. There's a bit of different bit distribution (some frames look better in one, some in the other, but mainly due to different frame size), but the main thing I see is the 0.45 has terribly uneven quantizer distribution, wasting bits, while the 0.48 has much smoother distribution.

I cannot see any real problem visually in either encode.

The most noticeable difference between the clips is when the dark area slowly reveals the character Usui at about 13 seconds in. The fade-in effect is less blocky on the v0.45 clip. But yeah I have to agree that for the most part its comparable. Perhaps the rest is up to tweaking the strength for optimal results.

Both of these clips look pretty much fine to me, too. That's my $0.02... Just out of curiosity, what's the SSIM for each of them?

SSIM...? lol, it's gonna be a long time before I stop considering myself a n00b :D. All of my observations so far were done purely subjectively by eye.

[CruNcher]

@Dark Shikari
i lost a little track of what's going on but i tried the latest AQ and got this result i had to use --aq-strength 1.0 else the background (left of here) was to blocky everything looks fine except the silhouette of her face it shows extreme ringing like the AQ amplified it to much.

http://mirror05.x264.nl/CruNcher/force.php?file=./strange-ringing.mkv

hehe ok my old matrix workaround fixed it again,at least lowered the problem so that's visual not so noticeable anymore :)

http://mirror05.x264.nl/CruNcher/force.php?file=./ringing-killed.mkv

[Morte66]

0.48 with defaults is looking good, and making good use of bitrate, on my crf 18/16 encodes from cleaned up SDDVD/HDDVD. My thanks to all concerned.

[LoRd_MuldeR]

Just for info:
An "experimental" build of Avidemux that includes the x264 VAQ Patch v0.48 is available now :)
There are no GUI controls to adjust the AQ settings yet, but it will be on by default.

http://mulder.dummwiedeutsch.de/home/?page=projects#avidemux
http://razorbyte.com.au/dev_dump/

[bob0r]

x264.736.modified.01.exe (http://files.x264.nl/x264.736.modified.01.exe)

General thread:
http://forum.doom9.org/showthread.php?t=130364

x264_aq_var.48.diff
http://forum.doom9.org/showthread.php?t=132760
x264.gaussian.cplxblur.01.diff
Dark Shikari: - gaussian cplxblur: gives a tiny improvement in 2pass ratecontrol
x264_me-prepass_DeathTheSheep.01.diff
http://forum.doom9.org/showthread.php?p=1093523
x264_2pass_vbv.4.MatMaul.diff
http://mailman.videolan.org/pipermail/x264-devel/2008-January/004015.html
x264_hrd_pulldown.04.diff
- HRD and pulldown for HD compatibility

[DeathTheSheep]

Here's a little test I performed, which supports my earlier postulate.

SETUP
Default baseline level 1.3 settings plus: -q30 --no-fast-pskip -m6 --me umh --keyint 1500.
This commandline was chosen for its representative simplicity. Testing indicates that the results below hold for more advanced commandlines as well.

Source: 2039 frame Bleach anime high-motion intro, 320x240 (QVGA) resolution, ~3mbps XviD encoded.

Binaries: 0.47 (http://files.x264.nl/AQ/force.php?file=./x264.721.dark.aq.0.47.exe) and 0.46 (http://mirror05.x264.nl/Dark/force.php?file=./x264_AQ_0.46.exe).

AQ strength settings:
0.47: Strength 1.0 (0.5 is default/recommended, and I used this first, but was advised against doing so, so repeated test with 1.0 for uniformity. Similar results.
0.46: Strength 1.0 (remember, the old algorithm had a different strength system, and 1.0 was recommended).

Threshold was adjusted so that final size was roughly the same. Because of resolution, higher-than-anticipated threshold applies (around 24.5 for 0.46 and 20.3 for 0.47).

RESULTS:
0.46 was clearly the winner (at least in regard to this source/bitrate).
SSIM
0.46: 0.9747229
0.47: 0.971398
Note: 0.46 is the metric winner, its SSIM score higher, but even so, the closeness of the score is deceptive.

PICTURES:
Visual evidence, chosen out of a group of 15 randomly selected screenshots in the 2039-frame sample. The source is not shown; however, at this bitrate, the quality difference should be apparent, regardless. Note: Zooming in may be required to asses images if viewed on a high-resolution display.

LEFT (or top) IMAGES FROM 0.47, RIGHT (or bottom) IMAGES FROM 0.46.

http://i249.photobucket.com/albums/gg201/saskura_80211b/x264/b116.pnghttp://i249.photobucket.com/albums/gg201/saskura_80211b/x264/46-116.png
Outline around orange-hair character is cleaner, sharper. More ground detail present, and character's arm is more clearly visible. Less artifacting visible around clouds, sword hilt, under top subtitles.

http://i249.photobucket.com/albums/gg201/saskura_80211b/x264/b215.pnghttp://i249.photobucket.com/albums/gg201/saskura_80211b/x264/46-215.png
Reflection in sword in clearer, sharper, less blurred edges, proper color proportion, less artifacting all around (especially around reflection edges).

http://i249.photobucket.com/albums/gg201/saskura_80211b/x264/b246.pnghttp://i249.photobucket.com/albums/gg201/saskura_80211b/x264/46-246.png
Entire outer circumference of female character's hair much less severely blurred (especially on left of head in darkest area).

http://i249.photobucket.com/albums/gg201/saskura_80211b/x264/b666.pnghttp://i249.photobucket.com/albums/gg201/saskura_80211b/x264/46-666.png
Much fewer border artifacts, clearer edges, more edge detail kept. More accurate preservation of character's feet.

http://i249.photobucket.com/albums/gg201/saskura_80211b/x264/b770.pnghttp://i249.photobucket.com/albums/gg201/saskura_80211b/x264/46-770.png
Slightly less blocking and artifacting on color gradients.

http://i249.photobucket.com/albums/gg201/saskura_80211b/x264/b1343.pnghttp://i249.photobucket.com/albums/gg201/saskura_80211b/x264/46-1343.png
Much clearer edge and detail retention in dark areas. Specifically note the 2 parallel lines (center) on character's suit.

http://i249.photobucket.com/albums/gg201/saskura_80211b/x264/b1693.pnghttp://i249.photobucket.com/albums/gg201/saskura_80211b/x264/46-1693.png
Better fine detail retention, edge preservation, lack of artifacts and color smearing. Specifically note the cat's whiskers and right ear.

CONCLUSION AND GENERAL REMARKS:
If not already readily obvious, 0.46, in technical terms, is t3h w1n in this case, sample, resolution, bitrate, source (to be slightly specific). However, due to the representative nature of this test, I would make so bold as to extend these results to many other sources of this nature; that is, due to this test and previous experience, I believe these results apply to other low-bitrate anime samples, if not a more general range of samples as well. More testing may be required to verify this assumption.

The following statements can be deemed invalid in this case:
- SSIM increases with the new AQ.
- The new AQ allocates bits in areas that need them most (see results of previous test for more concrete evidence of this).
- Visual quality improves with the new AQ.
Again, the evidence shared above tends to refute these statements, for the most part.

RAW .264 LINKS:
0.46 Clip (http://gabext.com/samples/ssim-0.9747229_AQ46.264)
0.47 Clip (http://gabext.com/samples/9741398_AQ47_1.0test.264)

[edit1] Added descriptions under images to make known what to look for.
[edit2] Reuploaded images as VGA PNGs.
[edit3] Replaced strength 0.5 AQ 0.47 with strength 1.0, as used in 0.46. All results and pictures updated (descriptions didn't change much, in honesty). There is a slight improvement in the areas of mention, but marked degradation in high-contrast areas, which I didn't touch on.

[Dark Shikari]

AQ strength settings:
0.47: Strength 0.5 (default/recommended, also tried 0.3 as suggested for anime with worse effect, as well as 0.6 with little difference).
0.46: Strength 1.0 and 1.1Your entire test is invalid (and one could probably say, rigged). The reason the "old AQ looks worse" is because you used a lower strength on the new one. Looking at the quantizer distribution, it appears part of the problem is my algorithm to reduce quantizer bit cost, which was present in previous versions of AQ. Its just that at a low enough strength, the quantizer change is near nil--looking at the quantizer distribution in your 0.47 encode, that appears to be the case. It has nothing to do with 0.47's "different algorithm"--its simply because your strength was so weak it did nearly nothing at all on your anime source.
(remember, the old algorithm had a different strength system, and 1.0 was recommended).No, it didn't have a different strength system. 0.5 is recommended just because its better to be conservative and avoid artifacting than be aggressive and result in too much artifacting.

In particular, the primary reason for 0.46 looking "better" is because it spent less bits coding the subtitles.

Looking at both your encodes though, it appears as if something is seriously wrong with the quantizers. There are some scenes with some serious contrast between flat and complex areas--yet the quantizer is nearly flat across the frame--and thats in your strength 1.0 encode! There is something obviously wrong here.

[DeathTheSheep]

Is that so? Then at least it makes a good comparison between the effectiveness of different strengths (or rather, old defaults vs new defaults). :)

I'll just go n' substitute strength 1.0 for 0.47 real quick.

PS: The visual quality of the subtitles differs very little between the two encodes, though the bits drawn from it seem to have been quite useful elsewhere.
PPS: It also means lower strength is NOT necessarily good for anime, as has been claimed throughout the thread. ;)

[Dark Shikari]

Is that so? Then at least it makes a good comparison between the effectiveness of different strengths (or rather, old defaults vs new defaults). :)

I'll just go n' substitute strength 1.0 for 0.47 real quick.

PS: The visual quality of the subtitles differs very little between the two encodes, though the bits drawn from it seem to have been quite useful elsewhere.Test this. After you're done with your 0.47 1.0 strength encode, do another with the following line commented out:

if(abs(new_qp - h->mb.i_last_qp) == 1) new_qp = h->mb.i_last_qp;

in ratecontrol.c. Post the .h264 files for both.

[DeathTheSheep]

Quite interesting. I'll certainly give that a spin...

I assume you mean commented out of .47/.48, right? :)

[Dark Shikari]

Quite interesting. I'll certainly give that a spin...

I assume you mean commented out of .47/.48, right? :)Yes. Well, try the same for 0.46 if you're curious. It'll make debugging much easier.

And seriously, get on MSN or something, it'll make this all much easier.

[DeathTheSheep]

There we go. Again, results not much different-- the areas of mention had some improvement, but the high-contrast regions suffer a huge quality drop, which isn't clear in those specific pictures (well, actually, now that I check, it's there somewhat. Whew). 0.46 is still the obvious winner, though.

MSN...well uh, when I'm in my gmail inbox, there's a chat window to the left. I wonder... :)

[edit]Don't you love when people edit posts? 80% of the time I never see all the little things they change. Sheesh. :D

And it's like 1AM here. I'm...gunna take this up tomorrow I guess. Lots of physics homework, ugh. Electric fields, capacitors, ugly surface integrals, sheesh. Unlike you, physics is not my cup-o-tea... :(

[Dark Shikari]

There we go. Again, results not much different-- the areas of mention had some improvement, but the high-contrast regions suffer a huge quality drop, which isn't clear in those specific pictures (well, actually, now that I check, it's there somewhat. Whew). 0.46 is still the obvious winner, though.

MSN...well uh, when I'm in my gmail inbox, there's a chat window to the left. I wonder... :)

[edit]Don't you love when people edit posts? 80% of the time I never see all the little things they change. Sheesh. :D

And it's like 1AM here. I'm...gunna take this up tomorrow I guess. Lots of physics homework, ugh. Electric fields, capacitors, ugly surface integrals, sheesh. Unlike you, physics is not my cup-o-tea... :(Words do me nothing, quantizer distributions (aka .h264 streams) are much more useful ;)

[wata]

is VAQ suitable for crf encoding?
i test it on a few clips crf=18
with 0.48
the filesize of all clips always increase (>20%) with default AQ setting then without AQ
with 0.45 strength 1 sensitivity 0
filesize is almost always lower but very close than without AQ

[Dark Shikari]

is VAQ suitable for crf encoding?
i test it on a few clips crf=18
with 0.48
the filesize of all clips always increase (>20%) with default AQ setting then without AQ
with 0.45 strength 1 sensitivity 0
filesize is almost always lower but very close than without AQYes, it is suitable for CRF encoding. If the filesize gets bigger than you want--raise the CRF.

[Sharktooth]

VAQ enabled x264 build is now included in MeGUI auto-update.
Hope it helps.

[wata]

test some more

for 0.48 i have to increase crf by 2 (default setting) to match no aq size
so now crf 18 = crf 20

for 0.45 i have to decrease crf by 1 (aq 1.0 sen 0)
crf 18 = crf 17

[CruNcher]

I see DeathTheSheep confirmed the edge ringing problem also with Anime :)

[DeathTheSheep]

Words do me nothing, quantizer distributions (aka .h264 streams) are much more useful ;)
I really did update the whole test for the new 1.0 strength, including all images and the new .264 stream (click the link!). Now it's no longer "rigged" by any means :(, but the end results are *very* similar except for slight improvements to the regions of mention and slight degradations near edges. That's exactly what I mean when I say nobody sees the extent to which someone edits their posts! I'll betcha nobody else noticed the update/refresh either. But I am going to use this quote of yours in a drama novel one day. "Alas dear Yorrik, but that thy words do me nothing. 'Tis your quantizer distributions which art far more useful." :D

CruNcher: Yep, maybe so, though unfortunately I wasn't using Shin Taketori Monogatari so it must have been painful to see. ;)

[DeathTheSheep]

With r736, I did the following tests at strength 1.0:
AQ46 original (o) and commented (c).
AQ47 original (o) and commented (c).

Clips found here (http://gabext.com/samples/Shikari).
Random comment: Looks like r736 is worse than r721 (lower SSIM, higher filesize) with same AQ46 settings. Why is that?

[akupenguin]

If the discrepancy appeared in r731, then look at http://akuvian.org/images/tesa2.png. Note that the umh change reduces quality at any given value of merange, but improves the quality-per-speed curve.
If the discrepancy appeared in any other rev, then I need more info.

[DeathTheSheep]

I'll check on that.

By the way, I did notice that the new tesa delivers less quality per bitrate, and the point of diminishing returns generally occurs at smaller meranges. Given its nature as an "insane" option in the first place (that is, only those who want max quality regardless of speed cost would use it regularly), is there a threshold I can adjust (line no.) to restore the higher quality but slower behavior of the old tesa? The difference is actually quite noticeable at the sources/bitrates above, even more so at higher meranges.

Also, for some reason, I find that revision 736 (3.7fps) is only marginally faster than 681 (3.3fps) with prepass and satd, so I'd certainly go with the quality boost of the latter's satd rather than experience the marginal speed boost.

[akupenguin]

is there a threshold I can adjust (line no.) to restore the higher quality but slower behavior of the old tesa?
line 493 (sad_thresh) and line 501 (17/16) and line 537 (limit).

old tesa is equivalent to sad_thresh=10, ads threshold=5/4, limit=infinity

[DeathTheSheep]

int sad_thresh = i_me_range <= 16 ? 10 : i_me_range <= 24 ? 11 : 12;

What! Dynamic merange-adaptive threshold? Genius! :p

What exactly does limit do? I'd assume scaling it by a factor rather than adding a constant, but what would a higher denominator (i_me_range / 3) do as opposed to a higher numerator (i.e. what happens if the limit goes up rather than down)?

[edit]Never mind, I'll figure it out soon enough.

[edit2] Yes, it appears to be r731. And in tests with other sources at higher bitrates, this umh quality decrease seems to be negligible anyway. It's faster, though.

[talen9]

VAQ enabled x264 build is now included in MeGUI auto-update.
Hope it helps.

It's compiled without MP4 support :rolleyes:

Oh well, using raw h.264 is the same for me, that's only that "MP4" is the default output format in MeGUI and this should raise many complaints, i think ;)

[Dark Shikari]

It's compiled without MP4 support :rolleyes:Oops! :p

[IgorC]

Also, for some reason, I find that revision 736 (3.7fps) is only marginally faster than 681 (3.3fps)
Marginally? +12% speed?

[ToS_Maverick]

Dark Shikari, just wanted to inform you, i did encode some live-concert footage, with smog and gradients in the fog and so on...
your default settings (0.5, 13) turned out to be the best choice visually!

[Sagekilla]

Marginally? +12% speed?

Indeed.. 3.7 fps vs 3.3 fps is a huge difference, especially with a movie that's 200,000 frames long. In that case, it can make a difference of up to 110 minutes!

[bob0r]

It's compiled without MP4 support :rolleyes:

...

x264.736.megui.exe --help
x264 core:58 svn-736M
Syntax: x264 [options] -o outfile infile [widthxheight]

Infile can be raw YUV 4:2:0 (in which case resolution is required),
or YUV4MPEG 4:2:0 (*.y4m),
or AVI or Avisynth if compiled with AVIS support (yes).
Outfile type is selected by filename:
.264 -> Raw bytestream
.mkv -> Matroska
.mp4 -> MP4 if compiled with GPAC support (yes)


x264.736.megui.exe --threads 2 -B5000 -m6 -r5 --direct=temporal --me=hex -b2 -w --qcomp=0.10 -A"p8x8,i8x8,i4x4" -8 --fps=25 --output test.mp4 720p50_mobcal_ter.yuv 1280x720

encoded 504 frames, 6.09 fps, 5168.29 kb/s

test.mp4 plays fine with MPC (internal splitter)

or Haali Splitter:
http://x264.nl/x264.736.megui.mp4.output.jpg

C:\Program Files\megui\tools\x264\x264.exe <-- OLD
C:\Program Files\megui\tools\x264\x264.736.megui.exe <-- NEW build

*slaps Sharktooth for making all of the above information useless!

Rename x264.736.megui.exe to x264.exe and your problem is solved.

The old x264.exe is Cef's 709 build, which also has .mp4 output, i can not reproduce.

[DeathTheSheep]

I don't think there's a person on the planet who'd want to encode 200,000 frames with high merange satd in baseline profile. Simply why bother? :p

Yes 12% is big. But relative to the speed/quality loss you'll already be destined to suffer by that point, it's already negligible. For instance, sane settings for that same clip (the 3.3 vs 3.7), I get well over 60fps. I'd say there's less than 10% quality loss at that speed, too. No, tesa isn't about the 12%--when you're already reduced to the single-digits for minimal gain, it's really a drop in the bucket.

DS, how goes the analysis of the clips I uploaded? Does the "problem" you mentioned look solvable?

[Sagekilla]

*Coughs* Well.. I didn't encode in baseline but I did encode close to 200,000 frames using a high merange + esa.

[DeathTheSheep]

With both limiting if() statements commented out and old satd thresholds restored (thresh=10, ads=5/4), speed is significantly slower than r681. Filesize also higher, ssim lower.

Interesting. I don't think this has anything to do with r731. :) I've also noticed that r736 doesn't achieve the high burst speeds that r681 did on low-motion scenes.

[MfA]

BTW, I'm curious ... why doesn't x.264 do plain RDO optimization of mb_qp_delta? Are the potential gains too small?

[Dark Shikari]

BTW, I'm curious ... why doesn't x.264 do plain RDO optimization of mb_qp_delta? Are the potential gains too small?Such optimization would only be useful if one did it as a trellis, mostly likely; and such a thing is quote possible.

If you mean "RDO" as in somehow optimizing each block to its optimal QP, rather than just RDO to save bits spent on qp_delta, thats absurdly slow and not even optimal.

[DeathTheSheep]

So what are your plans for AQ now? Do you have a list of priorities like:

1. Bugfix
2. Add lambda/enhancements
3. Bugfix
4. Get in CVS

...or something like that?
Also, do you have any remarks pertaining to the samples I uploaded?

Cheers!

[Dark Shikari]

DtS, I looked at your updated 0.47 and the quantizer distributions are completely impossible. After considerable testing with my own sources, there is no way they could possibly be generated by my algorithm [note: code != algorithm, a mistake in the code can result in deviation from the algorithm)]

Please post a link to your source so that I can attempt to replicate your results.

[DeathTheSheep]

Interesting... Okay, I'll get you (a) source. I can't just give you the plain XviD, since I used an avisynth script with functions that don't exist anymore, and a modded ffdshow with custom filter settings for the decoding.

So I'll give you a "new" source (already resized and filtered!), a very high bitrate (-q1) x264 re-encode of the original to replicate the results on.

I included:
- EXACT avs, cmd, source avi used.
- EXACT build used (also linked to in my test earlier, originated from bobor's mirror).
- EXACT output file produced by the above setup (its the only .264 file there).

If you download everything to the same directory, just double-click the cmd and you should get the exact output .264 I included.

http://gabext.com/samples/Shikari

Knock yourself out! (In a good way.) :)

[edit]Ups.

[Dark Shikari]

Interesting... Okay, I'll get you (a) source. I can't just give you the plain XviD, since I used an avisynth script with functions that don't exist anymore, and a modded ffdshow with custom filter settings for the decoding.

So I'll give you a "new" source (already resized and filtered!), a very high bitrate (-q1) x264 re-encode of the original to replicate the results on.

I included:
- EXACT avs, cmd, source avi used.
- EXACT build used (also linked to in my test earlier, originated from bobor's mirror).
- EXACT output file produced by the above setup (its the only .264 file there).

If you download everything to the same directory, just double-click the cmd and you should get the exact output .264 I included.

http://gabext.com/samples/Shikari

Knock yourself out! (In a good way.) :)

[edit]Ups.Answer: stop adjusting your sensitivity. It looks just fine at sensitivity 13. Kthx. :p

[Yoshiyuki Blade]

Answer: stop adjusting your sensitivity. It looks just fine at sensitivity 13. Kthx. :p

Yeah, I think upping the strength to 0.6 is a fairly decent number for some tests I've done. The results seem change rather drastically moving up in increments of 0.1 strength at a time (where 1.0 is horrible). At this point, the difference in quality is slight in comparison to the older AQ, so I'm probably not gonna sweat on the whole AQ v0.47+ vs v0.46 issue for anime.

[bkman]

Dark, I can't remember if you already have or not (this thread is rather large :P) but can you explain the exact meaning of the strength and sensitivity settings within your AQ method? And what is the expected result from changing each?

I'd like to be able to optimise the settings more intelligently than just trial and error.

[Dark Shikari]

Dark, I can't remember if you already have or not (this thread is rather large :P) but can you explain the exact meaning of the strength and sensitivity settings within your AQ method? And what is the expected result from changing each?

I'd like to be able to optimise the settings more intelligently than just trial and error.Sensitivity should not be changed, stop touching it. It will be removed when AQ hits SVN.

Strength just affects how wide the range of quantizers used is.

[Blue_MiSfit]

I'm very pleased with the quality of AQ. I have enabled it for my HD-DVD Planet Earth backups, and things look good at ~10mbit. I'm backing up to fit 2 episodes on a DVD9, and it's hard for me to tell a difference from the source.

Anyone with normal eyes wouldn't stand a chance.

The AQ really helps improve detailed areas, and also evens out the big flat areas which are so problematic.

I add a tiny bit of noise in ffdshow (10-15 luma) for 1080p content, and it's a very visually pleasing image to me!

~MiSfit

[burfadel]

I add a tiny bit of noise in ffdshow (10-15 luma) for 1080p content, and it's a very visually pleasing image to me!
~MiSfit

Adding a bit of noise through ffdshow does make things look better! However, the default settings aren't optimal (in my opinion). Chroma noise makes things look worse I think, and a setting for luma of 30 by default is too high as well!

The other problem with the noise function is that its different for each resolution, so what looks good at 1080 looks quite grainy at low res.

Anyways, AQ 0,48 is great now, I can raise the CRF a point or two (or more) without it looking worse than the lower CRF with AQ off - in other words its visually much more bit efficient. Keep up the good work!

[Blue_MiSfit]

I definitely alter the ffdshow settings considerably :) No chroma, much less luma, and I switch the noise mode to something other than the defaults (not sure what those are offhand though)

/OT

I still do 2 pass, but I find overall visual quality improves overall at the same bitrate, so that's all I care about!

~MiSfit

[Sagittaire]

For high quality encoding (1080p at more than 6 Mbps for example) the most difficult part are dark area. Why not make different AQ level with different mask (DarkMasking, ComplexityMasking ... ect ect) exactly like libavcodec.

[CruNcher]

Sagittaire Dark Areas are problematic indeed (alot is hidden in the luminance) but correctly calibrated it shouldn't be that visible @ all under normal viewing conditions for most people. Tough noise that gets really blocky can get even visible then to trained eyes (and that's where the new AQ does perfect correctly balanced and imho Akus first released version was the most efficient balanced compared to the latest), definitely non (wrong) calibrated it gets visible to everyone, tough for really low bitrate and a complete 1 pass scenario the old AQ seemed better balanceing everything and don't left any scenes without enough bitrate to be visual mega pleasing (for trained eyes correctly calibrated) in such low luminance parts but therfore it didn't enhance the detail preservation (frame wise) like Darks AQ does it, the most efficient of both worlds combined would be the ultimate for a 1 pass (lookahead) scenario i think :).

[DeathTheSheep]

Answer: stop adjusting your sensitivity. It looks just fine at sensitivity 13. Kthx. :p

Sensitivity 13? Strength 0.6? "Just fine?" Are you guys absolutely kidding me? If so, well... please don't. Kthx. :p

0.9726494 SSIM, at very junky quality (I'd honestly say a non-AQ encode looks better). It's a ringing and artifact melee, with a lot of dark edges utterly botched. No, increasing deblocking doesn't help, obviously. How does this compare to the sharp, crisp, clean effect of 0.46? Well, for a limited time only, you can Take a look(TM)!

x264.dark.aq.0.47.exe -o horrible.264 -q28 --aq-strength 0.6 --aq-sensitivity 13
--level 1.3 --no-cabac -m6 --thread-input --me umh --sar 1:1 --keyint 1500 script.avs --progress

Output here (http://gabext.com/samples/Shikari/strength06.264). Is this really what you meant, or am I getting my settings wrong here, or...?

[Dark Shikari]

Sensitivity 13? Strength 0.6? "Just fine?" Are you guys absolutely kidding me? If so, well... please don't. Kthx. :p

0.9726494 SSIM, at very junky quality (I'd honestly say a non-AQ encode looks better). It's a ringing and artifact melee, with a lot of dark edges utterly botched. No, increasing deblocking doesn't help, obviously. How does this compare to the sharp, crisp, clean effect of 0.46? Well, for a limited time only, you can Take a look(TM)!

x264.dark.aq.0.47.exe -o horrible.264 -q28 --aq-strength 0.6 --aq-sensitivity 13
--level 1.3 --no-cabac -m6 --thread-input --me umh --sar 1:1 --keyint 1500 script.avs --progress

Output here (http://gabext.com/samples/Shikari/strength06.264). Is this really what you meant, or am I getting my settings wrong here, or...?Looks perfectly fine to me... :rolleyes:

I don't know what kind of stuff you're on, but I want some of it.

PROTIP: If you're expecting perfect quality at QP28 in anime, you're crazy. And if you expect to get good efficiency encoding anime without B-frames, and without CABAC, you're an idiot. Your sample is using none of the settings that give x264 its efficiency--and the source has absurdly high motion--yet you're complaining about low quality at your low bitrate.

In my own tests, AQ 0.47 at default settings looked better than no AQ at all. I'm not responsible for other peoples' incompetence at encoding, or insisting that their quality should be perfect while using such an absurdly high quantizer.

Not to say that 450kbps isn't enough for good quality--with proper settings and encoding, you can fit pretty good quality in that. Baseline Profile H.264 is pretty damn weak though.

[DeathTheSheep]

DeathTheSheep munches dead grass, I'll have you know.

Of course no anime can look perfect at Q28. By definition of lossy encoding, nothing at all can except perhaps a blank screen or static edges that align perfectly with macroblock borders or some such artificial contrivance.

And of course baseline (as per Zune, QT, iPod, PSP, all decoder compatible) requires the absence of CABAC and B-frames and such--also by definition, obviously.

What's wrong with the clip, you ask? The edges, man, the edges are destroyed. All right, I'll prepare some more images... :rolleyes:

[Dark Shikari]

The edges, man, the edges are destroyed.

Q28

You keep answering your own question.

Moreso, you're doing your comparisons at a specific quantizer, rather than a specific bitrate, which is meaningless because AQ does not keep bitrate constant.

Looking over your encode again, the ringing isn't bad at all even at such a high quantizer; its only really visible when you zoom in, and you can't zoom in on an iPod. Ringing on low-resolution sources when one zooms in is expected.

[DeathTheSheep]

Wrong. Look at 0.46. It looks more than half decent. Look at 47. It doesn't. No fool questions why his high QP encodes don't look perfect; that's utterly ridiculous. When it's clear 0.46 produces better results than 0.47 on this clip, why do you keep refuting this? I thought we were interested in explaining (or bugfixing) rather than baselessly shooting people down or trying to debunk results. Sensitivity 13, strength 0.6 (or strength 1.0) is also worse than my previously chosen threshold in terms of quality, no matter how algorithmically "intentional" the QP distribution. You don't need any zoom to see this on the aforementioned devices/applications, but many forum members here are on a high-res, small-size monitor, which I've accounted for in using bilinear interpolation to achieve an easier viewing size.

What are you talking about, implying that I'm not keeping bitrate constant? Of course I am. All tests must end up within 1KB of each other for them to be valid to me. I thought you saw the files yourself, so what are you saying? Heck, letting x264 do a 2-pass encode with these settings doesn't even get the bitrates as close as I do with QP (1KB or less, in most cases, out of a ~4.5MB file).

Now the pictures. I don't even need an explanation. 0.47 sensitivity 13, compared to the same 0.46 I used in the big test. Filesizes ~1KB of one another. Here we go.

Here's 0.47:
http://gabext.com/samples/Shikari/7-2.PNG
and 0.46:
http://gabext.com/samples/Shikari/6-2.PNG
Less artifacting, smearing, tons more detail--Look at the character's arm! Also look at the edges of his head. Look at the sky--free of blotching and color trails. Crisper, sharper, objects look more solid, lines clearly straighter.

Again. Here's 0.47:
http://gabext.com/samples/Shikari/7-3.PNG
and .46:
http://gabext.com/samples/Shikari/6-3.PNG
Look at the reflection in the sword hilt. 0.46 is sharper, clearer, less artifacts. Left side of face is now intact, not chipped out. And look at that ugly smear in 0.47 to the left of the sword hilt. What is that? It's hideous, especially in motion.

Again! 0.47:
http://gabext.com/samples/Shikari/7-4.PNG
...and 0.46:
http://gabext.com/samples/Shikari/6-4.PNG
Look at left character's leg edges. Speaks for itself. Look at right character's mouth. Where is his mouth in 0.47?

Now for a real dark scene, where AQ supposedly helps most. Here's 0.47:
http://gabext.com/samples/Shikari/7-5.PNG
And 0.46:
http://gabext.com/samples/Shikari/6-5.PNG
Does this really need explanation? Very crisp, clean, sharp in 0.46 like expected. And here's the kingpin: *Less blocking in flat areas.*

And I just stopped here. It's not just on a smattering of scenes. It's really on just about all of them. In motion, it's even more apparent. Much, much more so, especially on the handheld devices it's made for. Need I reiterate the much higher SSIM of the 0.46 encode, either?

[edit]And on an encode of strength of 0.7 or higher (same sensitivity=13), it's much worse for the poor edges.

No, this isn't a matter of making something perfect at *any* bitrate. It's a question of which algorithm presents the better answer in these cases.

[CruNcher]

@DeathTheSheep
Sometimes i'm also suprised about Darks Visual Cortex ;)
and im with you and your 0.46 vs 0.47 results even as a non Anime guy i agree with them, Visual Quality decreased for low bitrate at least and it's hard to belive that people don't see that :)
could you please try to shift the AQ results of 0.47 useing a cqm best bet here is the prestige matrix (and no i don't either understand what people have against the prestige matrix).

[Dark Shikari]

Um... DtS... you're making no sense.

The purpose of AQ is to improve background areas.

It does that, perfectly, in every picture you've posted. 0.47 is better in all of them at what its supposed to do. The reason it sometimes looks worse in the background is because there's banding in your source, and lowering the quality actually helps improve that a bit.

You're complaining it makes edges less sharp.

That's what its supposed to do. If you don't like it, don't use it.

(Moreso: you're zooming into 320x240 images and reporting quality problems. This is a bad method of comparison because it creates aliasing and such that was not in the source.)

[DeathTheSheep]

@DeathTheSheep
Sometimes i'm also suprised about Darks Visual Cortex ;)
and im with you and your 0.46 vs 0.47 results even as a non Anime guy i agree with them, Visual Quality decreased for low bitrate at least and it's hard to belive that people don't see that :)

...DS, you...really can't see that difference? You're...kidding...right? Please tell me you're kidding? Okay, somebody tell me how to do a decent anigif...

could you please try to shift the AQ results useing a cqm best bet here is the prestige matrix (and no i don't either understand what people have against the prestige matrix).
Ah, the matrix of course helps a ton, and so does CABAC, trellis, 16 refs, and at least some B-frames! Unfortunately, my encoding is only for the devices/decoders I mentioned, which only support baseline profile. Sadly, CQMs and baseline don't tend to mix. :( That's why

[Dark Shikari]

OK, I just did my own encode and I still can't see what in the world you're talking about.

Top is encode, bottom is source, same bitrate as yours.

http://i29.tinypic.com/29w0qd5.png

Seriously, it looks fine. What are you complaining about? You're not going to get flawless detail retention at 450kbit.

[DeathTheSheep]

That's a keyframe. ^_^'

[Dark Shikari]

That's a keyframe. ^_^'Yes, a 1500 byte keyframe.

[CruNcher]

I see DeathTheSheep maybe posting pics of the source aside can make people realize the problems we are talking about ;)

[DeathTheSheep]

Keyframes always look almost the same regardless of what AQ you use.

So you found the one frame in the entire clip that comes close to 0.46...and it's a keyframe. I think I speak for more than myself when I say there's something suspicious about that.

...Almost as if your results are rigged. ;)

You want the source? I uploaded it, too. But in the source, the artifacts I mentioned obviously weren't present, and the characters did have mouths, believe it or not. The lines weren't hideously jagged. Color wasn't smeared everywhere.

It's a keyframe. It's not a predicted frame, where artifacts, motion trails, smearing, gradient motion blocking, etc really have a chance to occur. The problems I mention (and indeed, pretty much any possible problems at CQP) arise after the I-frames. :rolleyes:

All of them look fine, so does a non-AQ, so do old AQ (aku's, haali, et al). What are you trying to prove here by showing a frame that looks, for all intents and purposes, the same in almost any situation?

[Dark Shikari]

Keyframes always look almost the same regardless of what AQ you use.

So you found the one frame in the entire clip that comes close to 0.46...and it's a keyframe. I think I speak for more than myself when I say there's something suspicious about that.

...Almost like your results are rigged. ;)Its one of the smallest frames in the entire section; in fact, I'd say its far too small and the ratecontrol should be giving it more.

Anyways, I'm running a test of my own with 0.46 vs 0.48 at the same settings.

[Dark Shikari]

Oh wow. Now this really surprised me.

The reason AQ 0.46 looks better in background areas, even though AQ 0.48 lowers the quantizers there more, is because the deblocking filter is stronger at higher quantizers.

Literally, areas with lower quantizers look worse in 0.48 than 0.46 where the quantizers were higher--and that's the only explanation I can possibly think of. AQ 0.48 is actually backfiring despite doing an overall better job.

Indeed this seems to apply to a lot of things; the gradient in the anime clips is extremely smooth, and as such the deblocking filter tends to do a very good job at approximating it.

[DeathTheSheep]

Settings are not bizzare at all. They're actually what AVC is most used for these days--mobile encoding.

Mobile uses baseline.

Baseline cannot use B-frames or CABAC.

To "ignore" this obviousness is to choose "ignor"ance, quite literally. Anybody who has to pick between the 2 clips I linked to for download (with full commandline, nothing dubious here) will obviously choose 0.46, as even CruNcher admits.

If you don't like the keyframes in 0.46, use --ipratio 1.5. Then they'll get a boost.

This isn't dubious. This is, on the contrary, obvious. :rolleyes:

[DeathTheSheep]

Woah, just got your post. I gotta say...wow!

See, there IS an explanation!! But try this for even more excitement: use --ipratio 1.5 while maintaining bitrate. You might be quite surprised. SSIM goes even higher with 0.46.

[Dark Shikari]

To "ignore" this obviousness is to choose "ignor"ance, quite literally. Anybody who has to pick between the 2 clips I linked to for download (with full commandline, nothing dubious here) will obviously choose 0.46, as even CruNcher admits. There's a reason I suggested not using AQ with anime.

After inspection, I have come to the conclusion that the reason 0.46 looks better than 0.48 is because it is not as powerful. If you removed AQ altogether, it would likely look even better.

Anime is all about minimizing ringing. AQ increases ringing.

[DeathTheSheep]

There's a reason I suggested not using AQ with anime.

The reason 0.46 looks better than 0.48 is because it is not as powerful. If you removed AQ altogether, it would likely look even better.

Absolutely positively not. (See, I'm 100% sure now).

SSIM goes so far down it hurts, and things get so blocky and smeary I actually warn against eye bleeding. For our sake and yours, don't even suggest that--people might get the wrong idea, and seriously end up in the ER.

[Dark Shikari]

Absolutely positively not. (See, I'm 100% sure now).

SSIM goes so far down it hurts, and things get so blocky and smeary I warn against eye bleeding. For our sake and yours, don't even suggest that--people might get the wrong idea, seriously.Well I'm encoding a clip without AQ, so we'll see ;)

Edit: WOW. Without AQ its not even a contest, it absolutely shreds the AQ encode at this low resolution. Zooming in only makes it more obvious.

In that dark scene you pointed out earlier, AQ does better, but only because it adds more bits to it. The exact same effect could be gotten by reducing cplxblur and qcomp without AQ.

The non-AQ encode at the same CRF turned out higher quality and lower bitrate. At this point I think its safe to conclude that at these low resolutions, AQ is useless for anime.

[DeathTheSheep]

Spare me!!! Oh, the horror... Want some more screenshots? (you might wanna be in peak health before looking).


Here we go.

NO AQ:
http://gabext.com/samples/Shikari/noAQ.PNG
0.46:
http://gabext.com/samples/Shikari/6-5.PNG

I'd say noAQ is almost as bad as 0.47 (if not worse).

[DeathTheSheep]

Again with the riggin? I don't use Qcomp. This is CQP... *sigh*

I already mentioned how Qcomp destroys clips at this bitrate--in high motion scenes, there's nothing but slop.

No, for anime, AQ is a much better RC algorithm than standard Qcomp crf, as already verified here by different users.

And no, it's not for one measly scene, it's for the vast majority of the clip. Even bright edge details are brought out nicely, and if your theory about 0.46 is correct, the loopfilter keeps ringing to a minimum elsewhere. I dare you (seriously) to give me more optimal results than AQ.46. You can use any combination of qcomp and complexity reduction settings and AQ version, keeping all other options the same. Go for it, I'm open. :)

[Dark Shikari]

Spare me!!! Oh, the horror... Want some more screenshots? (you might wanna be in peak health before looking).Please read my post before responding. Almost the entire rest of the video looks better without AQ, and that one scene would look just fine without AQ if you lowered your qcomp.

You insist on using QP instead of CRF, even though CRF does a far better job in that scene yet looks drastically better than AQ everywhere else.

[CruNcher]

I just wana might add that i tested with Real Film Footage ;) with very fine details no Anime involved im my Visual tests and i got the same results especialy ringing got to problematic with 0.47 that it becomes a real Visual problem especialy for trained eyes.

[Dark Shikari]

Come on, you have to be kidding me.

http://i29.tinypic.com/2nle07a.png

http://i26.tinypic.com/einuwh.jpg

One of these is AQ 0.46, the other isn't (not saying which). The AQ 0.46 one is 20% higher bitrate than the other. Which one is it?

[CruNcher]

the top one hurts the bottom one looks ok but compareing 1 still frame is far from what i call a subjective quality test :D

[Dark Shikari]

the top one hurts the bottom one looks ok but compareing 1 still frame is far from what i call a subjective quality test :DIt looks nearly the same on all frames. AQ causes massive ringing, regardless of whether its 0.47 or 0.46 or whatever. The quality gain is near zero, and the quality loss is huge. I inspected dozens of frames throughout the video, and the effect is the same on all of them.

At this point, I trust my eyes. Any claim that AQ is good for this source (especially based on that single dark scene, while the entire rest of the video suffers horribly) is laughable and deserves to be mocked.

It is not my job to deal with people who are so in love with their own pet algorithm that they didn't even make (AQ 0.46) that they ignore that AQ as a whole causes a problem for the source. I stick to my original statement that AQ is probably not a good idea for anime in general.

Complaining about dark scenes while at the same time refusing to use qcomp, which is designed to improve quality in dark/flat scenes, is absolutely stupid. I do not argue with stupidity. I am done with this conversation.

[DeathTheSheep]

Top is AQ, bottom isn't. Give me your result clip. :)

I'd trade in good dark scenes for this kind of negligible difference any day. You can always reduce the strength, too, if you don't like it. 0.46 outperforms .47 at lower strengths too, you see. ;)

[CruNcher]

http://forum.doom9.org/showpost.php?p=1095104&postcount=663 <- Dark and whats about this no Anime here same Visual Problems
compare this with http://forum.doom9.org/showpost.php?p=1091711&postcount=514

[Dark Shikari]

Strange ringing

AQ strength 1.0

:rolleyes::rolleyes::rolleyes::rolleyes::rolleyes::rolleyes::rolleyes::rolleyes::rolleyes::rolleyes::rolleyes::rolleyes::rolleyes::rolleyes::rolleyes:

[DeathTheSheep]

But according to him, any less than 1.0 and results are sub-optimal for 0.47 as well (he posted about this before, remember?).

So yes, there is a problem there, too, no getting around it (unless you use a good matrix and such to mitigate losses).

It all comes down to how much you're willing to sacrifice in terms of high-contrast areas (and the strengths/thresholds you use to do so).

[CruNcher]

yes but to low and the noise in the black area gets to blocky no way to compensate that with 0.47 0.46 didn't need that as you can see if you watch all areas closely.

[Dark Shikari]

AQ increases ringing and decreases blocking/blurring.

If you cannot decrease blocking without causing ringing problems, you need more bitrate.

How many times do I have to explain this?

If people cannot accept simple limitations of the algorithm like this, I'm going to simply delete this thread and continue any further development in private.

[DeathTheSheep]

CruNcher: So you're saying .47 doesn't even work with your matrix fix?

DS: Of course we know this. We know what AQ does by now. We're just saying 0.46 may be the better of the two, after all.

[IgorC]

What about public blind test with something like at least 18 samples and 20 viewers? Would be reasonable. Perception isn't the same for all persons.Each one has its own idea about subjective quality, visual deficiencies etc.

[Sharktooth]

D_S: that's why i was so harsh when ppl asked if AQ was going to be commited to SVN...
ppl never learn, ppl think they know it all even if they dont understand what they say is their subjective opinion...
so, dont listen to ppl when they become silly and go ahead.

[Dark Shikari]

CruNcher: So you're saying .47 doesn't even work with your matrix fix?

DS: Of course we know this. We know what AQ does by now. We're just saying 0.46 may be the better of the two, after all.The only thing that I have seen of any comparison is that 0.46 is better because it is weaker, and it is only better in the case in which no AQ is better than AQ.

:rolleyes:

Sorry, at this point, if you want to improve AQ in a way you like it, do it yourself.

[DeathTheSheep]

Who's being silly here--isn't that your own subjective opinion? Did you even see the results of the tests I've posted, Sharktooth?

It's always debatable whether AQ or noAQ looks better in certain scenes--it's all a matter of tradeoffs--ringing for better dark scenes.

But between the two AQs themselves (0.46 and 0.48), that's a matter that does need looking into before blindly "going ahead."

IgorC's blind test is an excellent idea, but if that doesn't satisfy people, why not include both? It's a matter of using one rounding method or another, no more, no less. A matter of a one-line if statement to chose which to invoke.

I'm just worried (and for good reason) that 0.46, which to my eyes and some others' is the superior algorithm, will simply get the shaft before it's committed.

[CruNcher]

CruNcher: So you're saying .47 doesn't even work with your matrix fix?

DS: Of course we know this. We know what AQ does by now. We're just saying 0.46 may be the better of the two, after all.

It improves it it's shifting it and so the ringing @ the edges is not that visible anymore it almost looks like 0.46 then again.
And yeah DS and me are talking about what's balanced and what isn't or overdone and will hurt at lower bitrate scenarios and highering the bitrate is no option here but anyway i found a visual aceptable workaround for this so im not scared if it goes like this into SVN @ all now.
And yeah i know that im @ the edge here im allways @ it and so i might see what others don't but every visual improvement for low bitrate is important jesus i wish people would think a little bit more like the Ateme Devs (not so metric obsesed) in this specific situation we just talking about what's more efficiently balanced and if people that do low bitrate (very high quant encoding) have to suffer from this im not sure if it should go like this into SVN, if thats the case as it wouldn't be balanced in my eyes then.
Quality should scale with bitrate and if you can prevent ringing in low bitrate situations you should do it @ all costs (that's what H.264 stands for compared to ASP) as the rest (more details) come with a bitrate increase @ low bitrates Picture stability should go over Detail Preservation in any case.

[Dark Shikari]

I'm just worried (and for good reason) that 0.46, which to my eyes and some others' is the superior algorithm, will simply get the shaft before it's committed.From an unbiased perspective here (ignoring which AQ is better), there is a snowball's chance in hell that you will get 0.46 committed. Pengvado will never, ever accept patches with arbitrary rounding bugs in them, even if they have a positive effect.

[IgorC]

D_S: that's why i was so harsh when ppl asked if AQ was going to be commited to SVN...
ppl never learn, ppl think they know it all even if they dont understand what they say is their subjective opinion...
so, dont listen to ppl when they become silly and go ahead.

I think there is another problem that people don't realize what they see (or heard). Same things happened for example during LAME delevopment. When people say they can heard weird artifacts on mp3 files. But blind test (with hidden original) can't prove that.
That why all statements about audio quality should based on blind test in hydrogenaudio forum. http://www.hydrogenaudio.org/forums/index.php?showtopic=16295
If there is no blind test it's violation of the rules.
Don't listen one person, than another and another. It's easy way to lose yourself in statements of others.
Sorry if it seems like hypocrisy.

[DeathTheSheep]

No, it's a great idea, IgorC.

The only thing that I have seen of any comparison is that 0.46 is better because it is weaker, and it is only better in the case in which no AQ is better than AQ.
That's not the whole story. If it was simply a matter of adjusting strengths, one would just lower 0.47's strength to get the better result of the two. 0.47 does not distribute QPs in a visually pleasing way in the scenes that need it most, whereas 0.46 does, at the same bitrate. Using no AQ doesn't benefit the scenes of interest at all, which are the worst scenes in the encode. We're sacrificing quality in some places to retain it in others, for a constant quality effect. Keep in mind, for dark/flat scenes (which is all that matters at this bitrate--who cares about a little ringing here and there when you can't even see what's happening in darker frames?), 0.46 is better than 0.47, which is the purpose of the AQ. SSIM is higher, which you certainly tote as a key point, too. In the high-contrast scenes, there is little difference between the two algorithms anyway.

All that's left is to do the double blind. I tried to convey the expression in my test post containing screenshots of the worst-looking scenes. I thought, people who see that the horrible quality in these areas (which is why we're using AQ even at its potential ringing cost), can get a good idea of which is better. The people I've shown agreed with me here, so I made the post.

We just need IgorC's test to prove it.

[Dark Shikari]

No, it's a great idea, IgorC.


That's not the whole story. If it was simply a matter of adjusting strengths, one would just lower 0.47's strength to get the better result. 0.47 does not distribute QPs in a visually pleasing way in the scenes that need it most, whereas 0.46 does, at the same bitrate. Using no AQ doesn't benefit the scenes of interest at all. Keep in mind, for dark/flat scenes (which is all that matters at this bitrate--who cares about a little ringing here and there when you can't even see what's happening in darker frames?), 0.46 is better than 0.47, which is the purpose of the AQ. SSIM is higher, which you certainly tote as a key point, too.

All that's left is to do the double blind. I tried to convey the expression in my test post containing screenshots of the worst-looking scenes. I thought, people who see that the horrible quality in these areas (which is why we're using AQ even at its potential ringing cost), can get a good idea of which is better. The people I've shown agreed with me here, so I made the post.

We just need IgorC's test to prove it.Testing specific frames on a single source is not a way to prove that an algorithm is useful. It only proves that the algorithm is useful on those frames, on that source. The fact that your source is incredibly low resolution means the test is particularly invalid, because it does not correlate with what most sources are like.

Just because one algorithm is better than another for one source does not mean it is useful in general.

Moreso, the fact that you're not using qcomp is basically rigging the test in favor of AQ versus non-AQ, even though in any normal encode one would use qcomp.

[IgorC]

Nobody told that it would be evaluated by frames. There is MSU ap that admites blind test by playback. Real conditions.

[Dark Shikari]

Nobody told that it would be evaluated by frames. There is MSU ap that admites blind test by playback. Real conditions.Yup, I've used it.

Anyways, this thread is now pretty much pointless.

1. I am not going to work any more on the AQ algorithm for the time being.

2. 0.46 cannot possibly be committed, for the reason I stated earlier.

3. Therefore, if you want any aspect of 0.46's AQ to be committed, someone else must code it.

[CruNcher]

I have a better idea, just commit it and then lets wait what happens, if we see a big demand from low bitrate encoders that have ringing problems that they didn't had before, we know something should be changed don't we? In some way that's a big subejctive test and it doesn't need to be setup ;)
And yeah i used the MSU Subjective test to rate 0.46 vs 0.47 too :) the left/right decission setup

[Guest]

If people cannot accept simple limitations of the algorithm like this, I'm going to simply delete this thread and continue any further development in private. For the record, you're allowed to delete only your own posts. Yes, I know the forum software allows you to delete the entire thread, but mods will restore it if you do that.

I can see that some friction is developing here. Maybe CruNcher can start a new thread for his variant so that Dark Shikari can continue this thread in the direction of his vision?

[Dark Shikari]

For the record, you're allowed to delete only your own posts. Yes, I know the forum software allows you to delete the entire thread, but mods will restore it if you do that.

I can see that some friction is developing here. Maybe CruNcher can start a new thread for his variant so that Dark Shikari can continue this thread in the direction of his vision?I don't need this thread anymore; development is done, and while I don't deny there could certainly be improvements to the algorithm, I am done discussing it for now. Nothing constructive has been discussed in a dozen pages.

I'm happy to support an attempt to make a better AQ--but I'm not going to be the one coding, at least not now.

[CruNcher]

For me it's also not important to continue for now as time will tell how people gona react to the New AQ in a bigger user scale my usage behaviour isn't really important here and as i said i've got my workaround so nope no need to continue. People know what for problems could arise from the New AQ Visualy under some circumstances and how to avoid them by now and so everything has discussed and is publicly available :)
And Dark Shikari please don't feel in anyway atacked you really did a great job with the AQ it's just criticism about the way it gets implemented now and what for possible outcomes this might have in terms of Visual Perceptive Problems.

[DeathTheSheep]

Before you make any drastic decisions, I'm going to try to choose my words carefully here in an attempt to offset the risks of coming off "silly," upsetting anyone, or disseminating an inaccurate impression of my intentions.

Firstly, it may be noted that the frames I chose were representative of the worst-looking scenes in the original. They are not intended to represent all scenes in the source video, nor are they to be taken as the net visual result of the video in its entirety. They are, however, intended to represent what I have alluded to before as "scenes of interest," a denotation which I had understood to be self-explanatory, but which I will now undertake to elaborate upon in greater degree and verbosity.

Consider the following scenario. Suppose we have a test clip in which there are some mid-motion dark scenes and some low-motion/no-motion high-contrast scenes. Qcomp would assign lower QPs to the high-contrast still scenes at the expense of the darker ones. Also recall that the same specific QP X assigned to a low-contrast/dark scene will tend to look worse than the same QP X assigned to a high-contrast scene, since a high-contrast scene is allotted more bitrate, the uniform rounding mechanism would favor the higher bitrate of the high-contrast scene over the the low bitrate of the low-contrast one. In this case, the fact that qcomp is motion compensated turns into a deleterious aspect of the algorithm in that the darker scenes (or darker areas in brighter scenes) will remain characteristically lower in subjective quality than their high-contrast counterparts. I do not profess an intimate knowledge of the precise means by which x264 gives rise to said degradation of visual quality in dark/flat/low-contrast areas with uniform QP, but I will make so bold as to profess intimate familiarity with the results of such a distribution (in both QP and CRF modes).

I thereby take "scenes of interest" to denote these areas wherein the degradation of visual quality is apparent in proportion to that of the higher-bitrate, heightened-contrast scenes of prior allusion.

It might also be worth mention that at lower bitrates, quality degradation is perhaps more apparent and more distracting in the scenes of interest than the presence of artifacts in high-contrast regions, for ringing and noise artifacts will inevitably abound at this bitrate in any scene. Obvious reduction of the overall "fogginess" and "blockiness" of the scenes of interest, however, which alleviates the difficulty to discern even rudimentary shapes and other elements therein (even more so in the adverse qcomp situation I described above), is often well worth the introduction of such artifacts elsewhere in otherwise "clearer" scenes.

Please bear in mind also that qcomp is an adaptive quality rate-control mechanism, as is VAQ; however, whereas qcomp relies on a complexity metric (which may and does backfire, as in such cases as aforementioned), VAQ relies on a different, variance-based metric. This is the reason you advise against using qcomp with VAQ enabled at any significant strength. It stands to reason that one such algorithm can potentially be chosen over the other.

I believe you are asserting a logical fallacy as well. You assume that my source and settings are not representative of the bulk of consumer AVC encoding; in fact, you claim the entire setup to be abnormal. However, in so doing, you are neglecting to account for the considerable hand-held, mobile, compatibility, and low-complexity demands that dominate many aspects of the AVC encoding arena. I'll grant you that the bitrate in the above tests is low; however, it was chosen for illustrative purposes such that the visual differences might be made more readily apparent.

The topic of discussion at hand isn't even whether or not the usage of AQ is beneficial, since any large-scale optimization for one subjective quality metric/preference is bound to contradict with another to some extent, and different scenes tend to benefit differently for each. The point at hand is that VAQ 0.46 produces significantly higher visual quality on the scenes of interest, objectively and metrically, than does 0.47. It can also be argued, unlike as in the case of no-AQ vs AQ, that 0.46 produces better results on the entire clip, and perhaps even on a range of sources and bitrates, if CruNcher's remarks are to be taken into account, than 0.47.

I also believe a large-scale blind test on a wide variety of sources, bitrates, and AQ settings is in order to verify these and other assertions, as it is foolhardy to proceed with what may actually be a substandard algorithm.

[Atak_Snajpera]

@Dark Shikari
You've made alot to improve quality in x264 and I'm fully satisfied with default settings! There is no need to adjust anything! I don't really understand why some people are still looking for holy gral? They use low bitrate with crazy settings (no cabac...) and They are still crying that they cannot achive perfect results. This patch should be commited to SVN as soon as possible! No further pointless discussion. Anybody who is not satisfied with current AQ should write his own algorithm. Let's see what you can do 'smart' boys :)

[bob0r]

I love you all, let's group-hug.

[DeathTheSheep]

Lets!

[Inventive Software]

AQ is not perfect, by design, and cannot make a clip perfect on it's own without assistance from other factors. End-of.

[DeathTheSheep]

Correct, there's always some degree of sacrifice involved. It must draw the bits from somewhere--they don't spontaneously spring into existence. It's simply a new method controlling the distribution of QPs by redistribution.

[ChronoCross]

can you duplicate this on another clip?

[Adub]

Okay, big group hug continued!!!

I just want to say that I really love your hard work, DS, and I hope that you continue your efforts with a brilliant eye towards the future. And I hope that, with time, I join you in your efforts and all will be good. Just got to get through college right now.

Thanks again and code on!

[ToS_Maverick]

@Dark Shikari
is there any way we can donate, so that you (and maybe your girlfriend) could go and have a nice dinner?

[*.mp4 guy]

Dark Shikari, Thank you for all of your work on aq, I must admit that I haven't been keeping up to date with X264, but If vaq gets added to the svn I'll finally have a good reason to hunt down a new revision.

Feel free to ignore this next part, I'm sure your quite tired of the debating by now, but my curiosity is getting the better of me.

Avoiding any debate as to the merits of the various vaq incarnations, is it possible that the differences people are posting about are caused by a bias in the rate of 4x4/8x8/intra/inter blocks chosen by x264, introduced by the rounding erros in 0.46? from the screenshots I looked at, I can't see quantization alone acounting for some of the differences in the examples, it looks to me like 0.46 may be using the 4x4 transform less often then noaq/0.48/0.47. Obviously this is just wild speculation, and honestly I don't see what all the noise is about, 0.46 looks better sometimes, 0.47/0.48 looks better other times; but there does apear to be a trend in the way they look compared to each other, 0.46 looks to me like it has more 8x8 transform induced ringing, while the others apear to have more 4x4 transform induced aliasing/microblocking. I suppose it could also be a peculiarity of the AVC deblocking filter, reacting to the different realtive quantizations between blocks or something...Meh.

I bet I'll end up dieing of unfounded curiosity someday. "no really I'm not a terrorist, that screensaver just looked so cool...".

[Atak_Snajpera]

AQ is not perfect, by design, and cannot make a clip perfect on it's own without assistance from other factors. End-of.

Since when any compression is perfect. I think your expectations from h.264 standard are a little bit to high. Use higher bitrate and get used to it.

[CruNcher]

Atak ehh sure it's not perfect but to find the correct Visual (or near that) Balance for every bitrate should be the goal, it shouldn't drift in 1 direction. But hell it will improve over time im sure, and what *.mp4 guy mentions is interesting indeed :)
Most people will be fine @ the moment with the AQ some will hit walls with it but those that do will find ways to visually circumvent these walls ("that's what a good compresionist makes") :) So this is only a problem for beginners, but those are not trying such ultra crazy low bitrate stuff so it will be no real problem @ all for the mass of users as i said. But all of this doesn't change the fact that this AQ is better then no AQ @ all and perfect work by Dark Shikari and a big visual quality step for X264 in many situations.

[Inventive Software]

Since when any compression is perfect. I think your expectations from h.264 standard are a little bit to high. Use higher bitrate and get used to it.

I think you mis-understood the point of my post. It was to clear up the arguments over how (non)important AQ is to x264 at the moment. I don't have any expectations of H.264, and I personally don't use AQ because my H.264 encodes at CRF20 are very pleasing on the eye. ;)

[Atak_Snajpera]

I don't have any expectations of H.264, and I personally don't use AQ because my H.264 encodes at CRF20 are very pleasing on the eye.

Blocks are less visible on gloss LCDs but more visible on mat LCDs

Friend of mine had HDTV 40'' Samsung 1360x768 (mat lcd) and blocks were quite visible then he bought 43'' samsung 1920x1080 (gloss) and all ugly blocks on flat areas suddenly disappear despite bigger screen! Before you ask. In both cases colors were set correctly. Black was black,white didn't kill eyes and so on. FFDshow was set to HQ-RGB32. I've also notice the same results on LCD monitors (gloss vs mat)

[akupenguin]

Blocks are less visible on gloss LCDs but more visible on mat LCDs
I noticed the same thing, but I attributed it to the fact that my glossy LCD has a much brighter black than my matte LCD. Not a difference in adjustment, but rather that black pixels don't completely shut out the backlight.

[Maccara]

Please all who have complains / praises, also specify your display devices!

There's a HUGE difference on properly calibrated (for graphics work, not video) CRT (~10y old, still haven't found decently priced LCD that comes even close) & LCD (<1y old), especially in dark scenes.

I always compare those side by side, as there's zero possibility to make an encode look "perfect" on both (except when I tested Eizo ColorEdge LCD, which came close to CRT qual) so I'll have to compromise (or target one platform).

[*.mp4 guy]

In your experience in what areas are LCD's more demanding then CRT's and vice versa.

[Maccara]

In your experience in what areas are LCD's more demanding then CRT's and vice versa.

Some (many?) LCDs still have 6bit panels to me it seems smooth color gradients can be quite challenging (and YV12 already poses a bit of challenge here, so it is only pronounced on LCDs). Also due to black levels, blocking in dark areas can become quite prominent.

I've noticed that I've had to sometimes throw quite a bit much more bitrate to get rid of blocking in dark areas whereas CRT copes already with much less. (also, AQ seems to help quite a lot and other parameters, of course)

Of course, this is all subjective. Without going into personal preferences, I just wanted to remind that different display technologies have quite a bit different characteristics and it might be a good idea to keep that in mind when tuning PSY algorithms and possibly test on various hardware before making any conclusions.

[TheRyuu]

I ran some tests in anime.
New AQ vs old (Haali) AQ.

The old AQ was done using aq-strength 1.0, and the new AQ was done with the defaults (1.0 on the old aq could be the default, not sure).

What I found was that not old did the new VAQ result in a lower average quant (18.5 vs 19, not too big :p), the new VAQ also, in MOST scenes that I looked at, did a better job at keeping details and avoiding blocks.

There were places were the old AQ was ever so slightly better (really, really, flat places), but on a whole, I would definity go with the new VAQ because as a WHOLE, it performs better then the old AQ, even on anime.

I did this with the new AQ v0.48.
I'm testing v0.46 but from what I've seen so far, v0.48 is the way to go.

The tests I did are subjective but I'm just reporting what I've found.
They are also done at a bitrate of ~900, which is fine for the anime dvd backups I'm doing. (avg quant goes about 17-19).

However, I'm not sure which is better if bitrate isn't a factor.

[TheRyuu]

I ran some additional tests with VAQ v0.46 as well with anime.
From what I can tell, 0.46 does perform better then 0.48 in anime from a "visual" standpoint.

I'm not sure of the technical details of it, but not matter how you look at it, using AQ is a trade off.

Once again, looking at it from an "overall" standpoint, 0.46 is slightly superior to that of 0.48. I was looking for blocking and background details and it looks like 0.46 comes out on top.

I compared 0.48 with 0.5 strength and 13 sensitivity, with 0.46 with 0.6 strength and 20 sensitivity.

I'm not sure if 0.48 can be tuned to perform similar to that of 0.46, but I'm still testing stuff out. This is just what I've found SO FAR. I think next I'll try 0.48 with strength of 0.5 and a higher sensitivity like 15-20 which may be the cause of 0.46's better visual quality.

[Terranigma]

Can't anyone see that in reality, this thread's closed? There's no new posts by Dark_Shikari: He asked for this thread to be closed/deleted, and although he didn't get exactly what he wished, he took a different approach called the silent treatmemnt. :D
You can blame it on rio (that's right, that's a movie, but there are blamers to be blamed: just do some reading =P).

[Dark Shikari]

Can't anyone see that in reality, this thread's closed? There's no new posts by Dark_Shikari: He asked for this thread to be closed/deleted, and although he didn't get exactly what he wished, he took a different approach called the silent treatmemnt. :DThis is moreso do to the fact that I'm spending all my coding time on fixing 1pass VBV ;)

[microchip8]

This is moreso do to the fact that I'm spending all my coding time on fixing 1pass VBV ;)


oh, what about QNS? you forgot about that? :P :P

[TheRyuu]

Can't anyone see that in reality, this thread's closed? There's no new posts by Dark_Shikari: He asked for this thread to be closed/deleted, and although he didn't get exactly what he wished, he took a different approach called the silent treatmemnt. :D
You can blame it on rio (that's right, that's a movie, but there are blamers to be blamed: just do some reading =P).

Just reporting what I see.
Either case, the new VAQ is superior to the old Haali AQ even in anime.

Thanks Dark Shirkari! :)
:thanks:

[Atak_Snajpera]

I noticed the same thing, but I attributed it to the fact that my glossy LCD has a much brighter black than my matte LCD. Not a difference in adjustment, but rather that black pixels don't completely shut out the backlight.

I also see difference on blue sky. Gloss = no dancing blocks , mat = blocks. Difference is not huge but visible :(

[DeathTheSheep]

0.46 works even better if you use strength near 1.1, and possibly better if you use --ipratio 1.45 or 1.5 (to account for keyframe effects like we discussed before).

Lower sensitivities work okay too on anime. Just make sure you lower the crf to compensate. Cheers!

[Sharktooth]

VAQ 0.46 is dead. if you still didnt understood, it's useless you compare it to the newer one coz 0.46 is going nowhere.

[Morte66]

So if I buy that LCD TV I've been thinking about, it should be glossy...

[DeathTheSheep]

VAQ 0.46 is dead. if you still didnt understood, it's useless you compare it to the newer one coz 0.46 is going nowhere.

Not necessarily true... As long as it exists and is used, I'll be happy to maintain it myself. Same with me-prepass. (I also keep satd-me modified to my standards). Nothing is dead in OS if there's someone around to maintain/use it. :)

[Atak_Snajpera]

So if I buy that LCD TV I've been thinking about, it should be glossy...

It depends. Mat LCD gives more accurate colors (in my opinion) therefore you see blocks.
Glossy LCD shows less colors so blocks are less visible.

Some my old post regarding LCDs
http://forum.doom9.org/showthread.php?p=1044240#post1044240
http://forum.doom9.org/showthread.php?p=1044306#post1044306

[CruNcher]

It depends. Mat LCD gives more accurate colors (in my opinion) therefore you see blocks.
Glossy LCD shows less colors so blocks are less visible.

Some my old post regarding LCDs
http://forum.doom9.org/showthread.php?p=1044240#post1044240
http://forum.doom9.org/showthread.php?p=1044306#post1044306

But glossy they look more hmm how could you say "vivid"? (you can very well compare it with a Print on glossy photo paper compared to normal paper) tough the bigest problem with glossy panels is the reflection problem with ambient light sources, and so even a small light in direction to it can couse viewing problems, the same with sunlight coming from a window. So you should know the enviroment you gonna use such a LCD in very well or be ready to change the light sources if it seems to be problematic. You really should go in a shop and compare what fits best for you it's also a very subjective thing as Atak allready noted.

And the problems about all this lossy viewing stuff goes very deep (especialy in Windows) their are many factors that play a role in how the end result looks on your monitor (thats also a reason why for DTP and Video MACs are used so often, they have a very good defined color, brightness,contrast and gama system)

in Windows you have (The Driver, The YV12 renderer, The Decoder, The Player) and if 1 does it wrong or in bad combination with those others the end result will look worse (and haveing all from different Developers is gonna make problems for sure maybe not today but tommorow, especialy with DirectShow) on your system compared to a correctly calibrated one. Especialy as alot of stuff is hidden in Dark areas and correctly calibrated can't be seen most of the times in motion (by untrained eyes) but wrong it becomes anoying to everyone (of course you can use that knowledge to optimize the stuff visualy in those extreme uncalibrated viewing conditions and that is very powerfull in the end result for a calibrated system specialy for trained eyes) and if you work in a calibrated system and would like to know what might can go wrong in a lossy source visualy Avisynths histogram is perfect to show you that :) histogram(mode="luma").

[ChronoCross]

Not necessarily true... As long as it exists and is used, I'll be happy to maintain it myself. Same with me-prepass. (I also keep satd-me modified to my standards). Nothing is dead in OS if there's someone around to maintain/use it. :)

until of course pengvado changes something that breaks it and only adjusts 0.48. More than likely it would completely break or alter the ability of 0.46 beyond a point that simply changing values would fix (happened with haali's patch quite a few times).

[DeathTheSheep]

Ah, too true. We'll cross that road when we get to it though, won't we?

...Or so they say. :eek:

[Morte66]

My computer monitor is well calibrated and ICM profiled.

As for the LCD TV, I was prepared to darken the room for a projector so glossy LCD shouldn't be too bad. OF course I will look at the TV before buying, I'm mostly waiting for the whole 24fps --> 120fps thing to become ubiquitous (in Britain).

[Stingrey]

Is it possible, that with your build die CRF parameter can't be chosen in 0.1 step's, only in 0.5 step's?
CRF 21.0 to 21.4 all gives me the same bitrate, 21.5 gives me a much lower one!
I would need 21.1!
With CRF 21.0 1.109,16 kb/s
CRF 21.5 885,24 kb/s

--crf 21.5 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --threads auto --thread-input --progress --output "output" "input"

[Dark Shikari]

Is it possible, that with your build die CRF parameter can't be chosen in 0.1 step's, only in 0.5 step's?
CRF 21.0 to 21.4 all gives me the same bitrate, 21.5 gives me a much lower one!
I would need 21.1!
With CRF 21.0 1.109,16 kb/s
CRF 21.5 885,24 kb/s

--crf 21.5 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --threads auto --thread-input --progress --output "output" "input" Yes, this is because AQ + CRF = AQ + CQP, because qcomp gets disabled.

What I could do is map fractional changes in CRF to changes in sensitivity.

[Stingrey]

Mhm, I think that could be a good thing.
The jump in the bitrate is to high, some points between would be great.

[Dark Shikari]

Mhm, I think that could be a good thing.
The jump in the bitrate is to high, some points between would be great.In the meantime, feel free to adjust sensitivity up and down very slightly (fractionally) in order to change the bitrate between CRFs.

[Stingrey]

Ok, I will do some tests an then post it here.

Test:
CRF 21, sens 13: 1109.16 kb/s
CRF 21, sens 12: 995,98 kb/s
CRF 21, sens 11: 885,24 kb/s equal to CRF 21.5 with sens 13

so just changing the sensitivity won't do much.

[DeathTheSheep]

In the meantime, feel free to adjust sensitivity up and down very slightly (fractionally) in order to change the bitrate between CRFs.

Hey, that's precisely the kind of adjusting I did to fine-tune my filesizes, and you yelled at me for it! :devil:

:)

[Dark Shikari]

Hey, that's precisely the kind of adjusting I did to fine-tune my filesizes, and you yelled at me for it! :devil:

:)You were fine-tuning by more than 1 sensitivity ;)

You went all the way up to 19 or 20, which appeared to cause quantization problems.

[DeathTheSheep]

Ah, too true. Oddly enough the sensitivity 19 encodes looked, well, not too bad! My resolution is also smaller (people mentioned that differing resolutions have an impact on how well the threshold stays true to the original crf/cqp), and a higher sensitivity was needed to keep my original QP anyway.

If you don't mind me asking, what were the quantization problems?

[Dark Shikari]

Ah, too true. Oddly enough the sensitivity 19 encodes looked, well, not too bad! My resolution is also smaller (people mentioned that differing resolutions have an impact on how well the threshold stays true to the original crf/cqp), and a higher sensitivity was needed to keep my original QP anyway.

If you don't mind me asking, what were the quantization problems?For some reason which I do not understand, using high sensitivity on your small clip caused AQ to do basically nothing at all. As you raised sensitivity from 13, it progressed from acting normal to doing nothing, in a quite smooth fashion.

[au]

Is there a way to completely eliminate banding in dark areas (smooth gradients etc.)?
I've tried different combinations of aq-strength(0-1), aq-sensitivity(0-20) and various bitrate(6-20Mbps), but it came to nothing.

The settings I use:
x264aq048.exe --pass 1 --bitrate 9000 --level 4.1 --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -3:-3 --subme 7 --partitions all --8x8dct --me umh --threads auto --thread-input --cqmfile "matrix/prestige.cfg" --progress --deadzone-inter 6 --deadzone-intra 4 --aq-strength 0.5 --qcomp 1.0 --aq-sensitivity 15 --no-dct-decimate --output NUL "xxx"
x264aq048.exe --pass 2 --bitrate 9000 --level 4.1 --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --deblock -3:-3 --subme 7 --partitions all --8x8dct --me umh --threads auto --thread-input --cqmfile "matrix/prestige.cfg" --progress --deadzone-inter 6 --deadzone-intra 4 --aq-strength 0.5 --qcomp 1.0 --aq-sensitivity 15 --no-dct-decimate --output "yyy" "xxx"

[Dark Shikari]

Is there a way to completely eliminate banding in dark areas (smooth gradients etc.)?
I've tried different combinations of aq-strength(0-1), aq-sensitivity(0-20) and various bitrate(6-20Mbps), but it came to nothing.That basically isn't possible in the YV12 color space.

Try running gradfun2db first?

[Sagittaire]

Is there a way to completely eliminate banding in dark areas (smooth gradients etc.)?
I've tried different combinations of aq-strength(0-1), aq-sensitivity(0-20) and various bitrate(6-20Mbps), but it came to nothing.


make dithering in pre-process ...

[Jawed]

Simplest sanity check for an x264 encode is --qp 0 :)

Jawed

[Inventive Software]

Cue the terabyte hard drive...

[nm]

No need to encode a complete movie. Just pick a scene that shows banding in the previous tries and encode that with --qp 0 to see if the problem goes away. If not, it could also be due to bad rendering.

[LoRd_MuldeR]

Yes, this is because AQ + CRF = AQ + CQP, because qcomp gets disabled.
If VAQ+CRF = VAQ+CQP and if VAQ is enabled by default, will CRF still be there as a separate mode when VAQ hits the SVN?

[Dark Shikari]

If VAQ+CRF = VAQ+CQP and if VAQ is enabled by default, will CRF still be there as a separate mode when VAQ hits the SVN?Sure, because you can still disable AQ.

[LoRd_MuldeR]

Sure, because you can still disable AQ.

Yeah, but wouldn't it make more sens to make three modes then?
* CQP (classic CQP, no VAQ)
* CRF (classic CRF, no VAQ)
* VAQ (CQP/CRF with VAQ on)

At the moment we have four combination, where two are exactly the same...

[hkazemi]

In the interest of backwards compatibility with the command lines rate control mode parameters, I suggest the following implementation (if this suggestion should also be made in another place, someone please let me know). (Backwards compatibility is important for easily using new x264 builds in existing scripts that assume what each mode does) :

VAQ=variance AQ
CRF=constant rate factor
CQP=constant quantization

1.) use VAQ if rate control mode is not specified (default)
2.) use VAQ if -vaq=xx (e.g. --vaq=20) is specified (this is the new CRF mode aka VAQ)
3.) use CRF if -crf=xx (e.g. --crf=20) is specified (this is the classical CRF mode)
4.) use CQP if -qp=xx (e.g. --qp=20) is specified (this is the classical CQP mode)

According to akupenguin (http://forum.doom9.org/showthread.php?t=132760&page=24#post1091386):
Current CRF is constant quality as measured by one particular metric (not psnr or ssim or any of the normal metrics, but an ad-hoc one implicit in the CRF algorithm). It's similar bitrate on average as the same value of CQP, because I multiplied the CRF scale by a magic number to make that true, not by anything inherent in the algorithm.

CRF+HaaliAQ is constant quality as measured by another metric. It's always at least a little higher bitrate per CRF value than plain CRF because HaaliAQ has a negative average QP bias, but that's just a UI tuning issue and could be fixed if desired.

CRF+VAQ(static) is constant quality as measured by another metric. If you pick some random value of aq-sensitivity, it won't be equal on average, but will have some bias. We're trying to find a sensitivity value such that that bias averages to 0, thus it will hopefully be about the same bitrate on average as a given value of plain CRF if we do the tuning right (unless we choose to keep psy quality the same instead of bitrate).

CRF+VAQ(auto) does the frame-wise bit allocation using the same metric as plain CRF, but reallocates bits within the frame using the VAQ metric.

Was it ever decided whether to have two kinds of VAQ (auto and static)? I hope that the final VAQ implementation will have the parameter value tuned like the CRF parameter was tuned...so that the average bitrate of the output files are similar.

[Dark Shikari]

In the interest of backwards compatibility with the command lines rate control mode parameters, I suggest the following implementation (if this suggestion should also be made in another place, someone please let me know):

VAQ=variance AQ
CRF=constant rate factor
CQP=constant quantization

1.) use VAQ if rate control mode is not specified (default)
2.) use VAQ if -vaq=xx (e.g. --vaq=20) is specified
3.) use CRF if -crf=xx (e.g. --crf=20) is specified
4.) use CQP if -qp=xx (e.g. --qp=20) is specified

According to akupengiun (http://forum.doom9.org/showthread.php?t=132760&page=24#post1091386):

Was it ever decided to have types VAQ? (auto and static)? I hope that the final VAQ implementation will have the parameter value tuned like the CRF parameter was tuned...so that the average bitrate of the output files are similar.But how do you intend to implement AQ with 1pass and two pass bitrate modes?

[hkazemi]

But how do you intend to implement AQ with 1pass and two pass bitrate modes?

For completely compatible behavior with current builds, I think the bitrate modes should have a new parameter that enables VAQ. (I know this goes against the idea of VAQ being enabled by default, however I think maintaining compatibility, and maintaining the expected output is important.)

Current:
x264 --bitrate 3000
x264 --pass 1 --bitrate 3000
x264 --pass 2 --bitrate 3000

Proposed (preferred):
x264 --bitrate 3000 (VAQ not used)
x264 --pass 1 --bitrate 3000 (VAQ not used)
x264 --pass 2 --bitrate 3000 (VAQ not used)
x264 --bitrate 3000 --vaqon (VAQ used)
x264 --pass 1 --bitrate 3000 --vaqon (VAQ used)
x264 --pass 2 --bitrate 3000 --vaqon (VAQ used)

The other way, if we don't mind breaking user expectations on the codec output, is to have a new parameter that brings back the 'old' style output. This new parameter would force VAQ off. It could be something like:
--vaqoff or --novaq

Alternate proposal:
x264 --bitrate 3000 (VAQ used)
x264 --pass 1 --bitrate 3000 (VAQ used)
x264 --pass 2 --bitrate 3000 (VAQ used)
x264 --bitrate 3000 --vaqoff (VAQ not used)
x264 --pass 1 --bitrate 3000 --vaqoff (VAQ not used)
x264 --pass 2 --bitrate 3000 --vaqoff (VAQ not used)

I do not know if it makes sense to include a value with the --vaqon parameter, like --vaqon=20, when using a fixed average bitrate. If it doesn't make sense, then don't use the number. If it does make sense, the proposed --vaq=20 parameter could be overloaded to serve two similar functions, one by itself when it is the main rate control function, or two, when it is subserviant to the average bitrate rate control function.

[hkazemi]

Here's my revised recommendation for full backwards compatibility, so as not to change the expected output as obtained by older command line parameters as used in existing x264 GUIs and scripts:
ABR=average bitrate
CBR=constant bitrate
CQP=constant quantization
CRF=constant rate factor
VAQ=variance adaptive quantizer

ABR and CBR are similar in that the bitrates should be equal over the whole encode, but differ in that ABR may have areas in the video above the specified bitrate, and other areas below the specified bitrate. CBR uses the same bitrate everywhere (whether or not that means too much or too little for particular scene). CBR has traditionally been used for streams where the bitrate must not exceed a certain value, such as on cpu-constrained playback devices that cannot handle the peaks in ABR. However, if you use ABR with --vbv-maxrate, you can put a ceiling on the output stream to not exceed a particular bitrate, thereby letting the x264 encoder use fewer bits in places they are not needed and also saving space. (Using the --vbv-maxrate parameter also requires you to use the --vbv-bufsize parameter.)

The command line should specify one of the following rate control factor mechanisms:

Quality Priority/Quality First Modes:
The average bitrates generated by the same parameter values will ideally be similar.
1.) --qp=xx means use CQP (e.g. --qp=20) (this is the classical CQP/constant quantization parameter mode) (the original constant quality mode)
2.) --crf=xx means use CRF (e.g. --crf=20) (this is the classical CRF/constant rate factor mode) (the second constant quality mode)
3.) --vaq=xx means use VAQ (e.g. --vaq=20) (this is the new mode using VAQ/variance adaptive quantizer) (the newest constant quality mode, meant to reduce banding and artifacts in dark areas)

Bitrate Priority/Bitrate First Modes:
4.) --bitrate 3000 means use ABR (this is the classical single pass ABR/average bitrate mode)
5.) --pass 1 --bitrate 3000 means run the 1st pass of a 2-pass ABR (this is the classical 1st pass ABR/average bitrate mode)
6.) --pass 2 --bitrate 3000 means run the 2nd pass of a 2-pass ABR (this is the classical 2nd pass ABR/average bitrate mode)
7.) the above three modes with the addition of the --vbv-maxrate and --vbv-bufsize parameters to constrain the maximum bitrate
8.) --bitrate 3000 --qcomp 0 means use CBR (this is the classical single pass CBR/constant bitrate mode)

Bitrate Priority/Bitrate First Modes with VAQ:
9.) --bitrate 3000 --vaq means use ABR+VAQ (this is the single pass ABR/average bitrate mode with VAQ enabled)
10.) --pass 1 --bitrate 3000 --vaq means run the 1st pass of a 2-pass ABR+VAQ (this is the 1st pass ABR/average bitrate mode with VAQ enabled)
11.) --pass 2 --bitrate 3000 --vaq means run the 2nd pass of a 2-pass ABR+VAQ (this is the 2nd pass ABR/average bitrate mode with VAQ enabled)
12.) the above three modes with the addition of the --vbv-maxrate and --vbv-bufsize parameters to constrain the maximum bitrate
13.) --bitrate 3000 --qcomp 0 --vaq means use CBR+VAQ (this is the single pass CBR/constant bitrate mode with VAQ enabled)

If it makes any sense to include a value with the --vaq parameter when using ABR, like --bitrate 3000 --vaq=20, then simply add that parameter. I am suggesting overloading the --vaq parameter to serve two similar functions, first when used by itself as the main rate control function, and second when it is subserviant to the average bitrate rate control function.

Do you know if the VAQ(auto) and the VAQ(static) options still exist as described by akupenguin (quoted in my previous post above)?

Also, I suggest renaming --aq-sensitivity=xx to --vaq-sensitivity=xx (--aq-strength is going away/being turned into --vaq=xx, right?).

Finally, for a default rate control mode if none is specified on the command line, something like this default may make sense:
--vaq=20 (bitrate equivalent to --crf=20)

P.S. CBR, ABR notes, particularly on using vbv-maxrate:
http://forum.doom9.org/printthread.php?t=130125&pp=40
http://forum.doom9.org/showthread.php?t=124560
http://forum.doom9.org/showthread.php?t=102893
http://forum.doom9.org/showthread.php?p=873073
http://forum.doom9.org/showthread.php?p=699479#post699479
http://forum.handbrake.fr/viewtopic.php?f=7&t=4099&start=20&st=0&sk=t&sd=a
http://forum.handbrake.fr/viewtopic.php?f=7&t=1732&st=0&sk=t&sd=a&start=40

[Dark Shikari]

Do you know if the VAQ(auto) and the VAQ(static) options still exist as described by akupenguin (quoted in my previous post above)?Yes, they still exist, and will still exist, especially since auto is very useful for CBR mode.

[hkazemi]

Yes, they still exist, and will still exist, especially since auto is very useful for CBR mode.

So how does one choose between VAQ (auto) and VAQ (static) on the command line?

Is VAQ (static) where a 'value' is provided, as in when people currently use '--crf 20' with the 20 being the value with the AQ-enabled-by-default builds?

Is VAQ (auto) only useful for the ABR/CBR mode? (Assuming I'm safe assuming ABR=CBR (average bitrate=constant bitrate) in x264.)

[Dark Shikari]

So how does one choose between VAQ (auto) and VAQ (static) on the command line?

Is VAQ (static) where a 'value' is provided, as in when people currently use '--crf 20' with the 20 being the value with the AQ-enabled-by-default builds?

Is VAQ (auto) only useful for the ABR/CBR mode? (Assuming I'm safe assuming ABR=CBR (average bitrate=constant bitrate) in x264.)Static is useful for unrestricted VBR of any sort (CRF/QP or bitrate). Auto is useful for CBR. There might be some sort of median for restricted VBR, but I haven't done much research into that.

One idea would be to have a scale between "auto" and "static" that is set automatically based on the ratio of VBV maxrate to bitrate.

[CruNcher]

make dithering in pre-process ...

fine tuning VAQ and useing a custom matrix can do it, see my last examples for some heavy (even luma) ROI banding problems and how i (almost) fixed them (at least they become very hard perceptable) Visually under low bitrate conditions :)

[cheshire2k]

does anyone know if i can use this with fairuse wizard ?

[vpupkind]

Constant Bitrate Modes with VAQ:
7.) --bitrate 3000 --vaq means use ABR+VAQ (this is the single pass ABR/average bitrate mode with VAQ enabled)
8.) --pass 1 --bitrate 3000 --vaq means run the 1st pass of a 2-pass ABR+VAQ (this is the 1st pass ABR/average bitrate mode with VAQ enabled)
9.) --pass 2 --bitrate 3000 --vaq means run the 2nd pass of a 2-pass ABR+VAQ (this is the 2nd pass ABR/average bitrate mode with VAQ enabled)


How would you actually make AQ work with ABR?
From my tests with 0.48, I have seen relatively non-constant behavior of the bitrate.

[Sharktooth]

no. FUW is old and it uses an interface x264 does no longer support.
use a proper program wich correctly support x264 as atak_snjpera suggested in the other thread.
also have a look here: http://forum.doom9.org/showthread.php?t=129748

[Dark Shikari]

How would you actually make AQ work with ABR?
From my tests with 0.48, I have seen relatively non-constant behavior of the bitrate.I've never had a problem with AQ and ABR as long as the same settings are used on both passes.

[cheshire2k]

theres so many to choose from any idea on what to use other than automkv coz that craps out on me now :( never use to.

[LoRd_MuldeR]

Well, recent VfW builds with VAQ included can be found here:
http://sf.net/project/showfiles.php?group_id=213809

You'll have to use "x264vfw experimental", if you want to use VAQ ;)


[EDIT]

Also you might want to give Avidemux a try:
http://forum.doom9.org/showthread.php?t=126164

There is an Avidemux build with x264+VAQ available now and it works fine :)

[Adub]

theres so many to choose from any idea on what to use other than automkv coz that craps out on me now :( never use to.

You should bring it up with the author then, and try to get the problem resolved.

[cheshire2k]

already have mate with no joy. Even updated tried the beta. Ive just looked at that avidemux but I cant put my dvd iso or folder thru it im after a dvd to x264 with this vaq built in but gui so i can crop the black bars.

[LoRd_MuldeR]

Ive just looked at that avidemux but I cant put my dvd iso or folder thru it

You have already posted the same question in the "Avidemux FAQ" thread :rolleyes:
See my answer there and avoid double posting in future...

[foxyshadis]

You have already posted the same question in the "Avidemux FAQ" thread :rolleyes:
See my answer there and avoid double posting in future...

Better yet, make your own thread if you suspect you'll be asking for help on a lot of scattered topics. That way everyone's on the same page. And don't crosspost it, just pick a likely-looking forum.

[au]

That basically isn't possible in the YV12 color space.


Yes, it's a limitation of YV12. But I don't actually understand how it works in AVISynth. As I remember, YV12 uses 3x 8 bit color components (Y, Cr and Cb) per pixel, so the color depth must not be reduced.

I've started the new thread in AVISynth development (http://forum.doom9.org/showthread.php?p=1102819#post1102819)

[Wilbert]

YV12 uses 3x 8 bit color components (Y, Cr and Cb) per pixel, so the color depth must not be reduced.
YV12 uses 12 bits per pixel: 8 bits for Y and 4 bits for CbCr (the chroma is shared between four pixels - 2*8bit/4pix = 4 bits per pixel). That's why it's called YV12 :)

[G_M_C]

A slight offtopic question:
Is the VAQ option allready implemented as option in apps/GUI's like MeGUI ?

[Dark Shikari]

A slight offtopic question:
Is the VAQ option allready implemented as option in apps/GUI's like MeGUI ?You can use it through the custom commandline option in MeGUI. Hopefully they'll add an actual GUI option soon.

Its enabled by default.

[Sagittaire]

YV12 uses 12 bits per pixel: 8 bits for Y and 4 bits for CbCr (the chroma is shared between four pixels - 2*8bit/4pix = 4 bits per pixel). That's why it's called YV12 :)

well in fact chroma chanel use 8 bits but with half resolution for chroma -> imply in practice 2 bits per pixel in full resolution.

YV12 is definitively the best compromise for quality/size in losseless/lossy mode.

[MasterNobody]

Dark Shikari
may be replace
float qp_adj = 3 * (logf(energy) - h->rc->aq_threshold);
with
float qp_adj = 3 * (logf(energy) / h->rc->aq_threshold - 1);
I don't test would it be good but QP would change more gradually.

[Atak_Snajpera]

Is the VAQ option allready implemented as option in apps/GUI's like MeGUI ?

RipBot264 has AQ support from very begining.

[Dark Shikari]

RipBot264 has AQ support from very begining.I think his question is whether Ripbot has the VAQ build or a Haali's AQ (HAQ?) build.

[Atak_Snajpera]

Before VAQ it had Haali's but know VAQ is used with default settings

[chipzoller]

Might VAQ make it into the SVN for x264 anytime soon? I notice it's already up to rev. 745 with no mention in the changelog of VAQ.

[Schrade]

Is there an ffmpeg build somewhere with the VAQ patch included?

This build doesn't appear to have it:

http://oss.netfarm.it/mplayer-win32.php

[netsnake]

Might VAQ make it into the SVN for x264 anytime soon? I notice it's already up to rev. 745 with no mention in the changelog of VAQ.

I'm waiting for this too

[vpupkind]

I am seeing that with v0.48 colors look washed out.
I tried aq-strength of [0...0.5] with the default sensitivity, and it looks like the colors become more washed-out as the aq-strength grows.
Has anyone encountered this?

[Dark Shikari]

I am seeing that with v0.48 colors look washed out.
I tried aq-strength of [0...0.5] with the default sensitivity, and it looks like the colors become more washed-out as the aq-strength grows.
Has anyone encountered this?Are you sure you're not just opening up two videos at once in two media players, and so one's luma isn't being converted properly by the renderer?

AQ does not affect color or brightness.

[bob0r]

Might VAQ make it into the SVN for x264 anytime soon? I notice it's already up to rev. 745 with no mention in the changelog of VAQ.

VAQ is still expermimental.

http://x264.nl is updated with GIT (original source) revision 748 + a link to 748 patched, the lack of sleep and some other issues caused me to make it unclear.
12 hours sleep made me look clear again :D ( yeah yeah pengvado, sleep is for pussies )

Anyways here is 748 patched:



Thanks to akupenguin (pengvado) x264.nl is auto updating again!!

So here is 748 (based on GIT updates count) + fixed HRD interlacing patch

x264.748.modified.exe (http://files.x264.nl/x264.748.modified.exe)

General thread:
http://forum.doom9.org/showthread.php?t=130364

x264_aq_var.48.diff
http://forum.doom9.org/showthread.php?t=132760
x264.gaussian.cplxblur.01.diff
Dark Shikari: - gaussian cplxblur: gives a tiny improvement in 2pass ratecontrol
x264_me-prepass_DeathTheSheep.01.diff
http://forum.doom9.org/showthread.php?p=1093523
x264_2pass_vbv.6.diff
http://thread.gmane.org/gmane.comp.video.x264.devel/3093/focus=3550
x264_hrd_pulldown.04_interlace.diff
- HRD and pulldown for HD compatibility, updated patch for interlacing
http://forum.doom9.org/showthread.php?p=1047919#post1047919

Link to x264 patches collected: http://files.x264.nl/x264_patches/

[vpupkind]

Brightness is not influenced, but chroma seems to be.
I tried displaying things in VLC and Nero, running 2-3 instances at the same time, side by side, on several different LCD displays; using both v0.48 / r747 and r736/v0.47.

[Dark Shikari]

Brightness is not influenced, but chroma seems to be.
I tried displaying things in VLC and Nero, running 2-3 instances at the same time, side by side, on several different LCD displays; using both v0.48 / r747 and r736/v0.47.Have you tried *not* displaying them at the same time to avoid renderer issues?

(I highly doubt you're getting a chroma shift from AQ).

[CruNcher]

Are you sure you're not just opening up two videos at once in two media players, and so one's luma isn't being converted properly by the renderer?

AQ does not affect color or brightness.

remember remember the 5th of december ;) *insider* (just ignore this)

[aking80]

Sorry to ask if this has been already but too many pages to read ~_~

Is this patches version the one that MeGUI automatically updates to (763 - Jared's Patched Build)? Also since there's no GUI option for these settings in MeGUI and yet it doesn't show in the command line... I thought this was "on" by default?

[Dark Shikari]

Sorry to ask if this has been already but too many pages to read ~_~

Is this patches version the one that MeGUI automatically updates to (763 - Jared's Patched Build)? Also since there's no GUI option for these settings in MeGUI and yet it doesn't show in the command line... I thought this was "on" by default?Yes, its on by default.

[Dark Shikari]

VAQ 1.0 has been released to Akupenguin for integrating with the official x264.

And here's a preview of VAQ 2.0 Pre-Alpha (links go to videos). The difference is... wow.
SSIM with VAQ 2.0 strength 1.0 (http://www.mediafire.com/?nmzhuturdm2): 0.9092416 (48.1% improvement over no AQ)
SSIM with VAQ 2.0 (http://www.mediafire.com/?vdimcbmzdiz): 0.9016686 (36.7% improvement over no AQ)
SSIM with VAQ 0.48/1.0 (http://www.mediafire.com/?dhastytmz5w): 0.8881315 (13.8% improvement over no AQ)
SSIM with no VAQ (http://www.mediafire.com/?d1byyzumied): 0.8655435

Settings: --bframes 3 --no-b-adapt --bime --weightb --subme 7 --keyint 300 --ref 16 --trellis 2 --mixed-refs --8x8dct --partitions all --b-rdo --direct auto --b-pyramid --pass 2 --bitrate 3700 --aq-strength 0.65 --no-fast-pskip --me tesa
Source: First 200 non-gray frames of parkrun.yuv

Thanks to Alex W for the basic idea behind VAQ 2.0.

API changes in the upcoming VAQ 1.0:

--aq-strength 1.0 is equivalent to the old --aq-strength 0.5. Akupenguin insisted on this. Therefore, 1.0 is now default, and everything else scales accordingly.

--aq-sensitivity no longer exists.

--aq-mode chooses between off (0), auto (1), and static sensitivity (2 [default] ).

Possible API changes in VAQ 2.0 could include:

--aq-metric, which would choose between the following (subject to change on a whim by me based on further developments):
0: Current metric from VAQ 0.48/1.0 (estimated speed: ~500 clocks per MB)
1: Block-based 8x8 window search of 1/4 of the pixels in each block (estimated speed: ~6,000 clocks per MB)
2: 7x7 Gaussian window search of 1/4 of the pixels in each block (estimated speed: ~15,000 clocks per MB)
3: 7x7 Gaussian window search of all the pixels in each block (estimated speed: ~60,000 clocks per MB)

[DeathTheSheep]

If it's as good as 0.46 was, I'm more than happy. :) Well, I'm happy anyway, just saying. :cool:

[J_Darnley]

Very nice work Dark Shikari. I do have a few questions though. With VAQ 1.0 can you turn it off with both --aq-strength 0 and --aq-mode 0? Is the current static mode equivalent/similar to the former static sensitivity? Also what is featured in 2.0 that makes such a large improvement?

Oh yes, any links to 1.0 or 2.0 or are we not allowed give them a test?
[EDIT] Ah I just saw your posts here: http://forum.doom9.org/showthread.php?p=1115512#1115512

[Fantacinni]

Why I can't make x264 r786 with x264_aq_var.48?

[bob0r]

Why I can't make x264 r786 with x264_aq_var.48?

Because some code has changed in x264.
Now we truely have to wait for AQ to hit git :D

Revision 785 is the last where current AQ 0.48 compiling works.

All other patches still work.

[MythCreator]

Because some code has changed in x264.
Now we truely have to wait for AQ to hit git :D

Revision 785 is the last where current AQ 0.48 compiling works.

All other patches still work.


Is that means AQ 1.0 will be soon?

[Sagittaire]

The patch is here
http://mailman.videolan.org/pipermail/x264-devel/2008-March/004221.html

... but don't work for me. Problem at line 5 in compilation.

diff --git a/common/common.c b/common/common.c
index 44d9113..17f37f5 100644
--- a/common/common.c
+++ b/common/common.c
@@ -123,6 +123,8 @@ void x264_param_default( x264_param_t *param )
param->analyse.i_chroma_qp_offset = 0;
param->analyse.b_fast_pskip = 1;
param->analyse.b_dct_decimate = 1;
+ param->analyse.f_aq_strength = 0.5;
+ param->analyse.i_aq_mode = 2;
param->analyse.i_luma_deadzone[0] = 21;
param->analyse.i_luma_deadzone[1] = 11;
param->analyse.b_psnr = 1;
@@ -455,6 +457,10 @@ int x264_param_parse( x264_param_t *p, const char
*name, const char *value )
p->analyse.b_fast_pskip = atobool(value);
OPT("dct-decimate")
p->analyse.b_dct_decimate = atobool(value);
+ OPT("aq-strength")
+ p->analyse.f_aq_strength = atof(value);
+ OPT("aq-mode")
+ p->analyse.i_aq_mode = atoi(value);
OPT("deadzone-inter")
p->analyse.i_luma_deadzone[0] = atoi(value);
OPT("deadzone-intra")
@@ -883,6 +889,10 @@ char *x264_param2string( x264_param_t *p, int b_res )
s += sprintf( s, " ip_ratio=%.2f", p->rc.f_ip_factor );
if( p->i_bframe )
s += sprintf( s, " pb_ratio=%.2f", p->rc.f_pb_factor );
+ if( p->analyse.i_aq_mode )
+ s += sprintf( s, " aq=%d:%.1f", p->analyse.i_aq_mode,
p->analyse.f_aq_strength );
+ else
+ s += sprintf( s, " aq=0" );
if( p->rc.psz_zones )
s += sprintf( s, " zones=%s", p->rc.psz_zones );
else if( p->rc.i_zones )
diff --git a/encoder/analyse.c b/encoder/analyse.c
index 0264621..0f313a9 100644
--- a/encoder/analyse.c
+++ b/encoder/analyse.c
@@ -2064,8 +2064,13 @@ void x264_macroblock_analyse( x264_t *h )
int i_cost = COST_MAX;
int i;

- /* init analysis */
- x264_mb_analyse_init( h, &analysis, x264_ratecontrol_qp( h ) );
+ h->mb.i_qp = x264_ratecontrol_qp( h );
+
+ if( h->param.analyse.i_aq_mode )
+ x264_adaptive_quant( h );
+
+ /* init analysis */
+ x264_mb_analyse_init( h, &analysis, h->mb.i_qp );

/*--------------------------- Do the analysis ---------------------------*/
if( h->sh.i_type == SLICE_TYPE_I )
diff --git a/encoder/encoder.c b/encoder/encoder.c
index 3dadb02..3bd9e70 100644
--- a/encoder/encoder.c
+++ b/encoder/encoder.c
@@ -401,6 +401,7 @@ static int x264_validate_parameters( x264_t *h )
h->param.analyse.b_fast_pskip = 0;
h->param.analyse.i_noise_reduction = 0;
h->param.analyse.i_subpel_refine = x264_clip3(
h->param.analyse.i_subpel_refine, 1, 6 );
+ h->param.analyse.i_aq_mode = 0;
}
if( h->param.rc.i_rc_method == X264_RC_CQP )
{
@@ -475,6 +476,11 @@ static int x264_validate_parameters( x264_t *h )
if( !h->param.b_cabac )
h->param.analyse.i_trellis = 0;
h->param.analyse.i_trellis = x264_clip3(
h->param.analyse.i_trellis, 0, 2 );
+ h->param.analyse.i_aq_mode = x264_clip3(h->param.analyse.i_aq_mode, 0, 2);
+ if(h->param.analyse.f_aq_strength <= 0) h->param.analyse.i_aq_mode = 0;
+ /* VAQ on mode 1 effectively replaces qcomp, so qcomp is raised
towards 1 to compensate. */
+ if(h->param.analyse.i_aq_mode == 2)
+ h->param.rc.f_qcompress = x264_clip3f(h->param.rc.f_qcompress
+ h->param.analyse.f_aq_strength * 0.4 / 0.28, 0, 1);
h->param.analyse.i_noise_reduction = x264_clip3(
h->param.analyse.i_noise_reduction, 0, 1<<16 );

{
diff --git a/encoder/ratecontrol.c b/encoder/ratecontrol.c
index 0c8a6d7..30c0994 100644
--- a/encoder/ratecontrol.c
+++ b/encoder/ratecontrol.c
@@ -127,6 +127,10 @@ struct x264_ratecontrol_t
predictor_t *pred_b_from_p; /* predict B-frame size from P-frame satd */
int bframes; /* # consecutive B-frames before this
P-frame */
int bframe_bits; /* total cost of those frames */
+
+ /* AQ stuff */
+ float aq_threshold;
+ int *ac_energy;

int i_zones;
x264_zone_t *zones;
@@ -169,6 +173,92 @@ static inline double
qscale2bits(ratecontrol_entry_t *rce, double qscale)
+ rce->misc_bits;
}

+// Find the total AC energy of the block in all planes.
+static int ac_energy_mb( x264_t *h, int mb_x, int mb_y, int *satd )
+{
+ DECLARE_ALIGNED( static uint8_t, flat[16], 16 ) =
{128,128,128,128,128,128,128,128,128,128,128,128,128,128,128,128};
+ unsigned int var=0, sad, ssd, i;
+ for( i=0; i<3; i++ )
+ {
+ int w = i ? 8 : 16;
+ int stride = h->fenc->i_stride[i];
+ int offset = h->mb.b_interlaced
+ ? w * (mb_x + (mb_y&~1) * stride) + (mb_y&1) * stride
+ : w * (mb_x + mb_y * stride);
+ int pix = i ? PIXEL_8x8 : PIXEL_16x16;
+ stride <<= h->mb.b_interlaced;
+ sad = h->pixf.sad[pix](flat, 0, h->fenc->plane[i]+offset, stride);
+ ssd = h->pixf.ssd[pix](flat, 0, h->fenc->plane[i]+offset, stride);
+ var += ssd - (sad * sad >> (i?6:8));
+ // SATD to represent the block's overall complexity (bit
cost) for intra encoding.
+ // exclude the DC coef, because nothing short of an actual
intra prediction will estimate DC cost.
+ if( var && satd )
+ *satd += h->pixf.satd[pix](flat, 0,
h->fenc->plane[i]+offset, stride) - sad/2;
+ }
+ return var;
+}
+
+void x264_autosense_aq( x264_t *h )
+{
+ double total = 0;
+ double n = 0;
+ int mb_x, mb_y;
+ /* FIXME: Some of the SATDs might be already calculated elsewhere
(ratecontrol?). Can we reuse them? */
+ /* FIXME: Is chroma SATD necessary? */
+ for( mb_y=0; mb_y<h->sps->i_mb_height; mb_y++ )
+ for( mb_x=0; mb_x<h->sps->i_mb_width; mb_x++ )
+ {
+ int energy, satd=0;
+ energy = ac_energy_mb( h, mb_x, mb_y, &satd );
+ h->rc->ac_energy[mb_x + mb_y * h->sps->i_mb_width] = energy;
+ /* Weight the energy value by the SATD value of the MB.
This represents the fact that
+ the more complex blocks in a frame should be weighted
more when calculating the optimal threshold.
+ This also helps diminish the negative effect of large
numbers of simple blocks in a frame, such as in the case
+ of a letterboxed film. */
+ if( energy )
+ {
+ x264_cpu_restore(h->param.cpu);
+ total += logf(energy) * satd;
+ n += satd;
+ }
+ }
+ x264_cpu_restore(h->param.cpu);
+ /* Calculate and store the threshold. */
+ h->rc->aq_threshold = n ? total/n : 15;
+}
+
+/*****************************************************************************
+* x264_adaptive_quant:
+ * adjust macroblock QP based on variance (AC energy) of the MB.
+ * high variance = higher QP
+ * low variance = lower QP
+ * This generally increases SSIM and lowers PSNR.
+*****************************************************************************/
+void x264_adaptive_quant( x264_t *h )
+{
+ int qp = h->mb.i_qp;
+ int energy;
+ if(h->param.analyse.i_aq_mode == 2)
+ energy = ac_energy_mb( h, h->mb.i_mb_x, h->mb.i_mb_y, NULL );
+ else
+ energy = h->rc->ac_energy[h->mb.i_mb_xy];
+ if(energy == 0)
+ h->mb.i_qp = h->mb.i_last_qp;
+ else
+ {
+ x264_cpu_restore(h->param.cpu);
+ float result = energy;
+ /* Adjust the QP based on the AC energy of the macroblock. */
+ float qp_adj = 3 * (logf(result) - h->rc->aq_threshold);
+ if(h->param.analyse.i_aq_mode == 1) qp_adj =
x264_clip3f(qp_adj, -5, 5);
+ int new_qp = x264_clip3(qp + qp_adj *
h->param.analyse.f_aq_strength + .5, h->param.rc.i_qp_min,
h->param.rc.i_qp_max);
+ /* If the QP of this MB is within 1 of the previous MB, code
the same QP as the previous MB,
+ * to lower the bit cost of the qp_delta. */
+ if(abs(new_qp - h->mb.i_last_qp) == 1) new_qp = h->mb.i_last_qp;
+ h->mb.i_qp = new_qp;
+ }
+ h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp +
h->pps->i_chroma_qp_index_offset, 0, 51 )];
+}

int x264_ratecontrol_new( x264_t *h )
{
@@ -244,7 +334,7 @@ int x264_ratecontrol_new( x264_t *h )
rc->rate_tolerance = 0.01;
}

- h->mb.b_variable_qp = rc->b_vbv && !rc->b_2pass;
+ h->mb.b_variable_qp = (rc->b_vbv && !rc->b_2pass) ||
h->param.analyse.i_aq_mode;

if( rc->b_abr )
{
@@ -458,10 +548,13 @@ int x264_ratecontrol_new( x264_t *h )
x264_free( p );
}

- for( i=1; i<h->param.i_threads; i++ )
+ for( i=0; i<h->param.i_threads; i++ )
{
h->thread[i]->rc = rc+i;
- rc[i] = rc[0];
+ if( i )
+ rc[i] = rc[0];
+ if( h->param.analyse.i_aq_mode == 1 )
+ rc[i].ac_energy = x264_malloc( h->mb.i_mb_count * sizeof(int) );
}

return 0;
@@ -623,6 +716,8 @@ void x264_ratecontrol_delete( x264_t *h )
x264_free( rc->zones[i].param );
x264_free( rc->zones );
}
+ for( i=0; i<h->param.i_threads; i++ )
+ x264_free( rc[i].ac_energy );
x264_free( rc );
}

@@ -729,6 +824,12 @@ void x264_ratecontrol_start( x264_t *h, int i_force_qp )

if( h->sh.i_type != SLICE_TYPE_B )
rc->last_non_b_pict_type = h->sh.i_type;
+
+ /* Adaptive AQ thresholding algorithm. */
+ if( h->param.analyse.i_aq_mode == 2 )
+ h->rc->aq_threshold = logf(14280.0); /* Arbitrary value for
"center" of AQ curve. */
+ else if( h->param.analyse.i_aq_mode == 1 )
+ x264_autosense_aq(h);
}

double predict_row_size( x264_t *h, int y, int qp )
diff --git a/encoder/ratecontrol.h b/encoder/ratecontrol.h
index d4af2c0..e8b2ea1 100644
--- a/encoder/ratecontrol.h
+++ b/encoder/ratecontrol.h
@@ -34,6 +34,7 @@ void x264_ratecontrol_mb( x264_t *, int bits );
int x264_ratecontrol_qp( x264_t * );
void x264_ratecontrol_end( x264_t *, int bits );
void x264_ratecontrol_summary( x264_t * );
+void x264_adaptive_quant ( x264_t * );

#endif

diff --git a/x264.c b/x264.c
index f68755d..cf318ed 100644
--- a/x264.c
+++ b/x264.c
@@ -244,6 +244,14 @@ static void Help( x264_param_t *defaults, int b_longhelp )
" - 2: enabled on all mode
decisions\n", defaults->analyse.i_trellis );
H0( " --no-fast-pskip Disables early SKIP detection
on P-frames\n" );
H0( " --no-dct-decimate Disables coefficient
thresholding on P-frames\n" );
+ H0( " --aq-strength <float> Reduces blocking and blurring
in flat and\n"
+ " textured areas. [%.1f]\n"
+ " - 0.2: weak AQ\n"
+ " - 1.0: very strong AQ\n",
defaults->analyse.f_aq_strength );
+ H0( " --aq-mode <integer> How AQ distributes bits [%d]\n"
+ " - 0: Disabled\n"
+ " - 1: Avoid moving bits
between frames\n"
+ " - 2: Move bits between
frames\n", defaults->analyse.i_aq_mode );
H0( " --nr <integer> Noise reduction [%d]\n",
defaults->analyse.i_noise_reduction );
H1( "\n" );
H1( " --deadzone-inter <int> Set the size of the inter luma
quantization deadzone [%d]\n", defaults->analyse.i_luma_deadzone[0] );
@@ -407,6 +415,8 @@ static int Parse( int argc, char **argv,
{ "trellis", required_argument, NULL, 't' },
{ "no-fast-pskip", no_argument, NULL, 0 },
{ "no-dct-decimate", no_argument, NULL, 0 },
+ { "aq-strength", required_argument, NULL, 0 },
+ { "aq-mode", required_argument, NULL, 0 },
{ "deadzone-inter", required_argument, NULL, '0' },
{ "deadzone-intra", required_argument, NULL, '0' },
{ "level", required_argument, NULL, 0 },
diff --git a/x264.h b/x264.h
index 70c9eaf..c367c41 100644
--- a/x264.h
+++ b/x264.h
@@ -232,6 +232,8 @@ typedef struct x264_param_t
int i_trellis; /* trellis RD quantization */
int b_fast_pskip; /* early SKIP detection on P-frames */
int b_dct_decimate; /* transform coefficient
thresholding on P-frames */
+ float f_aq_strength; /* psy adaptive QP */
+ int i_aq_mode; /* 0 = off, 1 = auto, 2 = static sensitivity */
int i_noise_reduction; /* adaptive pseudo-deadzone */

/* the deadzone size that will be used in luma quantization */

[bob0r]

fix:
change this part:
DECLARE_ALIGNED( static uint8_t, flat[16], 16 ) =
to
DECLARE_ALIGNED_16( static uint8_t flat[16] ) =

Also works for AQ 0.48.

Dark_Shikari still recommends we use 0.48, as 1.0 will have changed defaults, the same workings, just other values!

[J_Darnley]

I've already done that for 0.48 (and called it 0.49) and it still fails.
encoder/ratecontrol.c:179:50: macro "DECLARE_ALIGNED_16" passed 2 arguments, but takes just 1
encoder/ratecontrol.c:179:50: macro "DECLARE_ALIGNED_16" passed 2 arguments, but takes just 1
encoder/ratecontrol.c: In function `ac_energy_mb':
encoder/ratecontrol.c:179: error: `DECLARE_ALIGNED_16' undeclared (first use in this function)
encoder/ratecontrol.c:179: error: (Each undeclared identifier is reported only once
encoder/ratecontrol.c:179: error: for each function it appears in.)
encoder/ratecontrol.c:179: error: syntax error before '{' token
encoder/ratecontrol.c:179: warning: no return statement in function returning non-void
encoder/ratecontrol.c: At top level:
encoder/ratecontrol.c:182: error: syntax error before "for"
encoder/ratecontrol.c:185: error: `h' undeclared here (not in a function)
encoder/ratecontrol.c:187: error: `w' undeclared here (not in a function)
encoder/ratecontrol.c:187: error: `mb_x' undeclared here (not in a function)
encoder/ratecontrol.c:187: error: `mb_y' undeclared here (not in a function)
encoder/ratecontrol.c:189: error: initializer element is not constant
encoder/ratecontrol.c:190: error: syntax error before '<<=' token
encoder/ratecontrol.c:191: warning: type defaults to `int' in declaration of `sad'
encoder/ratecontrol.c:191: error: conflicting types for 'sad'
encoder/ratecontrol.c:181: error: previous declaration of 'sad' was here
encoder/ratecontrol.c:191: error: `flat' undeclared here (not in a function)
encoder/ratecontrol.c:191: warning: data definition has no type or storage class
encoder/ratecontrol.c:192: warning: type defaults to `int' in declaration of `ssd'
encoder/ratecontrol.c:192: error: conflicting types for 'ssd'
encoder/ratecontrol.c:181: error: previous declaration of 'ssd' was here
encoder/ratecontrol.c:192: warning: data definition has no type or storage class
encoder/ratecontrol.c:193: error: syntax error before '+=' token
encoder/ratecontrol.c: In function `parse_zone':
encoder/ratecontrol.c:570: warning: unused variable `saveptr'
encoder/ratecontrol.c: In function `parse_zones':
encoder/ratecontrol.c:613: warning: unused variable `saveptr'
make: *** [encoder/ratecontrol.o] Error 1
Is this an error on my end and/or do I need some other patch or make some other change? I tried using DECLARE_ALIGNED and it removed the problems with that but then it bitched about flat and the errors about w, h mb_x and mb_y remained. Also what I don't understand is why a 'simple' cosmetics change causes all these errors, ratecontrol.c wasn't changed in the last commit.

Anyway, I am sticking with r785 for now as the cosmetics don't make any change to the binary (right?).

[addit]

VAQ 2.0, impressive, most-impressive...

[bob0r]

@J_Darnley

You must have make a typ0 somewhere, can you paste the line you edited?

And yes you can use 785 just as fine.

[microchip8]

@ J_Darnley

you forgot to remove a comma between the ( .... ). The below code is correct

( static uint8_t flat[16] )

[Zep]

VAQ 1.0 has been released to Akupenguin for integrating with the official x264.

And here's a preview of VAQ 2.0 Pre-Alpha (links go to videos). The difference is... wow.
SSIM with VAQ 2.0 strength 1.0 (http://www.mediafire.com/?nmzhuturdm2): 0.9092416 (48.1% improvement over no AQ)


very nice detail but the contrast takes a hit and the over all encode looks a bit flat/washed out? Can that be improved upon? :D

[DeathTheSheep]

Where is this VAQ 2.0? Closed testing only? Or pre-pre-pre-pre-alpha? Or both? :) 50% SSIM improvement is surely nothing to scoff at.

[J_Darnley]

@ J_Darnley

you forgot to remove a comma between the ( .... ). The below code is correct

( static uint8_t flat[16] )
Thank you. I missed that lack of comma from bob0r's post.

Where is this VAQ 2.0? Closed testing only? Or pre-pre-pre-pre-alpha? Or both? :) 50% SSIM improvement is surely nothing to scoff at.
See my previous post where I asked the same question but then found the answer in another thread: http://forum.doom9.org/showthread.php?p=1115512#1115512
Its not released yet, still in early development ;)

1.0 will go into official GIT soon.

[Dark Shikari]

very nice detail but the contrast takes a hit and the over all encode looks a bit flat/washed out? Can that be improved upon? :DWhy do people keep opening multiple windows when comparing video, and then complain about the overlay not doing the proper TV -> PC luma conversion? :rolleyes:

[BoNz1]

VAQ 1.0 has been released to Akupenguin for integrating with the official x264.

And here's a preview of VAQ 2.0 Pre-Alpha (links go to videos). The difference is... wow.
SSIM with VAQ 2.0 strength 1.0 (http://www.mediafire.com/?nmzhuturdm2): 0.9092416 (48.1% improvement over no AQ)
SSIM with VAQ 2.0 (http://www.mediafire.com/?vdimcbmzdiz): 0.9016686 (36.7% improvement over no AQ)
SSIM with VAQ 0.48/1.0 (http://www.mediafire.com/?dhastytmz5w): 0.8881315 (13.8% improvement over no AQ)
SSIM with no VAQ (http://www.mediafire.com/?d1byyzumied): 0.8655435


Wow, that is absolutely huge. Seriously. Just look at the branches of the trees in the background and the ripples on the water in the foreground. They are completely washed out without VAQ but with it they are much clearer. Nicely done.

[bcrabl]

The difference is huge!

[burfadel]

The most striking thing is the AQ files are actually slightly smaller, so in effect VAQ2 strength 1 is even more efficient than 48.1% when taking in to account file size! (as you can slightly lower the crf).

[Dark Shikari]

The most striking thing is the AQ files are actually slightly smaller, so in effect VAQ2 strength 1 is even more efficient than 48.1% when taking in to account file size! (as you can slightly lower the crf).They're encoded in target bitrate mode (twopass), not CRF.

[burfadel]

ah ok! that does make for better comparison! If its a target bitrate mode, how come the no-aq file is larger?

[Dark Shikari]

ah ok! that does make for better comparison! If its a target bitrate mode, how come the no-aq file is larger?Because its only 200 frames, so the 2pass can't really get the bitrate perfectly.

Also note that I have already found some flaws in the algorithm used here that cause problems on some other sources, so as I said its definitely in early development. I have no doubt they can be fixed though.

[Zep]

Why do people keep opening multiple windows when comparing video, and then complain about the overlay not doing the proper TV -> PC luma conversion? :rolleyes:

huh? just 1 window. over lay is off anyway. I looked at one closed it then opened the new one.


NOTE: lets test your theory. I will open both in different order. Still washed out. Now I will open 2 copies of the same clip. exact same appearance. So we know that is not the problem.

[Razorholt]

@Darky: Can you please give us a rough idea on when both VAQ 1.0 and VQ 2.0 will be available for testing at least? I am about to encode more than 150 videos and I don't want to redo the whole batch after I find out your VAQ would have given a better result :D

Cheers,
- Dan

[Dark Shikari]

@Darky: Can you please give us a rough idea on when both VAQ 1.0 and VQ 2.0 will be available for testing at least? I am about to encode more than 150 videos and I don't want to redo the whole batch after I find out your VAQ would have given a better result :D

Cheers,
- DanMost current modified builds contain VAQ1 (0.48, technically, but 1.0 is just the cleaned up version for release; no algorithmic changes).

VAQ2 will be in a few weeks to months.

[DeathTheSheep]

Nice. Getting it perfected a bit, eh?

One question. Based on your testing so far, how good is it on anime?

[Dark Shikari]

Nice. Getting it perfected a bit, eh?

One question. Based on your testing so far, how good is it on anime?I just did some testing... and I'm not really sure.

In some cases AQ is vastly better than no AQ; in other cases its noticeably worse (due to the bits being moved from one place to another, lowering quality in those original places). There isn't too much difference between VAQ1 and VAQ2.

[DeathTheSheep]

Ah, you win some, you lose some. :(

Would you say it beats 0.46 for anime, assuming CQP (I know, I know...but still)?

[Dark Shikari]

Ah, you win some, you lose some. :(

Would you say it beats 0.46 for anime, assuming CQP (I know, I know...but still)?I really can't say at this point whether AQ is better than non-AQ or what. Almost all the AQ algorithms are nearly exactly the same for anime in terms of results, except that VAQ2 weights single edges somewhat lower and multiple tightly spaced edges a lot higher.

The general end result is that x264 looks more like Xvid; more blurred/ringy edges at low bitrates, but less blurring of background detail.

One of the general problems I'm encountering is that sharp edges cause every single VAQ so far to raise the quantizer a lot, even though this might not be justified. Perhaps I need an AQ metric that can somehow measure complexity independent of such edges?

Edit: I did a bit more looking over my tests and it appears the primary "problem" is that bits are redistributed--some scenes get more than before, some less. This actually seems to be a good thing overall, since there's huge amounts of background blurring that are fixed by AQ. Overall I'd say its a positive at this point, and the various VAQs are mostly indistinguishable. There's still room for improvement though.

[burfadel]

Modern animation usaually has very flat blocks (I say modern as for example, the original Tom & Jerry cartoons for the 40's were textured), so couldn't there be a way for the encoder to realise its an animation or animation segment and maybe lower the strength or change the sensitivity etc based on this? That would solve the lower quality with animation problem.

[ImmortAlex]

Smells like "cartoon mode" in XviD :)

[Lele-brz]

I was very impress by the VAQ 2.0 and I'm looking forward to using it.
I wrote a tool to compare two videos in the same window, and you can easily see how big the improvement is.
I don't know the policy about posting a link to a software and I don't even want to go off topic.

Anyway the software can be found here:
http://www.mediafire.com/?2j1h211ba1m

It's for Windows, after launching "CompVideo" just left click to switch from one version to the other.

Bye

PS: Hope this doesn't violate any posting rule.

[burfadel]

Kinda like cartoon mode I guess, except I mean a completely automatic decision based on the flatness of the current section. SO for normal video its at the default AQ settings, and when large flatness is detected a trimming of the strength or adjustment of the sensitivity etc so as to not lose line detail.

[Sharktooth]

I was very impress by the VAQ 2.0 and I'm looking forward to using it.
I wrote a tool to compare two videos in the same window, and you can easily see how big the improvement is.
I don't know the policy about posting a link to a software and I don't even want to go off topic.

Anyway the software can be found here:
http://www.mediafire.com/?2j1h211ba1m

It's for Windows, after launching "CompVideo" just left click to switch from one version to the other.

Bye

PS: Hope this doesn't violate any posting rule.
it should be ok unless you infested your software with viruses, malwares, etc.. :p
in that case, revenge is a meal best server cold... :D

[DeathTheSheep]

Lol, are you thinking Sony rootkits and DRM, Sharktooth? :p

[burfadel]

I guess the variable VAQ (VVAQ?!) I suggested earlier, where the strength is reduced and/or the sensitivity increased for clips with a high level of flatness (ie most animation) etc to overcome sharp line degradation wouldn't actually work?

[akupenguin]

Modern animation usaually has very flat blocks (I say modern as for example, the original Tom & Jerry cartoons for the 40's were textured), so couldn't there be a way for the encoder to realise its an animation or animation segment and maybe lower the strength or change the sensitivity etc based on this? That would solve the lower quality with animation problem.
Probably possible, but what do you gain by that? Detecting animation is just a user-interface convenience, a shortcut for saying "so use these other settings on anime". The hard part is writing the alternate algorithm that works there.

[burfadel]

The idea was more to automatically scale back the aq settings when flat scenes are detected, so bitrate is not lost where its not needed (say a bright very flat floor for example in animation). It seems what people refer to as a loss of clarity is in animation that is bright, and due to the smoothness of the surrounding pieces around the lines the shifting of bits from those lines becomes more noticeable. Although the settings can be done manually, by having it set to auto people not acustomed to choosing their own settings based on the source will benefit, as well as everyone else due to simplifying the adjustment of options.

Automatic AQ would be harder to implement for 2 pass, but the idea I had for CRF is to have say, the default strength of 0.5 and sensitivity 13 for normal shots, and then when a flatness threshold is achieved scale it back to say 0.4 and sensitivity 14 or something along those lines. The strength being +0.5 or -2. and sensitivity -1 to +4 would be good compromise and be suited for most sources in my opinion (maybe slightly less than that).

Wasn't automatic thresholding etc exist in earlier revisions? maybe that part can be reintroduced, possibly with the ability to turn it off. Anyways, a strength of 0.3 and sensitivity of 16 or so does seem better for some animation clips.

[DeathTheSheep]

Yeah, since I did promise to maintain it, and 2.0 may be a few days/weeks/months/hours away, why not have at 0.46 in the meantime. Special edition updated for 798. :p

Index: encoder/ratecontrol.h
===================================================================
--- encoder/ratecontrol.h (revision 721)
+++ encoder/ratecontrol.h (working copy)
@@ -34,6 +34,8 @@
int x264_ratecontrol_qp( x264_t * );
void x264_ratecontrol_end( x264_t *, int bits );
void x264_ratecontrol_summary( x264_t * );
+void x264_autosense_aq ( x264_t *);
+void x264_adaptive_quant ( x264_t * );

#endif

Index: encoder/encoder.c
===================================================================
--- encoder/encoder.c (revision 721)
+++ encoder/encoder.c (working copy)
@@ -472,6 +472,8 @@
if( !h->param.b_cabac )
h->param.analyse.i_trellis = 0;
h->param.analyse.i_trellis = x264_clip3( h->param.analyse.i_trellis, 0, 2 );
+ if( h->param.analyse.b_aq && h->param.analyse.f_aq_strength <= 0 )
+ h->param.analyse.b_aq = 0;
h->param.analyse.i_noise_reduction = x264_clip3( h->param.analyse.i_noise_reduction, 0, 1<<16 );

{
@@ -1046,6 +1048,18 @@
h->mb.i_last_qp = h->sh.i_qp;
h->mb.i_last_dqp = 0;

+ /* Adaptive AQ sensitivity algorithm. */
+ if(h->param.analyse.b_aq)
+ {
+ x264_cpu_restore(h->param.cpu);
+ if(h->param.analyse.f_aq_sensitivity != 0)
+ h->aq_threshold = powf(h->param.analyse.f_aq_sensitivity,4)/2;
+ else
+ {
+ x264_autosense_aq(h);
+ }
+ }
+
for( mb_xy = h->sh.i_first_mb, i_skip = 0; mb_xy < h->sh.i_last_mb; )
{
const int i_mb_y = mb_xy / h->sps->i_mb_width;
Index: encoder/ratecontrol.c
===================================================================
--- encoder/ratecontrol.c (revision 721)
+++ encoder/ratecontrol.c (working copy)
@@ -126,6 +126,9 @@
predictor_t *pred_b_from_p; /* predict B-frame size from P-frame satd */
int bframes; /* # consecutive B-frames before this P-frame */
int bframe_bits; /* total cost of those frames */
+
+ /* AQ stuff */
+ int *ac_energy;

int i_zones;
x264_zone_t *zones;
@@ -168,7 +171,134 @@
+ rce->misc_bits;
}

+//Finds the total AC energy of the block in all planes. Does not require the MB to be cached.
+static int ac_energy_mb_uncached(x264_t *h, int i_pix_offset0, int i_pix_offset1)
+{
+ DECLARE_ALIGNED_16( static uint8_t zero[16] );
+ int sad = h->pixf.sad[PIXEL_16x16](zero,0,&h->fenc->plane[0][i_pix_offset0], h->fenc->i_stride[0] << h->mb.b_interlaced) >> 4;
+ int ssd = h->pixf.ssd[PIXEL_16x16](zero,0,&h->fenc->plane[0][i_pix_offset0], h->fenc->i_stride[0] << h->mb.b_interlaced);
+ int totalSSD = ssd - (sad * sad);
+ sad = h->pixf.sad[PIXEL_8x8](zero,0,&h->fenc->plane[1][i_pix_offset1], h->fenc->i_stride[1] << h->mb.b_interlaced) >> 3;
+ ssd = h->pixf.ssd[PIXEL_8x8](zero,0,&h->fenc->plane[1][i_pix_offset1], h->fenc->i_stride[1] << h->mb.b_interlaced);
+ totalSSD += ssd - (sad * sad);
+ sad = h->pixf.sad[PIXEL_8x8](zero,0,&h->fenc->plane[2][i_pix_offset1], h->fenc->i_stride[2] << h->mb.b_interlaced) >> 3;
+ ssd = h->pixf.ssd[PIXEL_8x8](zero,0,&h->fenc->plane[2][i_pix_offset1], h->fenc->i_stride[2] << h->mb.b_interlaced);
+ totalSSD += ssd - (sad * sad);
+ return totalSSD;
+}
+
+//Find the total SATD score of a block. Represents the block's overall complexity (bit cost) for intra encoding.
+static int satd_mb(x264_t *h, int i_pix_offset0, int i_pix_offset1)
+{
+ DECLARE_ALIGNED_16( static uint8_t zero[16] );
+ int totalSATD = h->pixf.satd[PIXEL_16x16](zero,0,&h->fenc->plane[0][i_pix_offset0], h->fenc->i_stride[0] << h->mb.b_interlaced);
+ totalSATD += h->pixf.satd[PIXEL_8x8](zero,0,&h->fenc->plane[1][i_pix_offset1], h->fenc->i_stride[1] << h->mb.b_interlaced);
+ totalSATD += h->pixf.satd[PIXEL_8x8](zero,0,&h->fenc->plane[2][i_pix_offset1], h->fenc->i_stride[2] << h->mb.b_interlaced);
+ return totalSATD;
+}

+void x264_autosense_aq( x264_t *h )
+{
+ double total = 0;
+ double n = 0;
+ /* FIXME: Easier way to iterate over MBs? Do we need to do the full cache_load? */
+ /* FIXME: Some of the SATDs might be already calculated elsewhere (ratecontrol?). Can we reuse them? */
+ /* FIXME: Store the data, then do the logs after, to avoid the cpu_restores every single cycle? */
+ /* FIXME: Is chroma SATD necessary? */
+ int i_mb_y = 0;
+ int i_mb_x = 0;
+ while( i_mb_y < h->sps->i_mb_height )
+ {
+ int i_stride = h->fdec->i_stride[0];
+ const int i_pix_offset0 = h->mb.b_interlaced
+ ? 16 * (i_mb_x + (i_mb_y&~1) * i_stride) + (i_mb_y&1) * i_stride
+ : 16 * (i_mb_x + i_mb_y * i_stride);
+ i_stride = h->fdec->i_stride[1];
+ const int i_pix_offset1 = h->mb.b_interlaced
+ ? 8 * (i_mb_x + (i_mb_y&~1) * i_stride) + (i_mb_y&1) * i_stride
+ : 8 * (i_mb_x + i_mb_y * i_stride);
+ int energy = ac_energy_mb_uncached(h,i_pix_offset0,i_pix_offset1);
+ h->rc->ac_energy[i_mb_x + i_mb_y * h->sps->i_mb_width] = energy;
+ /* Weight the energy value by the SATD value of the MB. This represents the fact that
+ the more complex blocks in a frame should be weighted more when calculating the optimal sensitivity.
+ This also helps diminish the negative effect of large numbers of simple blocks in a frame, such as in the case
+ of a letterboxed film. */
+ if(energy != 0)
+ {
+ int satd = satd_mb(h,i_pix_offset0,i_pix_offset1);
+ x264_cpu_restore(h->param.cpu);
+ total += log(energy) * satd;
+ n += satd;
+ }
+ i_mb_x++;
+ if(i_mb_x == h->sps->i_mb_width)
+ {
+ i_mb_x = 0;
+ i_mb_y++;
+ }
+ }
+ x264_cpu_restore(h->param.cpu);
+ /* Calculate and store the threshold. */
+ if(n == 0) h->aq_threshold = 100000;
+ else h->aq_threshold = expf(total / n);
+}
+
+
+//Finds the total AC energy of the block in all planes.
+static int ac_energy_mb_cached(x264_t *h)
+{
+ DECLARE_ALIGNED_16( static uint8_t zero[16] );
+ int sad = h->pixf.sad[PIXEL_16x16](zero,0,h->mb.pic.p_fenc[0],FENC_STRIDE) >> 4;
+ int ssd = h->pixf.ssd[PIXEL_16x16](zero,0,h->mb.pic.p_fenc[0],FENC_STRIDE);
+ int totalSSD = ssd - (sad * sad);
+ sad = h->pixf.sad[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[1],FENC_STRIDE) >> 3;
+ ssd = h->pixf.ssd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[1],FENC_STRIDE);
+ totalSSD += ssd - (sad * sad);
+ sad = h->pixf.sad[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[2],FENC_STRIDE) >> 3;
+ ssd = h->pixf.ssd[PIXEL_8x8](zero,0,h->mb.pic.p_fenc[2],FENC_STRIDE);
+ totalSSD += ssd - (sad * sad);
+ return totalSSD;
+}
+
+/*****************************************************************************
+* x264_adaptive_quant:
+ * adjust macroblock QP based on variance (AC energy) of the MB.
+ * high variance = higher QP
+ * low variance = lower QP
+ * This generally increases SSIM and lowers PSNR.
+ * To save bits in B-frames, adaptive lambda is used instead of adaptive quantization.
+*****************************************************************************/
+void x264_adaptive_quant( x264_t *h )
+{
+ int qp = h->mb.i_qp;
+ int energy;
+ x264_cpu_restore(h->param.cpu);
+ if(h->param.analyse.f_aq_sensitivity != 0)
+ {
+ energy = ac_energy_mb_cached(h);
+ }
+ else energy = h->rc->ac_energy[h->mb.i_mb_xy];
+ if(energy == 0)
+ {
+ h->mb.i_qp = h->mb.i_last_qp;
+ h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
+ }
+ else
+ {
+ x264_cpu_restore(h->param.cpu);
+ float result = energy;
+ /* Adjust the QP based on the AC energy of the macroblock. */
+ int qp_adj = -3.0 * h->param.analyse.f_aq_strength * log(result / h->aq_threshold);
+ if(h->param.analyse.f_aq_sensitivity == 0) qp_adj = x264_clip3(qp_adj,-5*h->param.analyse.f_aq_strength,5*h->param.analyse.f_aq_strength);
+ int new_qp = x264_clip3(qp - qp_adj,h->param.rc.i_qp_min,h->param.rc.i_qp_max);
+ /* If the QP of this MB is within 1 of the previous MB, code the same QP as the previous MB, to lower the bit
+ cost of the qp_delta. */
+ if(abs(new_qp - h->mb.i_last_qp) == 1) new_qp = h->mb.i_last_qp;
+ h->mb.i_qp = new_qp;
+ h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
+ }
+}
+
int x264_ratecontrol_new( x264_t *h )
{
x264_ratecontrol_t *rc;
@@ -727,6 +857,19 @@

if( h->sh.i_type != SLICE_TYPE_B )
rc->last_non_b_pict_type = h->sh.i_type;
+
+ /* Adaptive AQ sensitivity algorithm. */
+ if(h->param.analyse.f_aq_strength != 0)
+ {
+ if(h->param.analyse.f_aq_sensitivity != 0)
+ {
+ h->aq_threshold = powf(h->param.analyse.f_aq_sensitivity,4)/2;
+ }
+ else
+ {
+ x264_autosense_aq(h);
+ }
+ }
}

double predict_row_size( x264_t *h, int y, int qp )
Index: encoder/analyse.c
===================================================================
--- encoder/analyse.c (revision 721)
+++ encoder/analyse.c (working copy)
@@ -29,6 +29,7 @@
#endif

#include "common/common.h"
+#include "common/cpu.h"
#include "macroblock.h"
#include "me.h"
#include "ratecontrol.h"
@@ -2047,8 +2048,13 @@
int i_cost = COST_MAX;
int i;

- /* init analysis */
- x264_mb_analyse_init( h, &analysis, x264_ratecontrol_qp( h ) );
+ h->mb.i_qp = x264_ratecontrol_qp( h );
+
+ if( h->param.analyse.b_aq )
+ x264_adaptive_quant( h );
+
+ /* init analysis */
+ x264_mb_analyse_init( h, &analysis, h->mb.i_qp );

/*--------------------------- Do the analysis ---------------------------*/
if( h->sh.i_type == SLICE_TYPE_I )
Index: x264.c
===================================================================
--- x264.c (revision 721)
+++ x264.c (working copy)
@@ -243,6 +243,14 @@
" - 2: enabled on all mode decisions\n", defaults->analyse.i_trellis );
H0( " --no-fast-pskip Disables early SKIP detection on P-frames\n" );
H0( " --no-dct-decimate Disables coefficient thresholding on P-frames\n" );
+ H0( " --aq-strength <float> Amount to adjust QP/lambda per MB [%.1f]\n"
+ " 0.0: no AQ\n"
+ " 1.0: medium AQ\n", defaults->analyse.f_aq_strength );
+ H0( " --aq-sensitivity <float> \"Center\" of AQ curve. [%.1f]\n"
+ " 0: automatic sensitivity (avoids moving bits between frames)\n"
+ " 10: most QPs are raised\n"
+ " 20: good general-use sensitivity\n"
+ " 30: most QPs are lowered\n", defaults->analyse.f_aq_sensitivity );
H0( " --nr <integer> Noise reduction [%d]\n", defaults->analyse.i_noise_reduction );
H1( "\n" );
H1( " --deadzone-inter <int> Set the size of the inter luma quantization deadzone [%d]\n", defaults->analyse.i_luma_deadzone[0] );
@@ -406,6 +414,8 @@
{ "trellis", required_argument, NULL, 't' },
{ "no-fast-pskip", no_argument, NULL, 0 },
{ "no-dct-decimate", no_argument, NULL, 0 },
+ { "aq-strength", required_argument, NULL, 0 },
+ { "aq-sensitivity", required_argument, NULL, 0 },
{ "deadzone-inter", required_argument, NULL, '0' },
{ "deadzone-intra", required_argument, NULL, '0' },
{ "level", required_argument, NULL, 0 },
Index: common/common.c
===================================================================
--- common/common.c (revision 721)
+++ common/common.c (working copy)
@@ -123,6 +123,9 @@
param->analyse.i_chroma_qp_offset = 0;
param->analyse.b_fast_pskip = 1;
param->analyse.b_dct_decimate = 1;
+ param->analyse.b_aq = 0;
+ param->analyse.f_aq_strength = 0.0;
+ param->analyse.f_aq_sensitivity = 20;
param->analyse.i_luma_deadzone[0] = 21;
param->analyse.i_luma_deadzone[1] = 11;
param->analyse.b_psnr = 1;
@@ -455,6 +458,13 @@
p->analyse.b_fast_pskip = atobool(value);
OPT("dct-decimate")
p->analyse.b_dct_decimate = atobool(value);
+ OPT("aq-strength")
+ {
+ p->analyse.f_aq_strength = atof(value);
+ p->analyse.b_aq = (p->analyse.f_aq_strength > 0.0);
+ }
+ OPT("aq-sensitivity")
+ p->analyse.f_aq_sensitivity = atof(value);
OPT("deadzone-inter")
p->analyse.i_luma_deadzone[0] = atoi(value);
OPT("deadzone-intra")
@@ -815,9 +825,9 @@
c = a % b;
while(c)
{
- a = b;
- b = c;
- c = a % b;
+ a = b;
+ b = c;
+ c = a % b;
}
*n /= b;
*d /= b;
@@ -935,10 +945,15 @@
s += sprintf( s, " pb_ratio=%.2f", p->rc.f_pb_factor );
if( p->rc.psz_zones )
s += sprintf( s, " zones=%s", p->rc.psz_zones );
+ if( p->analyse.b_aq )
+ s += sprintf( s, " aq=1:%.1f:%.1f", p->analyse.f_aq_strength, p->analyse.f_aq_sensitivity );
else if( p->rc.i_zones )
s += sprintf( s, " zones" );
}

+ if( p->analyse.b_aq )
+ s += sprintf( s, " aq=1:%.1f:%.1f", p->analyse.f_aq_strength, p->analyse.f_aq_sensitivity );
+
return buf;
}

Index: common/common.h
===================================================================
--- common/common.h (revision 721)
+++ common/common.h (working copy)
@@ -331,8 +331,9 @@
x264_frame_t *fref1[16+3]; /* ref list 1 */
int b_ref_reorder[2];

+ /* AQ stuff */
+ float aq_threshold;

-
/* Current MB DCT coeffs */
struct
{
Index: x264.h
===================================================================
--- x264.h (revision 721)
+++ x264.h (working copy)
@@ -230,6 +230,9 @@
int i_trellis; /* trellis RD quantization */
int b_fast_pskip; /* early SKIP detection on P-frames */
int b_dct_decimate; /* transform coefficient thresholding on P-frames */
+ int b_aq; /* psy adaptive QP */
+ float f_aq_strength;
+ float f_aq_sensitivity;
int i_noise_reduction; /* adaptive pseudo-deadzone */

/* the deadzone size that will be used in luma quantization */

[Ranguvar]

IIRC, Sharktooth once said something about not using his high bitrate AVC CQM with AQ, I believe because of something like them both doing the same thing.

Please, what are the effects of using custom matrices with VAQ? Is it recommended to choose one, or the other?

[Dark Shikari]

IIRC, Sharktooth once said something about not using his high bitrate AVC CQM with AQ, I believe because of something like them both doing the same thing.

Please, what are the effects of using custom matrices with VAQ? Is it recommended to choose one, or the other?They work fine with VAQ; its just that they might be less necessary.

[*.mp4 guy]

They work fine with VAQ; its just that they might be less necessary.

They work fine with it, but often they are both trying to achieve the same thing, and have similar disadvantages. Often what happens when you use a cqm and aq at the same time on qp ~20+ encodes is that the disadvantages outway the advantages. Both cqm's and aq are technically bad for efficiency, and both introduce ringing, putting them together never works well on low bitrates, and is risky until you get to very very high bitrates.

[CruNcher]

They work fine with it, but often they are both trying to achieve the same thing, and have similar disadvantages. Often what happens when you use a cqm and aq at the same time on qp ~20+ encodes is that the disadvantages outway the advantages. Both cqm's and aq are technically bad for efficiency, and both introduce ringing, putting them together never works well on low bitrates, and is risky until you get to very very high bitrates.

My Experience is that for High Profile a CQM with VAQ can
help visualy reducing the ringing introduced (or amplified "source allready had visible ringing @ edges") by it @ edges, unfortunately that isn't possible for Main Profile.

@DeathTheSheep
Nice thx and yeah the older generation of AQs was more sane with the Edges the new generation after 0.46 amplifies ringing or creates ringing very often (especialy in live encoding scenarios), especialy it's bad for Main Profile where you have no chance to compensate it visualy with a CQM.

[burfadel]

A strength of 0.3 and sensitivity of 17 goes a long way to help with that I feel.

[CruNcher]

jep burfadel sure pumping up the bitrate helps but exactly that's what i try to avoid here :D

[burfadel]

On a test I just did, which concurs with an earlier test of mine:

AQ Disabled:
Size:1242kb, SSIM:0.9796759

Strenth Default, Sensitivity Default
Size:1169kb, SSIM:0.9793588

Strenth 0.5, Sensitivity 0 (Sensitivity 0 = automatic)
Size:1240kb, SSIM:0.9809342

I also did some other values, but the file sizes were larger. Going by that, the default sensitivity is not ideal and you actually end up with a lower SSIM! Having the sensitivity as automatic (set to 0), the file size is essentially identical to that of no AQ but the SSIM is raised.

Wasn't it the default of 0.46 to have automatic sensitivity? that may be why it produces better results in certain circumstances. Have a try, it would be interesting to see whether you come to the same conclusions! Automatic sensitivity was also slightly faster :)

[Dark Shikari]

I also did some other values, but the file sizes were larger. Going by that, the default sensitivity is not ideal and you actually end up with a lower SSIM! Having the sensitivity as automatic (set to 0), the file size is essentially identical to that of no AQ but the SSIM is raised.Comparing between different file sizes is totally meaningless.
Wasn't it the default of 0.46 to have automatic sensitivity?No

[burfadel]

Comparing between different file sizes is totally meaningless.


I do realise that :) probably wasn't the best thing to say! I was more getting to the point that the SSIM increases without a penalty in terms of file size when the sensitivity is set to 0.

I did the same test again, with a different clip. This came up with more interesting results...

With default AQ settings:
SSIM: 9811467
Size: 3870kb

With Sensitivity 0, default strength:
SSIM: 9813178
Size: 3755kb

In this case sensitivity set to automatic increased SSIM over that of default, and had a smaller file size!... And yes I know file size isn't a good comparison, but the fact that its better quality and smaller does has some significance. Subjectively the results are identical.

For those with bad results with AQ 0.48, maybe they should try sensitivity of 0 and see if their results improve?

I do apologise for going on about it, especially if it is only I who have had these results! I just can't help to mention that it consistently comes up with better results with sensitivity of 0 over that of the default :)

[Dark Shikari]

I just can't help to mention that it consistently comes up with better results with sensitivity of 0 over that of the default :)Better SSIM doesn't actually necessarily mean better quality.

[burfadel]

Ah ok! that explains it then :) I'm looking forward to VAQ 2.0, keep up the good work!

[CruNcher]

burfadel sensitivity 0 is the worst you can do visualy in many cases (to less bits are moved into the background in most of the scenes)

[Razorholt]

Hopefully v2.0 will end the "VAQ 0.46 vs. VAQ 0.48" war. BTW, I like 0.46 better when I'm dealing with very low birates (550 to 640). Speaking of which, what is the minimum birates required for VAQ to act efficiently? Should I disable it when dealing with very low birates?

I'm on a challenge here :D : http://forum.doom9.org/showthread.php?p=1119102#post1119102


Thanks,
-Dan

[canuckerfan]

was digging through this thread and I must say it's a very interesting read. was wondering if VAQ is enabled by default in Jarod's patched builds?

[ToS_Maverick]

all current VAQ builds have VAQ enabled by defaul, IIRC.

[Umamio]

I have quite a lot of football (soccer) caps (in huffy). In the past I've tried to get around the loss of detail in the grass by raising qcomp and lowering deblocking to -4, -5 but I think that was just giving the illusion of more detail in the form of fast changing blockiness, but I don't really know what I'm talking about, I just know my football encodes make me sad when I watch them back, whatever settings I try.

Anyway, I am very interested in the developments of this and would like to be of some use however I am able.

So if anyone is interested in specific tests on football material (standard def, PAL) in particular then send me the command line args you'd like me to use and I could encode a 1 minute clip into whatever different specs for comparison.

USE ME.

VAQ2.0 alpha testing Football Comparisons Here (http://forum.doom9.org/showthread.php?p=1120492#post1120492)

[professor_desty_nova]

I have quite a lot of football (soccer) caps (in huffy). In the past I've tried to get around the loss of detail in the grass by raising qcomp and lowering deblocking to -4, -5 but I think that was just giving the illusion of more detail in the form of fast changing blockiness, but I don't really know what I'm talking about, I just know my football encodes make me sad when I watch them back, whatever settings I try.

Anyway, I am very interested in the developments of this and would like to be of some use however I am able.

So if anyone is interested in specific tests on football material (standard def, PAL) in particular then send me the command line args you'd like me to use and I could encode a 1 minute clip into whatever different specs for comparison.

USE ME.

Maybe you should go to the VAQ 2.0 Alpha Testing (http://forum.doom9.org/showthread.php?t=136445) thread, since VAQ 1.0 is finished and already in 805+ builds. I'm sure Dark Shikari will like to ear your tests with the new VAQ 2.0 ;)

[gav1577]

Hi guys i have lost track of this thread lately and was wondering are the aq strength & sensitivity parameters still the same for v805 0.5 and 13 by default
or have things changed? the reason i asked i read somewhere default is now 1.0 for strength or is that just for 2.0 alpha i am a bit confused could someone please explain thanks

[DarkZell666]

"x264 --longhelp" says default strength is 1.0 and doesn't say anything about --aq-sensitivity (which was to be expected, DarkShikari said at some point that this parameter would be removed) ;)

[microchip8]

"x264 --longhelp" says default strength is 1.0 and doesn't say anything about --aq-sensitivity (which was to be expected, DarkShikari said at some point that this parameter would be removed) ;)

it is not removed, it has been replaced by --aq-mode

[gav1577]

So where as before i used --aq-strength 1.0 does this now mean to achieve the same affect i would have to use --aq-strength 1.5 with this new version v805 ? :)

[Dark Shikari]

So where as before i used --aq-strength 1.0 does this now mean to achieve the same affect i would have to use --aq-strength 1.5 with this new version v805 ? :)No, you'd use 2.0.

Every value has been doubled.

[gav1577]

No, you'd use 2.0.

Every value has been doubled.

Ok thanks Dark Shikari great work btw :D

[ToS_Maverick]

is it just me, or did the filesize decrease a bit with 805+ and standard AQ settings?

[Dark Shikari]

is it just me, or did the filesize decrease a bit with 805+ and standard AQ settings?Yes, pengvado changed AQ so as to (hopefully) better correlate with CRF filesizes. This doesn't change twopass.

[Chabb]

In 805+ builds VAQ1 was implemented officially,
but with previous builds I was able
to fine-tune one pass QP ratecontrol using --aq-sensitivity
(fractional changes)
and usage of fractional CRF values for that purpose was
(and still is) impossible, because of fixed value qcomp=1.00
Is there any workaround in this case (except 2-pass)?

[Dark Shikari]

In 805+ builds VAQ1 was implemented officially,
but with previous builds I was able
to fine-tune one pass QP ratecontrol using --aq-sensitivity
(fractional changes)
and usage of fractional CRF values for that purpose was
(and still is) impossible, because of fixed value qcomp=1.00
Is there any workaround in this case (except 2-pass)?I will fix that some time this week.

[Chabb]

Waiting impatiently :)

[karasu]

I recently discovered AQ but I'm a little confused with my own tests.
In short, I cant get visible better quality with AQ on.

My source Is Elephant Dream in sd resolution, and here's my settings :

With AQ disabled :

--pass 2 --bitrate 700 --stats ".stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 64 --threads auto --thread-input --progress --no-psnr --output "output" "input" --aq-strength 0.0
SSIM Mean Y:0.9785036
sample 1 (http://gloomydream.net/bazar/AQ/1-noaq.png)
sample 2 (http://gloomydream.net/bazar/AQ/2-noaq.png)

With default AQ settings :

--pass 2 --bitrate 700 --stats ".stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 64 --threads auto --thread-input --progress --no-psnr --output "output" "input"
SSIM Mean Y:0.9791959
sample 1 (http://gloomydream.net/bazar/AQ/1-aq.png)
sample 2 (http://gloomydream.net/bazar/AQ/2-aq.png)

Even if the SSIM is higher for the AQ, the image is sharper on the non-AQ (look at the antenna in the background of the first sample, or the textures and the elephant head on the second sample)
The only frames where I found slightly better details on the AQ version are on scenes with high motion, so it's hard to see the improvement while watching the movie. The sample above are from low-motion scenes, where it's easier to see the improvement.

I also try with different settings for AQ ( such as --aq-strength 1.0 --aq-sensitivity 20) but my results are worse than with the default settings.

Is there something I'm doing wrong? Is the bitrate too low to see the benefits of AQ?

(I'm using megui with x264 jarod's patched build 798)

[Razorholt]

I recently discovered AQ but I'm a little confused with my own tests.
In short, I cant get visible better quality with AQ on.

My source Is Elephant Dream in sd resolution, and here's my settings :

With AQ disabled :

--pass 2 --bitrate 700 --stats ".stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 64 --threads auto --thread-input --progress --no-psnr --output "output" "input" --aq-strength 0.0
SSIM Mean Y:0.9785036
sample 1 (http://gloomydream.net/bazar/AQ/1-noaq.png)
sample 2 (http://gloomydream.net/bazar/AQ/2-noaq.png)

With default AQ settings :
--pass 2 --bitrate 700 --stats ".stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -3,-2 --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 64 --threads auto --thread-input --progress --no-psnr --output "output" "input"
SSIM Mean Y:0.9791959
sample 1 (http://gloomydream.net/bazar/AQ/1-aq.png)
sample 2 (http://gloomydream.net/bazar/AQ/2-aq.png)

Even if the SSIM is higher for the AQ, the image is sharper on the non-AQ (look at the antenna in the background of the first sample, or the textures and the elephant head on the second sample)
The only frames where I found slightly better details on the AQ version are on scenes with high motion, so it's hard to see the improvement while watching the movie. The sample above are from low-motion scenes, where it's easier to see the improvement.

I also try with different settings for AQ ( such as --aq-strength 1.0 --aq-sensitivity 20) but my results are worse than with the default settings.

Is there something I'm doing wrong? Is the bitrate too low to see the benefits of AQ?

(I'm using megui with x264 jarod's patched build 798)

I second that statement. Try the 0.46 version and see what you get. I've already asked DS regarding the minimum birates requirement for VAQ to be efficient.

x264 with VAQ 0.46: x264.772.modified.exe (http://70.86.69.186/~matrix2/x264.772.modified.exe) (use --aq-strength 0.5 --aq-sensitivity 11)

Now, I must say that VAQ2 is quite good @ 700kb/s - You should try it.

- Dan

[Dark Shikari]

Is there something I'm doing wrong? Is the bitrate too low to see the benefits of AQ?AQ is a great way for x264 to teach people about the disadvantages of using stupid deblocking settings, like you're using ;)I second that statement. Try the 0.46 version and see what you get.Can people seriously stop spreading bugged code with the bizarre claim that because it randomly makes mistakes, its "better," even though nobody has ever been able to prove it in the well over two months since the bug was fixed? This is ridiculous, and this sort of conduct is exactly why I dislike releasing patches.

[DarkZell666]

@karasu : One thing you did do wrong was to use different deblocking settings on each sample in the first place. The No AQ one is -2,-1, and the AQ one is -3,-2. That voids your comparison completely :)

[Razorholt]

Can people seriously stop spreading bugged code with the bizarre claim that because it randomly makes mistakes, its "better," even though nobody has ever been able to prove it in the well over two months since the bug was fixed? This is ridiculous, and this sort of conduct is exactly why I dislike releasing patches.

I don't want to argue with you on that subject because someone else is already doing it...

The point is: you said that the best way to judge the settings is by using your eyes (over SSIM). Well, I think it's no secret that people have different eyes and not the same definition of quality.

My experience with your VAQ releases is that at very low bitrates 0.46 was giving ME better results and MY EYES found it quite satisfactory (yes, I may have shitty eyes...). I know you find ridiculous to encode at low birates but sometimes that's the only option. Have you ever tried encoding at 600 or 700kb/s ?

Anyway, I'm quite happy with VAQ2, even at low birates :)

Keep up the great work, DS.

- Dan

[Dark Shikari]

I don't want to argue with you on that subject because someone else is already doing it...

The point is: you said that the best way to judge the settings is by using your eyes (over SSIM). Well, I think it's no secret that people have different eyes and not the same definition of quality.

My experience with your VAQ releases is that at very low bitrates 0.46 was giving ME better results and MY EYES found it quite satisfactory (yes, I may have shitty eyes...). I know you find ridiculous to encode at low birates but sometimes that's the only option. Have you ever tried encoding at 600 or 700kb/s ?Yes, I often test my AQ at bitrates as low as 200kbps or less. It generally has no effect on its usefulness.

Example: "blah blah blah AQ is useless on low-bitrate anime blah blah blah"

http://i31.tinypic.com/2ilg3ud.gif

[Southstorm]

I'm sold!

[Razorholt]

I never mentioned 200kbps and I never encode anime stuff.

[Dark Shikari]

I never mentioned 200kbps and I never encode anime stuff.I was referring to the general commentary that AQ wasn't good at low bitrate anime (including such comments made by myself).

[karasu]

I'm sorry for the confusion, the different settings in deblocking come form an error of copy/pasting. (I'm trying to vary each settings at once to see what's happens, I think it's the best way to learn about x264)

My mistake was to copy the wrong line on my "huge settings log file"...

Of course the deblock settings was the same for my two tests.

Perhaps a deblocking of -2-1 is stupid, but to my eyes it look more like the original than deblock 0 0 .

However, I'm still using an outdated version of AQ, I'll retry with one of the builds of this thread, or maybe VAQ2

[Dark Shikari]

Perhaps a deblocking of -2-1 is stupid, but to my eyes it look more like the original than deblock 0 0 .AQ in particular suffers from overly low deblocking strengths because they result in deblocking not being applied at all to background blocks.

[karasu]

AQ in particular suffers from overly low deblocking strengths because they result in deblocking not being applied at all to background blocks.

Ok, thank you for these explications.
I have made some tests with default deblocking. The SSIM boost is important.
High motions scenes are greatly improved,
high motion AQ (http://gloomydream.net/bazar/AQ/hm-aq.png)
high motion no AQ (http://gloomydream.net/bazar/AQ/hm-noaq.png)

but the the stills or very slow motions scenes are less sharper than without AQ.
low motion AQ (http://gloomydream.net/bazar/AQ/lm-aq.png)
low motion no AQ (http://gloomydream.net/bazar/AQ/lm-noaq.png)

Perhaps I'm completely wrong (I'm here to learn) but I think that we are more sensible to quality in slow motion scenes. So for the human eye, the gain in perceptual quality isn't as good as the SSIM gain.
Do you think that a way to adapt the AQ strength in function of the speed of the motion can be useful in that case?

[Inventive Software]

For the record, if you're encoding SD Elephants Dream at 500 Kbits or around that mark, the MINIMUM deblocking you can get away with is -1:-1 (not tried it, but might help in the telephone wires scene), highly recommended is the default (0:0). I did a comparison between VC-1 and H.264 a while back, I did one encode with all the bells and whistles, long before VAQ was around, and Haali's AQ was disabled. It looked very good! :)

[Lele-brz]

I had the same impression of Karasu.
Sometimes AQ seems to decrease the perceived quality on some videos.
After some test I would use it on high motion (especially soccer games where the grass with AQ looks much better), but not always.
So, I'm wondering why it's turned on by default.

[MfA]

Do you think that a way to adapt the AQ strength in function of the speed of the motion can be useful in that case?
Maybe someone can step up to do something like this after Aki Jäntti's SoC project, once his temporal search and per MB record keeping code is in there (hopefully) you could add future motion on top of future coding relevance (which he is working on). Just naively weighting based on current frame motion is a bad idea, a large foreground object which moves predictably can be in perfect focus on your retina ... only for extreme velocities would this kind of naive weighting make sense, you need more data and better metrics.

[Ranguvar]

By far the worst "problem", in my opinion, with VAQ, is the mosquito noise that inevitably surrounds hardcoded subtitles. For example, in The Lord of the Rings, when there are hardcoded subs for the Elvish parts. I can make a 2CD backup that looks pretty perfect in my opinion, except for said noise.

Are there any plans to alleviate this, or can you recommend settings that will mitigate that without impacting the rest of the video much?

[Dark Shikari]

By far the worst "problem", in my opinion, with VAQ, is the mosquito noise that inevitably surrounds hardcoded subtitles. For example, in The Lord of the Rings, when there are hardcoded subs for the Elvish parts. I can make a 2CD backup that looks pretty perfect in my opinion, except for said noise.

Are there any plans to alleviate this, or can you recommend settings that will mitigate that without impacting the rest of the video much?On the one hand, without VAQ, hardcoded subtitles end up using up an absurd proportion of the number of bits in a video because of their sharp edges. VAQ's algorithm makes the assumption--a valid one, I think--that you probably don't want the subtitles wasting so many bits.

On the other hand, you could just try weakening AQ a bit; IMO subtitles can go quite a bit up quantizer-wise before there's a serious visual problem.

[Ranguvar]

Alright, thanks. I guess I'm just screwed because my eyes seem to hone in on subtitles... too much anime xD

Didn't know they took up so many bits, however, so I'm at least somewhat put at ease :p

[Ranguvar]

Hey, me again with little annoying questions :)

This thread says:
Version history:

0.48: AQ strength 0.5, sensitivity 13 made the defaults. Updated to r736. Qcomp is now scaled based on AQ strength automatically.

However, --longhelp on r899 reports that a strength of 1.0 is the default.

Which is it, please? I poked around the source a little, but I am still an über-noob at code...

[Dark Shikari]

Hey, me again with little annoying questions :)

This thread says:


However, --longhelp on r899 reports that a strength of 1.0 is the default.

Which is it, please? I poked around the source a little, but I am still an über-noob at code...The strengths were all doubled so that 1.0 would be the default, for interface reasons.

Old 0.5 == New 1.0

[egrimisu]

What is the recomended variance aq setting to achive highest quality in megui: strengh 1.0 or 2.0 ??? i' using 967-1 skytrife release

[Atak_Snajpera]

default strengh 1.0 should be optimal for most cases

[egrimisu]

Thanks, i used 2.0 for an encode and the results where NOT that greate, retrying using 1.0. THANKS again

default strengh 1.0 should be optimal for most cases

[Zwitterion]

Now that we have PsyRD, I find the AQ default value of 1.0 a little too high.
When AQ was tuned, trading off some edge definition for the vast improvement of flat areas was acceptable. But now PsyRD improves those flat areas a lot, so AQ doesn't need to be as high.

I recommend 0.5 as a sensible default value when using PsyRD.

[Dark Shikari]

Now that we have PsyRD, I find the AQ default value of 1.0 a little too high.
When AQ was tuned, trading off some edge definition for the vast improvement of flat areas was acceptable. But now PsyRD improves those flat areas a lot, so AQ doesn't need to be as high.

I recommend 0.5 as a sensible default value when using PsyRD.Quantizer-field smoothing is coming soon, and should resolve most of these issues.

[CruNcher]

1.0 can be very agressive in some low bitrate situations but hopefully quantizer field smoothing will really bring the visual fix for that, also trying to compensate some of these edge problems currently by useing higher chroma quants seems to sometimes work wonders currently, though it will use a little more bitrate. I guess what happens is it compensates AQ a little so it acts like a fine tuneing mechanism takeing efficiency of AQ and on the other side i guess as the chroma information most of the times is everywhere near edges it helps here by allocateing more bits to these so they dont (the chroma near the edges of a object) fuzz out to fast in motion :) (but all this only becomes a real visual problem at very low bitrates most of the times, and @ these you dont should care so much anymore about detail preservation but motion stability).

[salehin]

DarkShikari: is r736 the latest x264 build with your FGO? If not, can you please suggest any available build for a very grainy/noisy source. I see 998 (by skystrife) with MeGUI- iirc, that doesn't contain your FGO. The source is a lovely DVD that I own, mastered from a handheld camera contents, at least 20 yrs old (the programme)!

Cheers :)

[LoRd_MuldeR]

FGO has been superseeded by Psy-RDO and Psy-Trellis ;)

(I read that some people still claim that FGO is better for very grainy sources, but Psy RDO/Trellis is a more general approach and helpful for almost any source)

[burfadel]

Just remember Psy-trellis (or is it psy-rdo, can't check at the moment)! is disabled by default. The reason for the disable is some people claim in 'clean' (as in picture noise) sources, particularly some animation, it looks better without. At some stage it will be enabled by default, maybe after some slight tweaking. In terms of the source that you have described enabling it would be a good idea. It can be enabled by the command line option:
--psy-rd 1.0:1.0

[Sagekilla]

there's psy-rdo and psy-trellis: the first param in --psy-rd is the psy-rdo proper, and the second activates a psy-trellis mode.

[LoRd_MuldeR]

Just remember Psy-trellis (or is it psy-rdo, can't check at the moment)! is disabled by default. The reason for the disable is some people claim in 'clean' (as in picture noise) sources, particularly some animation, it looks better without. At some stage it will be enabled by default, maybe after some slight tweaking. In terms of the source that you have described enabling it would be a good idea. It can be enabled by the command line option:
--psy-rd 1.0:1.0

The current default is "--psy-rd 1.0:0.0", which means Psy-RDO is enabled and Psy-Trellis is disabled.

Some comparision can be found here:
* http://forum.doom9.org/showthread.php?t=141188
* http://forum.doom9.org/showthread.php?t=141249