New Adaptive Quantization

[Dark Shikari]

Original post edited since I now have a new algorithm.

Try out my new AQ method (http://tjhsst.edu/~jgarrett/x264aq.exe). Warning: It is very powerful and will butcher your PSNR, though it has little effect on SSIM. Don't judge by the metrics, judge by appearance. Also, note that it is not meant at all for 1-pass bitrate (CRF/QP is fine)--it butchers the quantizers in a way that doesn't keep the framewide quantizer even close to constant, so 1pass bitrate modes turn out horrible.

This is not at all final, so if it sucks, say so.

Also, note that only --aq-strength is active in this algorithm. Try 1.0 first, and lower it if its too powerful. Try to use it on videos that you know benefit a lot from AQ, or have serious blocking issues in the background.

[akupenguin]

This struck me as weird at first, as AQ is supposed to lower SSIM
The detail part of SSIM is weighted inversely by the stddev of each block. Thus assigning lower QP to blocks with lower stddev should result in higher SSIM per PSNR, and it doesn't especially surprise me that it's also higher SSIM per bit.

[Dark Shikari]

The detail part of SSIM is weighted inversely by the stddev of each block. Thus assigning lower QP to blocks with lower stddev should result in higher SSIM per PSNR, and it doesn't especially surprise me that it's also higher SSIM per bit.Ah, that makes sense then. I optimized for SSIM without even realizing it!

Anyways, its time to test how well this eliminates blocks. :)

[Sagittaire]

Well if you want implement complete AQ you can use multiple masking: luma, dark, temporal, spacial, contrast ... etc

IMO the most efficient HVS AQ masking are:
- Spacial complexity: use higher quantizer for complex texture
- Dark masking: use lower quantizer for dark zone

Peronnaly I use Dark noise filtering and Spacial complexity AQ. Dark noise filtering imply less complexity in dark zone and lower quantizer in dark zone if you use Spacial Complexity AQ.

[Terranigma]

Peronnaly I use Dark noise filtering and Spacial complexity AQ. Dark noise filtering imply less complexity in dark zone and lower quantizer in dark zone if you use Spacial Complexity AQ.
I would love to see this implemented a.s.a.p. It's very annoying when 90% of your encoded video is fine, and the rest for dark areas is hideous due to blocking. :scared:

[Dark Shikari]

I would love to see this implemented a.s.a.p. It's very annoying when 90% of your encoded video is fine, and the rest for dark areas is hideous due to blocking. :scared:If my method works properly it should do this, because if the block is very dark it'll have a very low standard deviation.

[Sagittaire]

I would love to see this implemented a.s.a.p. It's very annoying when 90% of your encoded video is fine, and the rest for dark areas is hideous due to blocking. :scared:

Yes with TFT/Plasma screen blocking in dark area is really annoying. But like a say Dark Noise Filtering + spacial complexity AQ solve that very well.

[Ranguvar]

Sounds awesome!

Can't wait to try this out on anime! Any idea whether this will have a greater advantage on that?

[Dark Shikari]

Sounds awesome!

Can't wait to try this out on anime! Any idea whether this will have a greater advantage on that?There is probably a lot of tuning I'll have to do; for anime one might also need a much higher strength, as anime tends to be overall much flatter than live-action footage.

[Dark Shikari]

Interesting--I tested it on a clip from the Chronicles of Riddick and it lowered quality because the average QP was raised by 2.5-3. I'm guessing that I will have to find a way for the encoder to figure out what the optimal "strength" for this option is, since it is so film, and perhaps even scene or frame dependent.

[Ranguvar]

Interesting--I tested it on a clip from the Chronicles of Riddick and it lowered quality because the average QP was raised by 2.5-3. I'm guessing that I will have to find a way for the encoder to figure out what the optimal "strength" for this option is, since it is so film, and perhaps even scene or frame dependent.

Interesting...

Sounds sweet; I will keep an eye on this one ;-)

[ToS_Maverick]

Dark Shikari you simply ROCK!

gimme an .exe and i'll test it with my blackpearl sample, where i know almost every frame ;)

can't wait to get my hands on this...

[Dark Shikari]

Dark Shikari you simply ROCK!

gimme an .exe and i'll test it with my blackpearl sample, where i know almost every frame ;)

can't wait to get my hands on this...Eh, the original algorithm turned out to be less than impressive at its original goal, which was to get rid of blocks.

So I've been modifying it somewhat... when I think its ready, I'll post an EXE.

[Dark Shikari]

Try out my new AQ method (http://tjhsst.edu/~jgarrett/x264aq.exe). Warning: It is very powerful and will butcher your PSNR, though it has little effect on SSIM. Don't judge by the metrics, judge by appearance. Also, note that it is not meant at all for 1-pass bitrate (CRF/QP is fine)--it butchers the quantizers in a way that doesn't keep the framewide quantizer even close to constant, so 1pass bitrate modes turn out horrible.

This is not at all final, so if it sucks, say so.

Also, note that only --aq-strength is active in this algorithm. Try 1.0 first, and lower it if its too powerful. Try to use it on videos that you know benefit a lot from AQ, or have serious blocking issues in the background.

[MarcioAB]

Moving in the right direction.
Except keeping the bitrate as closer as possible, I'm not relying on metrics, just the old and good eyes fixed over a bright 1680x1050 LCD (full screen from 720x480).

- I see improvement in blockness (on dark & etc areas): less, smaller and "less dancing" blocks.
- Results are better with the default matrix than with eqm_avc_hr matrix.
- not compiled to MP4 ? (but ok)
- much slower than current x264 ! (but again, ok).
- nice to see very low mean Q's on dark areas: 14's when Q(P)=24 (but still a bit blocky on these cases)

Good job. Looking forward.

[Dark Shikari]

Moving in the right direction.
Except keeping the bitrate as closer as possible, I'm not relying on metrics, just the old and good eyes fixed over a bright 1680x1050 LCD (full screen from 720x480).

- I see improvement in blockness (on dark & etc areas): less, smaller and "less dancing" blocks.
- Results are better with the default matrix than with eqm_avc_hr matrix.
- not compiled to MP4 ? (but ok)
- much slower than current x264 ! (but again, ok).
- nice to see very low mean Q's on dark areas: 14's when Q(P)=24 (but still a bit blocky on these cases)

Good job. Looking forward.The speed is just because its completely unoptimized: I intentionally used no assembly and such because I want to get it working before I try to speed it up. Optimized it should be almost negligable speed decrease.

Is QP 14 really blocky? I intentionally capped the drop of QP by QP/2 (so if frame QP is 24, it won't drop below 12). If one needs even lower QPs to get rid of blocking, I can slightly reduce that cap.

[woah!]

heres a few screens to look at, i used r676 as a compare. also yours doesnt thread cpu's so 4x slower for me:

partitions:

x264aq
http://www.cif-forums.com/png/x264aq.PART.png

676.x264
http://www.cif-forums.com/png/676.x264.PART.png

Quantizers: white – 0 / black – 31

x264aq
http://www.cif-forums.com/png/x264aq.QT.png

676.x264
http://www.cif-forums.com/png/676.x264.QT.png

Macroblocks: macroblock sizes (in bits) in the video window: black– 0; white – max for the current frame

x264aq
http://www.cif-forums.com/png/x264aq.MB.png

676.x264
http://www.cif-forums.com/png/676.x264.MB.png

[Dark Shikari]

Notice what I did there with the quantizers--the entire frame was relatively flat, and so I gave the entire frame more bits. This seems very effective in getting rid of blocks--one of the main problems I find with x264 is that it will decide that a scene doesn't need a lot of bits, skip almost all the blocks, and you'll get a high-PSNR but quite blocky encode in that scene. The bits saved in that scene go elsewhere, but are not as useful as they would be if they stayed in that scene.

[woah!]

yes the quant window does show a nice result, and to be honest i believe yours looks better as well. i used 0.4 AQ in both encodes, nothing was changed.

heres the 2 encodes complete :

encodes (http://www.cif-forums.com/png/N.rar)

[MarcioAB]

Is QP 14 really blocky? I intentionally capped the drop of QP by QP/2 (so if frame QP is 24, it won't drop below 12). If one needs even lower QPs to get rid of blocking, I can slightly reduce that cap.

I encoded with --qp 24 --bframe 16 and 1.0xAQ (24 is an example, I encoded with --qp variations around that to keep the same bitrate to compare), that means without AQ the P-frame will receive Q=24 and the B-frame will receive Q=26. Because I used --bframe 16, I told the Rate Control to use as much B-frames it consider necessary, what in my encode case is around 5 or 6.
Obs: Near on single very-trouble area I see sequences of 10 P-frames (without any B-frame), but in general I can say 1P+5B is the standard for this encode.

Anyway, as B-frame prevails ( Q=26 ) and your target for "trouble" macroblocks will be Q/2=13 and I notice ffdshow reporting entire frame with mean-Q=14, I guess everything is working as designed.

So, now finally answering your question: Yes, I still see blocky on "trouble" areas even with ffdshow indicating frame mean-Q as 14. And than I look back on the source MPEG2 (0.65 bpp) and it's not that bad over there. So, x264 is really not good on dark/flat/slow areas.

I fully agree with you: x264 rocks on light/complex areas but is not so good on dark/flat/slow areas.

Thank you.

[Sagittaire]

I can see the patch ... ?

IMO it's better to use filter for Dark zone and after use spacial AQ. IMO Use directly dark AQ masking for dark zone with heavy grain/noise is useless.

[MarcioAB]

*deleted* I guess you need the source code ...

[ToS_Maverick]

it's sceenshot time!

from left to right:
Source, CRF 20 Old AQ 0.6/10, CRF 20 New AQ 0.7, CRF 21 New AQ 0.9, CRF 22 New AQ 1.0
all about the same size

http://img217.imageshack.us/img217/6881/385sourcesmallhc9.th.png (http://img217.imageshack.us/my.php?image=385sourcesmallhc9.png) http://img464.imageshack.us/img464/3484/385oldaqsmallfb7.th.png (http://img464.imageshack.us/my.php?image=385oldaqsmallfb7.png) http://img521.imageshack.us/img521/4104/385newaq07smallyz1.th.png (http://img521.imageshack.us/my.php?image=385newaq07smallyz1.png) http://img338.imageshack.us/img338/4638/385newaq09smallfr4.th.png (http://img338.imageshack.us/my.php?image=385newaq09smallfr4.png) http://img337.imageshack.us/img337/2926/385newaq10smallty1.th.png (http://img337.imageshack.us/my.php?image=385newaq10smallty1.png)

http://img521.imageshack.us/img521/2301/4025sourcesmallcy2.th.png (http://img521.imageshack.us/my.php?image=4025sourcesmallcy2.png) http://img413.imageshack.us/img413/8276/4025oldaqsmallfi7.th.png (http://img413.imageshack.us/my.php?image=4025oldaqsmallfi7.png) http://img160.imageshack.us/img160/5958/4025newaq07smallcm3.th.png (http://img160.imageshack.us/my.php?image=4025newaq07smallcm3.png) http://img227.imageshack.us/img227/57/4025newaq09smallth8.th.png (http://img227.imageshack.us/my.php?image=4025newaq09smallth8.png) http://img221.imageshack.us/img221/7996/4025newaq10smallzh3.th.png (http://img221.imageshack.us/my.php?image=4025newaq10smallzh3.png)

Summary:
- the new AQ is always better than the old one, never worse.
- the new AQ seems to have a problem at CRF21 0.9 strength, take a look at the screen ;)
- these are 2 of the more impressive examples, the old AQ isn't that bad, in some cases, the new one is much better.

commandlines:

Old AQ:
--crf 20.0 --level 3 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 1835 --vbv-maxrate 10000 --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "" "F:\Video\Black.Pearl.Sample.avs" --aq-strength 0.6 --aq-sensitivity 10

New AQ:
x264aq.exe --crf 20.0 --level 3 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 1835 --vbv-maxrate 10000 --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "" "F:\Video\Black.Pearl.Sample.avs" --aq-strength 0.7


i don't post the logfiles, since this isn't about metrics, it's about the look :cool:

[Dark Shikari]

Here is the code. Note the complete lack of assembly, etc, etc which I'll fix.

void x264_adaptive_quant( x264_t *h, x264_mb_analysis_t *a )
{
DECLARE_ALIGNED( static uint8_t, zero[FDEC_STRIDE*8], 16 );
DECLARE_ALIGNED( int16_t, dct[8][8], 16 );
float fc;
int total = 0;
int qp = h->mb.i_qp, qp_adj;
int i;
int variance = h->pixf.count_16x16(h->mb.pic.p_fenc[0], FENC_STRIDE);
x264_emms();
float standardDeviation = sqrt(variance/256.0);
float relativeSD = standardDeviation /= (h->param.standardDeviation + 0.01);
if(relativeSD > 1)
{
qp_adj = (qp/2.0) * h->param.analyse.f_aq_strength * (relativeSD - 1.0);
qp_adj = X264_MIN(qp_adj, qp/2);
//printf("Positive Adjust: %d out of total QP %d\n",qp_adj,qp);
}
else
{
qp_adj = -(qp/4.0) * h->param.analyse.f_aq_strength * (0.1/(0.01+relativeSD));
qp_adj = X264_MAX(qp_adj, -qp/2);
//printf("Negative Adjust: %d out of total QP %d\n",qp_adj,qp);
}
if(h->param.standardDeviation < 30) qp_adj -= h->param.analyse.f_aq_strength*(30.0 - h->param.standardDeviation)*(30.0 - h->param.standardDeviation)/60.0;
qp_adj = X264_MAX(qp_adj, -qp/2);
h->mb.i_qp = a->i_qp = X264_MIN(qp + qp_adj,51);
h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
}

h->param.standardDeviation is not the standard deviation of the whole frame; its the average of the standard deviations of 16x16 blocks of that frame.

Also, I think the effectiveness of this patch is proof that PSNR is completely utterly useless ;)

[Sagittaire]

Here is the code. Note the complete lack of assembly, etc, etc which I'll fix.

void x264_adaptive_quant( x264_t *h, x264_mb_analysis_t *a )
{
DECLARE_ALIGNED( static uint8_t, zero[FDEC_STRIDE*8], 16 );
DECLARE_ALIGNED( int16_t, dct[8][8], 16 );
float fc;
int total = 0;
int qp = h->mb.i_qp, qp_adj;
int i;
int variance = h->pixf.count_16x16(h->mb.pic.p_fenc[0], FENC_STRIDE);
x264_emms();
float standardDeviation = sqrt(variance/256.0);
float relativeSD = standardDeviation /= (h->param.standardDeviation + 0.01);
if(relativeSD > 1)
{
qp_adj = (qp/2.0) * h->param.analyse.f_aq_strength * (relativeSD - 1.0);
qp_adj = X264_MIN(qp_adj, qp/2);
//printf("Positive Adjust: %d out of total QP %d\n",qp_adj,qp);
}
else
{
qp_adj = -(qp/4.0) * h->param.analyse.f_aq_strength * (0.1/(0.01+relativeSD));
qp_adj = X264_MAX(qp_adj, -qp/2);
//printf("Negative Adjust: %d out of total QP %d\n",qp_adj,qp);
}
if(h->param.standardDeviation < 30) qp_adj -= h->param.analyse.f_aq_strength*(30.0 - h->param.standardDeviation)*(30.0 - h->param.standardDeviation)/60.0;
qp_adj = X264_MAX(qp_adj, -qp/2);
h->mb.i_qp = a->i_qp = X264_MIN(qp + qp_adj,51);
h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
}

h->param.standardDeviation is not the standard deviation of the whole frame; its the average of the standard deviations of 16x16 blocks of that frame.

Also, I think the effectiveness of this patch is proof that PSNR is completely utterly useless ;)

I think that qp/2 is a too high overquantized limit. I will try to optimize that with best value for my eyes ... ;-)

There are theorical max value for standarddeviation?

[Dark Shikari]

I think that qp/2 is a too high overquantized limit. I will try to optimize that with best value for my eyes ... ;-)

There are theorical max value for standarddeviation?
Max value for standard deviation is extremely high, but will basically never be reached, or even close.

Another problem is that standard deviation varies based on the source, which is why I had it calculate framewide standard deviation to use as a reference.

Also note you need this code in encoder.c to calculate the standard deviation:

int x,y;
x264_emms();
double totalStandardDeviation = 0;
int num = 0;
for(x=0;x < h->param.i_width-16;x+=16)
for(y=0;y < h->param.i_height-16;y+=16)
{
num++;
int variance = h->pixf.count_16x16(h->fenc->plane[0]+x+y*h->fenc->i_stride[0], FENC_STRIDE);
totalStandardDeviation += sqrt(variance/256.0);
}
h->param.standardDeviation = totalStandardDeviation / num;

and of course you need to add param.standardDeviation in the struct in x264.h... I should really post a patch but I don't think its really ready for that yet.

[Sharktooth]

i think QP/2 is not good to.
i would use something like QP - f_aq_strength...

also, why
qp_adj = (qp/2.0) * h->param.analyse.f_aq_strength * (relativeSD - 1.0);
qp_adj = X264_MIN(qp_adj, qp/2);
and
qp_adj = -(qp/4.0) * h->param.analyse.f_aq_strength * (0.1/(0.01+relativeSD));
qp_adj = X264_MAX(qp_adj, -qp/2);
?!?!?

[Gabriel_Bouvigne]

I'd suggest you to do adjustment using q scale, and not qp scale, as it's a bit more linear (there are qp2qscale and qscale2qp functions).

[Dark Shikari]

i think QP/2 is not good to.
i would use something like QP - f_aq_strength...

also, why
qp_adj = (qp/2.0) * h->param.analyse.f_aq_strength * (relativeSD - 1.0);
qp_adj = X264_MIN(qp_adj, qp/2);
and
qp_adj = -(qp/4.0) * h->param.analyse.f_aq_strength * (0.1/(0.01+relativeSD));
qp_adj = X264_MAX(qp_adj, -qp/2);
?!?!?Because it worked :D

As I said, its very experimental and I'll definitely try using a better scaling method if I can find one that works.

[Sharktooth]

ok...

[MarcioAB]

My 2 cents ...

I think Q=10 is kind of yield number in x264. So instead push Q to QP/2, it could be pushed to 10 (or user defined, maybe function of something else).

Example: encode with --qp 18: A "trouble" macroblock may be given Q=9, but 9 is a waste of resource (IMO).

On the other side encode with --qp28: A really "trouble" macroblock will get Q=10, assuring max workout on such "trouble" area.

[Sharktooth]

that's why i proposed qp - f_aq_strength (assuming f_aq_streng is defined as int).
let's assume qp is 28 and aq_strength is set to 10 the result would be 18...
rising the strength will lower the qp...

EDIT: uhm... just saw the new aq_strength is float...
however is quite easy. something like this (for positive adjust... similar for other cases...):
int aq_bias;
....
....
if(relativeSD > 1)
{
aq_bias = qp - (param.analyse.f_aq_strength * 10);
qp_adj = aq_bias * h->param.analyse.f_aq_strength * (relativeSD - 1.0);
qp_adj = X264_MIN(qp_adj, aq_bias);
}
...
...

[Dark Shikari]

I came up with a new formula:

qfactor = exp(1.0*(relativeSD - 1.0)*(relativeSD - 1.0)*sign(relativeSD - 1.0));
qfactor *= h->param.standardDeviation/30.0;
float qp_adj = (qscale2qp(qp2qscale(qp)*qfactor)) - qp;
qp_adj *= h->param.analyse.f_aq_strength;
int newQP = (int)qp_adj + qp;
h->mb.i_qp = a->i_qp = X264_MAX(X264_MIN(X264_MAX(X264_MIN(newQP,3*qp/2),qp/2),51),10);

It uses qscale, not sure if its better or not but it seems to do a pretty good job overall.

[Sharktooth]

uhm.... isnt it a bit too 'complex' just for calculating the final qp? and the qp/2 is still there... :P

[Dark Shikari]

uhm.... isnt it a bit too 'complex' just for calculating the final qp?The formula has to take into account many things:

a) If the block is "flatter" than the rest of the scene, it needs to have a lower QP than the rest of the scene. This can't really be linear.
b) If the block is very complex, it needs to have a higher QP, to add more bits back into the rest of the scene. This can't really be linear either.
c) If the entire scene is flat, the entire scene's QP needs to be lower.

[Sharktooth]

edited the previous posts...

[Dark Shikari]

edited the previous posts...
Your formula multiplies in the strength twice, resulting in a nonlinear scaling of AQ strength.

[Sharktooth]

it was just an idea:p
however the qp/2 is empirical...

[Sagittaire]

The formula has to take into account many things:

a) If the block is "flatter" than the rest of the scene, it needs to have a lower QP than the rest of the scene. This can't really be linear.
b) If the block is very complex, it needs to have a higher QP, to add more bits back into the rest of the scene. This can't really be linear either.
c) If the entire scene is flat, the entire scene's QP needs to be lower.

Well I think that c is useless because if the entire frame is flat then RC will always use lower relative quant (Rate Control equation with texture complexity).

[Dark Shikari]

Well I think that c is useless because if the entire frame is flat then RC will always use lower relative quant (Rate Control equation with texture complexity).This doesn't seem to be the case at all in my experience, or at least not enough. A lot of blocking problems, as I've said before, are caused by x264 deciding that a flat scene needs very few bits, and as such hardly allocating any bits to it.

Another issue is that my formula sets the quantizer offset based on the SD relative to the frame's SD. Ideally, one would want to look at the entire film's SD, which would avoid having to deal with the problem of c). a) and b) only work when there are dark blocks in an otherwise normal frame--a frame entirely of dark/flat blocks needs c).

[Dark Shikari]

Wow, that was retarded, I was multiplying instead of dividing.

void x264_adaptive_quant( x264_t *h, x264_mb_analysis_t *a )
{
int qp = h->mb.i_qp;
float qfactor;
int variance = h->pixf.count_16x16(h->mb.pic.p_fenc[0], FENC_STRIDE);
x264_emms();
float standardDeviation = sqrt(variance/256.0);
float relativeSD = standardDeviation /= (h->param.standardDeviation + 0.01);
qfactor = exp(1.0*(relativeSD - 1.0)*(relativeSD - 1.0)*sign(relativeSD - 1.0));
qfactor /= sqrt(h->param.standardDeviation/30.0);
float qp_adj = (qscale2qp(qp2qscale(qp)*qfactor)) - qp;
qp_adj *= h->param.analyse.f_aq_strength;
int newQP = (int)qp_adj + qp;
h->mb.i_qp = a->i_qp = X264_MAX(X264_MIN(X264_MAX(X264_MIN(newQP,3*qp/2),qp/2),51),10);
h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
}

[Dark Shikari]

Results with some tweaks:

2000kbps, without AQ:

http://i5.tinypic.com/6hfy59s.png

2000kbps, with new AQ:

http://i12.tinypic.com/4l65bv7.png

The old AQ algorithm has similar appearance, but because it doesn't raise the quantizer of complex blocks, the frames with tons of AQ, like this one, get drastically larger than with my algorithm, hurting the rest of the video considerably. Interestingly enough, the old AQ algorithm on max strength doesn't lower the quantizers nearly as much as the new one does... a good or bad thing depending on how you look at it.

[CruNcher]

Yep it looks nice especialy how it preserves the grain structure (not the artificial flat Digital Cinema look), same as Atemes this way, did you tried it on ED the green background scene does it help there ? @ the 500 kbps low bitrate one, should also help @ your vendeta scene shouldn't it ?
Ateme does much better on the gradients there :) , i supose most of the blocks in those frames are intra 8x8 ?

[Dark Shikari]

Original post has been updated with new executable so you can test it out.

[Sagittaire]

Results with some tweaks:

2000kbps, without AQ:

http://i5.tinypic.com/6hfy59s.png

2000kbps, with new AQ:

http://i12.tinypic.com/4l65bv7.png

The old AQ algorithm has similar appearance, but because it doesn't raise the quantizer of complex blocks, the frames with tons of AQ, like this one, get drastically larger than with my algorithm, hurting the rest of the video considerably. Interestingly enough, the old AQ algorithm on max strength doesn't lower the quantizers nearly as much as the new one does... a good or bad thing depending on how you look at it.

Well but the most important is the frame size. AQ work well only if visual quality is better with constant frame size.

[Dark Shikari]

Well but the most important is the frame size. AQ work well only if visual quality is better with constant frame size.

This may come as a surprise:

http://i8.tinypic.com/62fpifl.png

:D

This is from the section of the clip which the picture I posted is from--the AQ version uses fewer bits for that set of frames! The magic is likely in the much higher quantizers for the high detail portions of the frame, which (as expected) have the most motion.

[CruNcher]

wow nice i have to test this imidiatly with my test sequence, so it seems you have changed the partition distro to be more efficient with this wasteing less bits perfect if it's real :D
even the intra frame seems to have a lower size, could you post the intra frame for both :D

[Daodan]

I would be curios as to how the old AQ handles that riddick scene. The quality in backgrounds was also raised quite a lot with old one in all my encodes.
I'll try some tests on more difficult stuff, like Underworld. That riddick pic just doesn't have any high fine detail area (like the faces in underworld in contrast to the noisy dark backgrounds) to see how much that is affected.

EDIT:
yes the quant window does show a nice result, and to be honest i believe yours looks better as well. i used 0.4 AQ in both encodes, nothing was changed.

heres the 2 encodes complete :

encodes (http://www.cif-forums.com/png/N.rar)
Interesting, new aq shows some improvement in certain areas, but overall one would say it's worse, the castle in light lost detail heavily. After all, the strong detail and areas in focus/interest should keep high detail first of all (since that's what we're watching), then try to make the rest look acceptable.
New aq pic: http://img341.imageshack.us/img341/1432/newaqpicob4.th.png (http://img341.imageshack.us/my.php?image=newaqpicob4.png)
Old aq pic: http://img341.imageshack.us/img341/3017/oldaqpicjx5.th.png (http://img341.imageshack.us/my.php?image=oldaqpicjx5.png)
New aq pic with quants: http://img341.imageshack.us/img341/5276/newaqquantsus7.th.png (http://img341.imageshack.us/my.php?image=newaqquantsus7.png)
Old aq pic with quants: http://img341.imageshack.us/img341/3360/oldaqquantsrf7.th.png (http://img341.imageshack.us/my.php?image=oldaqquantsrf7.png)

[Dark Shikari]

I would be curios as to how the old AQ handles that riddick scene. The quality in backgrounds was also raised quite a lot with old one in all my encodes.
I'll try some tests on more difficult stuff, like Underworld. That riddick pic just doesn't have any high fine detail area (like the faces in underworld in contrast to the noisy dark backgrounds) to see how much that is affected.Old AQ gives almost exactly the same result visually but has higher frame sizes.

Fine detail would definitely be a good test, as my test was basically just dark/flat areas with grain.

[woah!]

could you make a build that is --threads enabled ...pretty please then i can use my balls to the wall settings on my quadcore heh...