New Adaptive Quantization

[Dark Shikari]

Original post edited since I now have a new algorithm.

Try out my new AQ method (http://tjhsst.edu/~jgarrett/x264aq.exe). Warning: It is very powerful and will butcher your PSNR, though it has little effect on SSIM. Don't judge by the metrics, judge by appearance. Also, note that it is not meant at all for 1-pass bitrate (CRF/QP is fine)--it butchers the quantizers in a way that doesn't keep the framewide quantizer even close to constant, so 1pass bitrate modes turn out horrible.

This is not at all final, so if it sucks, say so.

Also, note that only --aq-strength is active in this algorithm. Try 1.0 first, and lower it if its too powerful. Try to use it on videos that you know benefit a lot from AQ, or have serious blocking issues in the background.

[akupenguin]

This struck me as weird at first, as AQ is supposed to lower SSIM
The detail part of SSIM is weighted inversely by the stddev of each block. Thus assigning lower QP to blocks with lower stddev should result in higher SSIM per PSNR, and it doesn't especially surprise me that it's also higher SSIM per bit.

[Dark Shikari]

The detail part of SSIM is weighted inversely by the stddev of each block. Thus assigning lower QP to blocks with lower stddev should result in higher SSIM per PSNR, and it doesn't especially surprise me that it's also higher SSIM per bit.Ah, that makes sense then. I optimized for SSIM without even realizing it!

Anyways, its time to test how well this eliminates blocks. :)

[Sagittaire]

Well if you want implement complete AQ you can use multiple masking: luma, dark, temporal, spacial, contrast ... etc

IMO the most efficient HVS AQ masking are:
- Spacial complexity: use higher quantizer for complex texture
- Dark masking: use lower quantizer for dark zone

Peronnaly I use Dark noise filtering and Spacial complexity AQ. Dark noise filtering imply less complexity in dark zone and lower quantizer in dark zone if you use Spacial Complexity AQ.

[Terranigma]

Peronnaly I use Dark noise filtering and Spacial complexity AQ. Dark noise filtering imply less complexity in dark zone and lower quantizer in dark zone if you use Spacial Complexity AQ.
I would love to see this implemented a.s.a.p. It's very annoying when 90% of your encoded video is fine, and the rest for dark areas is hideous due to blocking. :scared:

[Dark Shikari]

I would love to see this implemented a.s.a.p. It's very annoying when 90% of your encoded video is fine, and the rest for dark areas is hideous due to blocking. :scared:If my method works properly it should do this, because if the block is very dark it'll have a very low standard deviation.

[Sagittaire]

I would love to see this implemented a.s.a.p. It's very annoying when 90% of your encoded video is fine, and the rest for dark areas is hideous due to blocking. :scared:

Yes with TFT/Plasma screen blocking in dark area is really annoying. But like a say Dark Noise Filtering + spacial complexity AQ solve that very well.

[Ranguvar]

Sounds awesome!

Can't wait to try this out on anime! Any idea whether this will have a greater advantage on that?

[Dark Shikari]

Sounds awesome!

Can't wait to try this out on anime! Any idea whether this will have a greater advantage on that?There is probably a lot of tuning I'll have to do; for anime one might also need a much higher strength, as anime tends to be overall much flatter than live-action footage.

[Dark Shikari]

Interesting--I tested it on a clip from the Chronicles of Riddick and it lowered quality because the average QP was raised by 2.5-3. I'm guessing that I will have to find a way for the encoder to figure out what the optimal "strength" for this option is, since it is so film, and perhaps even scene or frame dependent.

[Ranguvar]

Interesting--I tested it on a clip from the Chronicles of Riddick and it lowered quality because the average QP was raised by 2.5-3. I'm guessing that I will have to find a way for the encoder to figure out what the optimal "strength" for this option is, since it is so film, and perhaps even scene or frame dependent.

Interesting...

Sounds sweet; I will keep an eye on this one ;-)

[ToS_Maverick]

Dark Shikari you simply ROCK!

gimme an .exe and i'll test it with my blackpearl sample, where i know almost every frame ;)

can't wait to get my hands on this...

[Dark Shikari]

Dark Shikari you simply ROCK!

gimme an .exe and i'll test it with my blackpearl sample, where i know almost every frame ;)

can't wait to get my hands on this...Eh, the original algorithm turned out to be less than impressive at its original goal, which was to get rid of blocks.

So I've been modifying it somewhat... when I think its ready, I'll post an EXE.

[Dark Shikari]

Try out my new AQ method (http://tjhsst.edu/~jgarrett/x264aq.exe). Warning: It is very powerful and will butcher your PSNR, though it has little effect on SSIM. Don't judge by the metrics, judge by appearance. Also, note that it is not meant at all for 1-pass bitrate (CRF/QP is fine)--it butchers the quantizers in a way that doesn't keep the framewide quantizer even close to constant, so 1pass bitrate modes turn out horrible.

This is not at all final, so if it sucks, say so.

Also, note that only --aq-strength is active in this algorithm. Try 1.0 first, and lower it if its too powerful. Try to use it on videos that you know benefit a lot from AQ, or have serious blocking issues in the background.

[MarcioAB]

Moving in the right direction.
Except keeping the bitrate as closer as possible, I'm not relying on metrics, just the old and good eyes fixed over a bright 1680x1050 LCD (full screen from 720x480).

- I see improvement in blockness (on dark & etc areas): less, smaller and "less dancing" blocks.
- Results are better with the default matrix than with eqm_avc_hr matrix.
- not compiled to MP4 ? (but ok)
- much slower than current x264 ! (but again, ok).
- nice to see very low mean Q's on dark areas: 14's when Q(P)=24 (but still a bit blocky on these cases)

Good job. Looking forward.

[Dark Shikari]

Moving in the right direction.
Except keeping the bitrate as closer as possible, I'm not relying on metrics, just the old and good eyes fixed over a bright 1680x1050 LCD (full screen from 720x480).

- I see improvement in blockness (on dark & etc areas): less, smaller and "less dancing" blocks.
- Results are better with the default matrix than with eqm_avc_hr matrix.
- not compiled to MP4 ? (but ok)
- much slower than current x264 ! (but again, ok).
- nice to see very low mean Q's on dark areas: 14's when Q(P)=24 (but still a bit blocky on these cases)

Good job. Looking forward.The speed is just because its completely unoptimized: I intentionally used no assembly and such because I want to get it working before I try to speed it up. Optimized it should be almost negligable speed decrease.

Is QP 14 really blocky? I intentionally capped the drop of QP by QP/2 (so if frame QP is 24, it won't drop below 12). If one needs even lower QPs to get rid of blocking, I can slightly reduce that cap.

[woah!]

heres a few screens to look at, i used r676 as a compare. also yours doesnt thread cpu's so 4x slower for me:

partitions:

x264aq
http://www.cif-forums.com/png/x264aq.PART.png

676.x264
http://www.cif-forums.com/png/676.x264.PART.png

Quantizers: white – 0 / black – 31

x264aq
http://www.cif-forums.com/png/x264aq.QT.png

676.x264
http://www.cif-forums.com/png/676.x264.QT.png

Macroblocks: macroblock sizes (in bits) in the video window: black– 0; white – max for the current frame

x264aq
http://www.cif-forums.com/png/x264aq.MB.png

676.x264
http://www.cif-forums.com/png/676.x264.MB.png

[Dark Shikari]

Notice what I did there with the quantizers--the entire frame was relatively flat, and so I gave the entire frame more bits. This seems very effective in getting rid of blocks--one of the main problems I find with x264 is that it will decide that a scene doesn't need a lot of bits, skip almost all the blocks, and you'll get a high-PSNR but quite blocky encode in that scene. The bits saved in that scene go elsewhere, but are not as useful as they would be if they stayed in that scene.

[woah!]

yes the quant window does show a nice result, and to be honest i believe yours looks better as well. i used 0.4 AQ in both encodes, nothing was changed.

heres the 2 encodes complete :

encodes (http://www.cif-forums.com/png/N.rar)

[MarcioAB]

Is QP 14 really blocky? I intentionally capped the drop of QP by QP/2 (so if frame QP is 24, it won't drop below 12). If one needs even lower QPs to get rid of blocking, I can slightly reduce that cap.

I encoded with --qp 24 --bframe 16 and 1.0xAQ (24 is an example, I encoded with --qp variations around that to keep the same bitrate to compare), that means without AQ the P-frame will receive Q=24 and the B-frame will receive Q=26. Because I used --bframe 16, I told the Rate Control to use as much B-frames it consider necessary, what in my encode case is around 5 or 6.
Obs: Near on single very-trouble area I see sequences of 10 P-frames (without any B-frame), but in general I can say 1P+5B is the standard for this encode.

Anyway, as B-frame prevails ( Q=26 ) and your target for "trouble" macroblocks will be Q/2=13 and I notice ffdshow reporting entire frame with mean-Q=14, I guess everything is working as designed.

So, now finally answering your question: Yes, I still see blocky on "trouble" areas even with ffdshow indicating frame mean-Q as 14. And than I look back on the source MPEG2 (0.65 bpp) and it's not that bad over there. So, x264 is really not good on dark/flat/slow areas.

I fully agree with you: x264 rocks on light/complex areas but is not so good on dark/flat/slow areas.

Thank you.

[Sagittaire]

I can see the patch ... ?

IMO it's better to use filter for Dark zone and after use spacial AQ. IMO Use directly dark AQ masking for dark zone with heavy grain/noise is useless.

[MarcioAB]

*deleted* I guess you need the source code ...

[ToS_Maverick]

it's sceenshot time!

from left to right:
Source, CRF 20 Old AQ 0.6/10, CRF 20 New AQ 0.7, CRF 21 New AQ 0.9, CRF 22 New AQ 1.0
all about the same size

http://img217.imageshack.us/img217/6881/385sourcesmallhc9.th.png (http://img217.imageshack.us/my.php?image=385sourcesmallhc9.png) http://img464.imageshack.us/img464/3484/385oldaqsmallfb7.th.png (http://img464.imageshack.us/my.php?image=385oldaqsmallfb7.png) http://img521.imageshack.us/img521/4104/385newaq07smallyz1.th.png (http://img521.imageshack.us/my.php?image=385newaq07smallyz1.png) http://img338.imageshack.us/img338/4638/385newaq09smallfr4.th.png (http://img338.imageshack.us/my.php?image=385newaq09smallfr4.png) http://img337.imageshack.us/img337/2926/385newaq10smallty1.th.png (http://img337.imageshack.us/my.php?image=385newaq10smallty1.png)

http://img521.imageshack.us/img521/2301/4025sourcesmallcy2.th.png (http://img521.imageshack.us/my.php?image=4025sourcesmallcy2.png) http://img413.imageshack.us/img413/8276/4025oldaqsmallfi7.th.png (http://img413.imageshack.us/my.php?image=4025oldaqsmallfi7.png) http://img160.imageshack.us/img160/5958/4025newaq07smallcm3.th.png (http://img160.imageshack.us/my.php?image=4025newaq07smallcm3.png) http://img227.imageshack.us/img227/57/4025newaq09smallth8.th.png (http://img227.imageshack.us/my.php?image=4025newaq09smallth8.png) http://img221.imageshack.us/img221/7996/4025newaq10smallzh3.th.png (http://img221.imageshack.us/my.php?image=4025newaq10smallzh3.png)

Summary:
- the new AQ is always better than the old one, never worse.
- the new AQ seems to have a problem at CRF21 0.9 strength, take a look at the screen ;)
- these are 2 of the more impressive examples, the old AQ isn't that bad, in some cases, the new one is much better.

commandlines:

Old AQ:
--crf 20.0 --level 3 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 1835 --vbv-maxrate 10000 --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "" "F:\Video\Black.Pearl.Sample.avs" --aq-strength 0.6 --aq-sensitivity 10

New AQ:
x264aq.exe --crf 20.0 --level 3 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 1835 --vbv-maxrate 10000 --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "" "F:\Video\Black.Pearl.Sample.avs" --aq-strength 0.7


i don't post the logfiles, since this isn't about metrics, it's about the look :cool:

[Dark Shikari]

Here is the code. Note the complete lack of assembly, etc, etc which I'll fix.

void x264_adaptive_quant( x264_t *h, x264_mb_analysis_t *a )
{
DECLARE_ALIGNED( static uint8_t, zero[FDEC_STRIDE*8], 16 );
DECLARE_ALIGNED( int16_t, dct[8][8], 16 );
float fc;
int total = 0;
int qp = h->mb.i_qp, qp_adj;
int i;
int variance = h->pixf.count_16x16(h->mb.pic.p_fenc[0], FENC_STRIDE);
x264_emms();
float standardDeviation = sqrt(variance/256.0);
float relativeSD = standardDeviation /= (h->param.standardDeviation + 0.01);
if(relativeSD > 1)
{
qp_adj = (qp/2.0) * h->param.analyse.f_aq_strength * (relativeSD - 1.0);
qp_adj = X264_MIN(qp_adj, qp/2);
//printf("Positive Adjust: %d out of total QP %d\n",qp_adj,qp);
}
else
{
qp_adj = -(qp/4.0) * h->param.analyse.f_aq_strength * (0.1/(0.01+relativeSD));
qp_adj = X264_MAX(qp_adj, -qp/2);
//printf("Negative Adjust: %d out of total QP %d\n",qp_adj,qp);
}
if(h->param.standardDeviation < 30) qp_adj -= h->param.analyse.f_aq_strength*(30.0 - h->param.standardDeviation)*(30.0 - h->param.standardDeviation)/60.0;
qp_adj = X264_MAX(qp_adj, -qp/2);
h->mb.i_qp = a->i_qp = X264_MIN(qp + qp_adj,51);
h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
}

h->param.standardDeviation is not the standard deviation of the whole frame; its the average of the standard deviations of 16x16 blocks of that frame.

Also, I think the effectiveness of this patch is proof that PSNR is completely utterly useless ;)

[Sagittaire]

Here is the code. Note the complete lack of assembly, etc, etc which I'll fix.

void x264_adaptive_quant( x264_t *h, x264_mb_analysis_t *a )
{
DECLARE_ALIGNED( static uint8_t, zero[FDEC_STRIDE*8], 16 );
DECLARE_ALIGNED( int16_t, dct[8][8], 16 );
float fc;
int total = 0;
int qp = h->mb.i_qp, qp_adj;
int i;
int variance = h->pixf.count_16x16(h->mb.pic.p_fenc[0], FENC_STRIDE);
x264_emms();
float standardDeviation = sqrt(variance/256.0);
float relativeSD = standardDeviation /= (h->param.standardDeviation + 0.01);
if(relativeSD > 1)
{
qp_adj = (qp/2.0) * h->param.analyse.f_aq_strength * (relativeSD - 1.0);
qp_adj = X264_MIN(qp_adj, qp/2);
//printf("Positive Adjust: %d out of total QP %d\n",qp_adj,qp);
}
else
{
qp_adj = -(qp/4.0) * h->param.analyse.f_aq_strength * (0.1/(0.01+relativeSD));
qp_adj = X264_MAX(qp_adj, -qp/2);
//printf("Negative Adjust: %d out of total QP %d\n",qp_adj,qp);
}
if(h->param.standardDeviation < 30) qp_adj -= h->param.analyse.f_aq_strength*(30.0 - h->param.standardDeviation)*(30.0 - h->param.standardDeviation)/60.0;
qp_adj = X264_MAX(qp_adj, -qp/2);
h->mb.i_qp = a->i_qp = X264_MIN(qp + qp_adj,51);
h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
}

h->param.standardDeviation is not the standard deviation of the whole frame; its the average of the standard deviations of 16x16 blocks of that frame.

Also, I think the effectiveness of this patch is proof that PSNR is completely utterly useless ;)

I think that qp/2 is a too high overquantized limit. I will try to optimize that with best value for my eyes ... ;-)

There are theorical max value for standarddeviation?

[Dark Shikari]

I think that qp/2 is a too high overquantized limit. I will try to optimize that with best value for my eyes ... ;-)

There are theorical max value for standarddeviation?
Max value for standard deviation is extremely high, but will basically never be reached, or even close.

Another problem is that standard deviation varies based on the source, which is why I had it calculate framewide standard deviation to use as a reference.

Also note you need this code in encoder.c to calculate the standard deviation:

int x,y;
x264_emms();
double totalStandardDeviation = 0;
int num = 0;
for(x=0;x < h->param.i_width-16;x+=16)
for(y=0;y < h->param.i_height-16;y+=16)
{
num++;
int variance = h->pixf.count_16x16(h->fenc->plane[0]+x+y*h->fenc->i_stride[0], FENC_STRIDE);
totalStandardDeviation += sqrt(variance/256.0);
}
h->param.standardDeviation = totalStandardDeviation / num;

and of course you need to add param.standardDeviation in the struct in x264.h... I should really post a patch but I don't think its really ready for that yet.

[Sharktooth]

i think QP/2 is not good to.
i would use something like QP - f_aq_strength...

also, why
qp_adj = (qp/2.0) * h->param.analyse.f_aq_strength * (relativeSD - 1.0);
qp_adj = X264_MIN(qp_adj, qp/2);
and
qp_adj = -(qp/4.0) * h->param.analyse.f_aq_strength * (0.1/(0.01+relativeSD));
qp_adj = X264_MAX(qp_adj, -qp/2);
?!?!?

[Gabriel_Bouvigne]

I'd suggest you to do adjustment using q scale, and not qp scale, as it's a bit more linear (there are qp2qscale and qscale2qp functions).

[Dark Shikari]

i think QP/2 is not good to.
i would use something like QP - f_aq_strength...

also, why
qp_adj = (qp/2.0) * h->param.analyse.f_aq_strength * (relativeSD - 1.0);
qp_adj = X264_MIN(qp_adj, qp/2);
and
qp_adj = -(qp/4.0) * h->param.analyse.f_aq_strength * (0.1/(0.01+relativeSD));
qp_adj = X264_MAX(qp_adj, -qp/2);
?!?!?Because it worked :D

As I said, its very experimental and I'll definitely try using a better scaling method if I can find one that works.

[Sharktooth]

ok...

[MarcioAB]

My 2 cents ...

I think Q=10 is kind of yield number in x264. So instead push Q to QP/2, it could be pushed to 10 (or user defined, maybe function of something else).

Example: encode with --qp 18: A "trouble" macroblock may be given Q=9, but 9 is a waste of resource (IMO).

On the other side encode with --qp28: A really "trouble" macroblock will get Q=10, assuring max workout on such "trouble" area.

[Sharktooth]

that's why i proposed qp - f_aq_strength (assuming f_aq_streng is defined as int).
let's assume qp is 28 and aq_strength is set to 10 the result would be 18...
rising the strength will lower the qp...

EDIT: uhm... just saw the new aq_strength is float...
however is quite easy. something like this (for positive adjust... similar for other cases...):
int aq_bias;
....
....
if(relativeSD > 1)
{
aq_bias = qp - (param.analyse.f_aq_strength * 10);
qp_adj = aq_bias * h->param.analyse.f_aq_strength * (relativeSD - 1.0);
qp_adj = X264_MIN(qp_adj, aq_bias);
}
...
...

[Dark Shikari]

I came up with a new formula:

qfactor = exp(1.0*(relativeSD - 1.0)*(relativeSD - 1.0)*sign(relativeSD - 1.0));
qfactor *= h->param.standardDeviation/30.0;
float qp_adj = (qscale2qp(qp2qscale(qp)*qfactor)) - qp;
qp_adj *= h->param.analyse.f_aq_strength;
int newQP = (int)qp_adj + qp;
h->mb.i_qp = a->i_qp = X264_MAX(X264_MIN(X264_MAX(X264_MIN(newQP,3*qp/2),qp/2),51),10);

It uses qscale, not sure if its better or not but it seems to do a pretty good job overall.

[Sharktooth]

uhm.... isnt it a bit too 'complex' just for calculating the final qp? and the qp/2 is still there... :P

[Dark Shikari]

uhm.... isnt it a bit too 'complex' just for calculating the final qp?The formula has to take into account many things:

a) If the block is "flatter" than the rest of the scene, it needs to have a lower QP than the rest of the scene. This can't really be linear.
b) If the block is very complex, it needs to have a higher QP, to add more bits back into the rest of the scene. This can't really be linear either.
c) If the entire scene is flat, the entire scene's QP needs to be lower.

[Sharktooth]

edited the previous posts...

[Dark Shikari]

edited the previous posts...
Your formula multiplies in the strength twice, resulting in a nonlinear scaling of AQ strength.

[Sharktooth]

it was just an idea:p
however the qp/2 is empirical...

[Sagittaire]

The formula has to take into account many things:

a) If the block is "flatter" than the rest of the scene, it needs to have a lower QP than the rest of the scene. This can't really be linear.
b) If the block is very complex, it needs to have a higher QP, to add more bits back into the rest of the scene. This can't really be linear either.
c) If the entire scene is flat, the entire scene's QP needs to be lower.

Well I think that c is useless because if the entire frame is flat then RC will always use lower relative quant (Rate Control equation with texture complexity).

[Dark Shikari]

Well I think that c is useless because if the entire frame is flat then RC will always use lower relative quant (Rate Control equation with texture complexity).This doesn't seem to be the case at all in my experience, or at least not enough. A lot of blocking problems, as I've said before, are caused by x264 deciding that a flat scene needs very few bits, and as such hardly allocating any bits to it.

Another issue is that my formula sets the quantizer offset based on the SD relative to the frame's SD. Ideally, one would want to look at the entire film's SD, which would avoid having to deal with the problem of c). a) and b) only work when there are dark blocks in an otherwise normal frame--a frame entirely of dark/flat blocks needs c).

[Dark Shikari]

Wow, that was retarded, I was multiplying instead of dividing.

void x264_adaptive_quant( x264_t *h, x264_mb_analysis_t *a )
{
int qp = h->mb.i_qp;
float qfactor;
int variance = h->pixf.count_16x16(h->mb.pic.p_fenc[0], FENC_STRIDE);
x264_emms();
float standardDeviation = sqrt(variance/256.0);
float relativeSD = standardDeviation /= (h->param.standardDeviation + 0.01);
qfactor = exp(1.0*(relativeSD - 1.0)*(relativeSD - 1.0)*sign(relativeSD - 1.0));
qfactor /= sqrt(h->param.standardDeviation/30.0);
float qp_adj = (qscale2qp(qp2qscale(qp)*qfactor)) - qp;
qp_adj *= h->param.analyse.f_aq_strength;
int newQP = (int)qp_adj + qp;
h->mb.i_qp = a->i_qp = X264_MAX(X264_MIN(X264_MAX(X264_MIN(newQP,3*qp/2),qp/2),51),10);
h->mb.i_chroma_qp = i_chroma_qp_table[x264_clip3( h->mb.i_qp + h->pps->i_chroma_qp_index_offset, 0, 51 )];
}

[Dark Shikari]

Results with some tweaks:

2000kbps, without AQ:

http://i5.tinypic.com/6hfy59s.png

2000kbps, with new AQ:

http://i12.tinypic.com/4l65bv7.png

The old AQ algorithm has similar appearance, but because it doesn't raise the quantizer of complex blocks, the frames with tons of AQ, like this one, get drastically larger than with my algorithm, hurting the rest of the video considerably. Interestingly enough, the old AQ algorithm on max strength doesn't lower the quantizers nearly as much as the new one does... a good or bad thing depending on how you look at it.

[CruNcher]

Yep it looks nice especialy how it preserves the grain structure (not the artificial flat Digital Cinema look), same as Atemes this way, did you tried it on ED the green background scene does it help there ? @ the 500 kbps low bitrate one, should also help @ your vendeta scene shouldn't it ?
Ateme does much better on the gradients there :) , i supose most of the blocks in those frames are intra 8x8 ?

[Dark Shikari]

Original post has been updated with new executable so you can test it out.

[Sagittaire]

Results with some tweaks:

2000kbps, without AQ:

http://i5.tinypic.com/6hfy59s.png

2000kbps, with new AQ:

http://i12.tinypic.com/4l65bv7.png

The old AQ algorithm has similar appearance, but because it doesn't raise the quantizer of complex blocks, the frames with tons of AQ, like this one, get drastically larger than with my algorithm, hurting the rest of the video considerably. Interestingly enough, the old AQ algorithm on max strength doesn't lower the quantizers nearly as much as the new one does... a good or bad thing depending on how you look at it.

Well but the most important is the frame size. AQ work well only if visual quality is better with constant frame size.

[Dark Shikari]

Well but the most important is the frame size. AQ work well only if visual quality is better with constant frame size.

This may come as a surprise:

http://i8.tinypic.com/62fpifl.png

:D

This is from the section of the clip which the picture I posted is from--the AQ version uses fewer bits for that set of frames! The magic is likely in the much higher quantizers for the high detail portions of the frame, which (as expected) have the most motion.

[CruNcher]

wow nice i have to test this imidiatly with my test sequence, so it seems you have changed the partition distro to be more efficient with this wasteing less bits perfect if it's real :D
even the intra frame seems to have a lower size, could you post the intra frame for both :D

[Daodan]

I would be curios as to how the old AQ handles that riddick scene. The quality in backgrounds was also raised quite a lot with old one in all my encodes.
I'll try some tests on more difficult stuff, like Underworld. That riddick pic just doesn't have any high fine detail area (like the faces in underworld in contrast to the noisy dark backgrounds) to see how much that is affected.

EDIT:
yes the quant window does show a nice result, and to be honest i believe yours looks better as well. i used 0.4 AQ in both encodes, nothing was changed.

heres the 2 encodes complete :

encodes (http://www.cif-forums.com/png/N.rar)
Interesting, new aq shows some improvement in certain areas, but overall one would say it's worse, the castle in light lost detail heavily. After all, the strong detail and areas in focus/interest should keep high detail first of all (since that's what we're watching), then try to make the rest look acceptable.
New aq pic: http://img341.imageshack.us/img341/1432/newaqpicob4.th.png (http://img341.imageshack.us/my.php?image=newaqpicob4.png)
Old aq pic: http://img341.imageshack.us/img341/3017/oldaqpicjx5.th.png (http://img341.imageshack.us/my.php?image=oldaqpicjx5.png)
New aq pic with quants: http://img341.imageshack.us/img341/5276/newaqquantsus7.th.png (http://img341.imageshack.us/my.php?image=newaqquantsus7.png)
Old aq pic with quants: http://img341.imageshack.us/img341/3360/oldaqquantsrf7.th.png (http://img341.imageshack.us/my.php?image=oldaqquantsrf7.png)

[Dark Shikari]

I would be curios as to how the old AQ handles that riddick scene. The quality in backgrounds was also raised quite a lot with old one in all my encodes.
I'll try some tests on more difficult stuff, like Underworld. That riddick pic just doesn't have any high fine detail area (like the faces in underworld in contrast to the noisy dark backgrounds) to see how much that is affected.Old AQ gives almost exactly the same result visually but has higher frame sizes.

Fine detail would definitely be a good test, as my test was basically just dark/flat areas with grain.

[woah!]

could you make a build that is --threads enabled ...pretty please then i can use my balls to the wall settings on my quadcore heh...

[Dark Shikari]

I would be curios as to how the old AQ handles that riddick scene. The quality in backgrounds was also raised quite a lot with old one in all my encodes.
I'll try some tests on more difficult stuff, like Underworld. That riddick pic just doesn't have any high fine detail area (like the faces in underworld in contrast to the noisy dark backgrounds) to see how much that is affected.

EDIT:

Interesting, new aq shows some improvement in certain areas, but overall one would say it's worse, the castle in light lost detail heavily. After all, the strong detail and areas in focus/interest should keep high detail first of all (since that's what we're watching), then try to make the rest look acceptable.
New aq pic: http://img341.imageshack.us/img341/1432/newaqpicob4.th.png (http://img341.imageshack.us/my.php?image=newaqpicob4.png)
Old aq pic: http://img341.imageshack.us/img341/3017/oldaqpicjx5.th.png (http://img341.imageshack.us/my.php?image=oldaqpicjx5.png)
New aq pic with quants: http://img341.imageshack.us/img341/5276/newaqquantsus7.th.png (http://img341.imageshack.us/my.php?image=newaqquantsus7.png)
Old aq pic with quants: http://img341.imageshack.us/img341/3360/oldaqquantsrf7.th.png (http://img341.imageshack.us/my.php?image=oldaqquantsrf7.png)
The reason for that is probably because the new AQ has a somewhat different philosophy. Notice my sample, for example: I'm encoding at 2000kbps, a very high value for a letterboxed DVD-resolution video, but without AQ there is still a serious problem with blocking!

You're encoding at a low enough bitrate that there is normally a blocking problem to begin with; no AQ can possibly fix that, as there aren't enough bits in the scene to do so. Using my AQ on a high strength value is just going to decimate the most complex parts of the scene. Using my AQ on a very high strength value is certain to lower visual quality in at that bitrate.

[Sharktooth]

what about a non linear deblocking strength? using higher inloop deblocking on high quants will avoid blocks but will drop details, so the new AQ can be a sort of balance between foreground and background details...

[Dark Shikari]

what about a non linear deblocking strength? using higher inloop deblocking on high quants will avoid blocks but will drop details, so the new AQ can be a sort of balance between foreground and background details...Is that even possible? I thought the deblock filter had to be a single constant value throughout the video and was equivalent on playback and encoding?

[Sharktooth]

since deblock is triggered by QP, using AQ will surely change the deblocking...
the ateme encoder used to have inloop deblocking matrices too...

[Sergey A. Sablin]

Is that even possible? I thought the deblock filter had to be a single constant value throughout the video and was equivalent on playback and encoding?

filter offsets are written in each slice header, so one may adapt the values to average slice qp.

[MarcioAB]

Original post has been updated with new executable so you can test it out.

Well, at least for me the previous build was MUCH better.

[Dark Shikari]

Well, at least for me the previous build was MUCH better.Note that I did change the build in the meantime with something I thought wouldn't affect anything but I might have messed up. I was asked by someone on IRC to make the max/min QP adjustable, so I tied them to qpmax and qpmin, which by default were 51 and 10, what I was already using.

I assume I probably messed something up; I don't think a dumb typo would be beyond me. I'll check it later when I get back to my computer.

[Razorholt]

Well, at least for me the previous build was MUCH better.

Can you - or Dark Shikari - re-post the previous build for me to compare it with the recent one?

Cheers,
- Dan

[Dark Shikari]

Strange, I didn't mess anything up, my change worked fine... in fact the build I used for the images I posted is exactly the same as the one I'm currently using.

So if you want the old build you're probably out of luck, I don't have it anymore. :(

Also, looking at more Elecard tests of my build one thing that I notice it does is that it drastically cuts the number of skipped blocks, by using the bits from the most complex blocks to avoid skipping less complex blocks. This seems to be the core of its benefit.

[ChronoCross]

Strange, I didn't mess anything up, my change worked fine... in fact the build I used for the images I posted is exactly the same as the one I'm currently using.

So if you want the old build you're probably out of luck, I don't have it anymore. :(



We need to get you a patch svn repository.....lol

[Dark Shikari]

We need to get you a patch svn repository.....lolI actually have a friend who is running an SVN for me but I never get around to actually using it... :D

I do have my ungodly-long undo history in Notepad++ :)

Why in god's name doesn't the x264 SVN have branches? If it did I could just get access to a branch and develop whatever I wanted there.

[Daodan]

Old (row 1) vs new aq (row 2):

http://img504.imageshack.us/img504/7396/oldaq10eh8.th.jpg (http://img504.imageshack.us/my.php?image=oldaq10eh8.jpg) http://img504.imageshack.us/img504/7581/oldaq20cf2.th.jpg (http://img504.imageshack.us/my.php?image=oldaq20cf2.jpg) http://img504.imageshack.us/img504/3964/oldaq30yk3.th.jpg (http://img504.imageshack.us/my.php?image=oldaq30yk3.jpg) http://img504.imageshack.us/img504/7461/oldaq40ta0.th.jpg (http://img504.imageshack.us/my.php?image=oldaq40ta0.jpg) http://img504.imageshack.us/img504/1599/oldaq50ur7.th.jpg (http://img504.imageshack.us/my.php?image=oldaq50ur7.jpg) http://img504.imageshack.us/img504/4998/oldaq60yz3.th.jpg (http://img504.imageshack.us/my.php?image=oldaq60yz3.jpg) http://img504.imageshack.us/img504/7108/oldaq70gv7.th.jpg (http://img504.imageshack.us/my.php?image=oldaq70gv7.jpg)


http://img504.imageshack.us/img504/2305/newaq10dk5.th.jpg (http://img504.imageshack.us/my.php?image=newaq10dk5.jpg) http://img504.imageshack.us/img504/7480/newaq20lk5.th.jpg (http://img504.imageshack.us/my.php?image=newaq20lk5.jpg) http://img504.imageshack.us/img504/3215/newaq30gb6.th.jpg (http://img504.imageshack.us/my.php?image=newaq30gb6.jpg) http://img504.imageshack.us/img504/5999/newaq40kt9.th.jpg (http://img504.imageshack.us/my.php?image=newaq40kt9.jpg) http://img504.imageshack.us/img504/9776/newaq50ob1.th.jpg (http://img504.imageshack.us/my.php?image=newaq50ob1.jpg) http://img504.imageshack.us/img504/1940/newaq60xv4.th.jpg (http://img504.imageshack.us/my.php?image=newaq60xv4.jpg) http://img504.imageshack.us/img504/7273/newaq70cx8.th.jpg (http://img504.imageshack.us/my.php?image=newaq70cx8.jpg)

This is a more life-like encode, using reasonable bitrates (not too high, not too low)

What seems to happen is that bframes seem to get increased quality especially, while p and i stays quite same (and in very few cases it gets worse). Overall I'd say there's an improvement though. Any chance of a real build with it?

[Mutant_Fruit]

Why in god's name doesn't the x264 SVN have branches? If it did I could just get access to a branch and develop whatever I wanted there.

It does, it's called: http://code.google.com/hosting ;) Check the code out from the x264 svn, check it in there, bada bing bada bang. Not quite as nice as having a branch in the same SVN, but just as easy to use.

[imcold]

Why in god's name doesn't the x264 SVN have branches? If it did I could just get access to a branch and develop whatever I wanted there.
Making a svn repository on your machine, importing the current x264 version to it and working in this local repo could also be an option. Almost the same as branch.

[Dark Shikari]

This is a more life-like encode, using reasonable bitrates (not too high, not too low)

What seems to happen is that bframes seem to get increased quality especially, while p and i stays quite same (and in very few cases it gets worse). Overall I'd say there's an improvement though. Any chance of a real build with it?
That's some very impressive grain retention.

Interesting that B-frames get increased quality--I wonder if that's a good or a bad thing; should I try to save those bits for use elsewhere, or do the B-frames need the boost? Perhaps its because one needs a similar amount of bits regardless of frametype for the grain retention.

I can post a patch later today if someone wants to make a threaded build of it.

[Atak_Snajpera]

Daodan
What settings did you use in both examples?

[Daodan]

5500 kbps, aq 0.7, mean quantizer somewhere at 21.3. Unfortunately I'm not sure if I used or not turbo on first sample (on second I know I didn't), because I deleted the logs. So for now let's say the test it's not conclusive in any way and wait for some other people's tests untill I can make some new ones.
As for bframes being good quality, I'd say yes, gives a more stable look overall.

[Atak_Snajpera]

I would like to see it in CRF22.For instance old --aq-strength 0.5 --aq-sensitivity 5 vs new equivalent.

[CruNcher]

@Dark Shikari
im not sure if this is a non mod16 problem but look @ this hmm something is wrong ;)

http://s6.directupload.net/images/070920/temp/2b2M9vhq.png (http://s6.directupload.net/images/070920/2b2M9vhq.png)

settings: x264aq --pass 2 --bitrate 3000 --stats "full.stats" --level 4.1 --min-keyint 1 --ref 3 --nf --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-maxrate 25000 --me dia --merange 8 --threads auto --thread-input --progress --no-psnr --no-ssim --output "full.264" "full.avs" --aq-strength 1.0

[Dark Shikari]

@Dark Shikari
im not sure if this is a non mod16 problem but look @ this hmm something is wrong ;)

http://s6.directupload.net/images/070920/temp/2b2M9vhq.png (http://s6.directupload.net/images/070920/2b2M9vhq.png)
Ah crap, it probably is a non-mod16 problem of some sort... can you test the same footage in mod16 proportions (crop it) just to make sure that it is a non-mod-16 problem?

[CruNcher]

im gonna do it new with a shorter test sample and some different settings just to be sure, somehow i doub't it has todo with mod16 as Daodan his sample is also not mod16 ;)

[Dark Shikari]

im gonna do it new with a shorter test sample and some different settings just to be sure, somehow i doub't it has todo with mod16 as Daodan his sample is also not mod16 ;)
Can you upload the .h264 file so I can check the quants?

[CruNcher]

yep no problem, but also the mod16 shows the same problem resized to 1280x720 with this settings

1st pass:
x264aq --pass 1 --bitrate 3000 --stats "full1.stats" --level 4.1 --min-keyint 1 --nf --subme 1 --analyse none --vbv-maxrate 25000 --vbv-init 1.0 --vbv-bufsize 14745 --mvrange 511 --me dia --merange 8 --threads auto --thread-input --progress --no-psnr --no-ssim --output NUL "full.avs" --aq-strength 1.0

2nd pass:
x264aq --pass 2 --bitrate 3000 --stats "full1.stats" --level 4.1 --min-keyint 1 --nf --subme 1 --analyse none --vbv-maxrate 25000 --vbv-init 1.0 --vbv-bufsize 14745 --mvrange 511 --me dia --merange 8 --threads auto --thread-input --progress --no-psnr --no-ssim --output "full.264" "full.avs" --aq-strength 1.0

[Dark Shikari]

x264aq --pass 1 --bitrate 3000 --stats "full1.stats" --level 4.1 --min-keyint 1 --nf --subme 1 --analyse none --vbv-maxrate 25000 --vbv-init 1.0 --vbv-bufsize 14745 --mvrange 511 --me dia --merange 8 --threads auto --thread-input --progress --no-psnr --no-ssim --output NUL "full.avs" --aq-strength 1.0

2nd pass:
x264aq --pass 1 --bitrate 3000 --stats "full1.stats" --level 4.1 --min-keyint 1 --nf --subme 1 --analyse none --vbv-maxrate 25000 --vbv-init 1.0 --vbv-bufsize 14745 --mvrange 511 --me dia --merange 8 --threads auto --thread-input --progress --no-psnr --no-ssim --output "full.264" "full.avs" --aq-strength 1.0

First of all, you're only using one pass, because your second pass has --pass 1. Second, --subme 1 is atrocious, and lowers quality/bitrate over 20-25% over --subme 2 in my tests. Third, your overall settings are so low that I wouldn't be surprised at that quality at all...

[Daodan]

2nd pass:
x264aq --pass 1 --bitrate 3000 --stats "full1.stats" --level 4.1 --min-keyint 1 --nf --subme 1 --analyse none --vbv-maxrate 25000 --vbv-init 1.0 --vbv-bufsize 14745 --mvrange 511 --me dia --merange 8 --threads auto --thread-input --progress --no-psnr --no-ssim --output "full.264" "full.avs" --aq-strength 1.0


OMG, CABAC is still ON? Get rid of it quickly :D

Seriosusly though, those are horrible settings. Not much point in making tests for something that no one will ever use. Kinda like Dark Shikari's V for V encode :D

[CruNcher]

eh sorry the --pass 1 is a c&p error it is correct and 2pass was used but anyway Dark Shikari it should work regardless if you find this settings to low, else i would say it's bugged and im testing a very advanced masking @ the moment and it doesn't show this kind of problems even @ low settings ;)

@Daodan
Cabac is usefull especialy @ what im doing but anyway who are you that you say no one will ever use those settings, only because you don't that's really egoistic and wrong you can really get unbalanced in your encodings thinking like that, but yeah the more compression the better (wrong) Metrics wise you might be right but Metrics and Look & Feel are 2 completely different things (HVS) ,you still have to learn alot it seems, but you @ the right place for that ;)

[Dark Shikari]

You're using a bitrate of 3 Mb/s on something that huge and expect to be able to retain grain?

All you're going to end up with are extremely high quants... like what happened :)

You can't compress 1080p at 3Mb/s and retain grain without using FGM. --aq-strength 1.0 is absurd for that low a bitrate. What happens is that it has so few bits that its forced to steal bits from the foreground, resulting in the ugly artifacting you saw.

Remember, I was using 2 Mb/s for 720x480 letterboxed. You're using only 50% more bits for 1080p.

[CruNcher]

you can't compress 1080p at 3Mb/s and retain grain without using FGM.

not the full grain layer i agree here with you, but a lossy perceptable version of it, see the low bitrate thread :)

[Dark Shikari]

not the full grain layer i agree here with you, but a lossy perceptable version of it, see the low bitrate thread :)Also note with your encode that here's what's happened:

1. AQ tries to reduce the quants in the background area. This requires a whole lot of bits.
2. Ratecontrol raises the overall frame quant in order to try to get the bits necessary.
3. Repeat 1 and 2 a few times.
4. Overall frame quant is now very high, on the order of 35 or more.
5. AQ keeps the background area quants lower, and raises the high-detail area quants by even more... with a max of qp/2, which now maxes out at QP 51!
6. A bunch of quants in the foreground go really high as a result, and it looks like shit. The background, due to both low quants and your very low motion search, consists of nothing but I-blocks and skipped blocks.

Without enough bits to keep the whole frame's quant below 30 or so, AQ-strength at 1.0 will have atrocious results.

Also notice your encode has no B-frames.

[CruNcher]

ok i see so i have to finetune the --aq-strength value then to avoid the artifacting gonna do some tests with not so high settings those where just thought as a quick test but yeah i have to tune this into the complete workflow (this gonna take time).
And yes i know that i can't keep the complete grain noise/layer (wich btw is lossy allready Mpeg-2 ;) ) but my target is a complete visualy different then what you might think, for me it's important that a small part of the spacial noise/grain survives the quantization so the result doesn't look in the end like Digital Cinema (H.264 tendency with high ME settings) but more like the input source and i almost arrived this with another implementation, different settings tough (higher then those posted above also motion estimation wise).
Im gonna try the same with X264 later but i still didn't finished with this on the other implementation, but the results are allready wow for my visual understanding @ this low bitrate.
It would be no problem to keep alot of the grain @ 6 mbit allready tested that also with x264 but my target is the lowest possible (the rest scales to the higher bitrate automaticly) an how it looks in the end, and don't tell me you can predict how this is gonna look finetuned (also don't forget it's variable bitrate so bitrate changes and so also the grain layer does change to high qp (low bitrate) it would be less perceptable in high bitrate zones it will be more perceptable) ;)
And about b-frames i never really was a fan of them they can introduce more visual problems then fix, it got better for sure these days but still not optimal, and later settings for sure will use them but this here was just a fast test how the grain preservation would look like with your AQ, and now we found out that --aq-strength 1.0 was to much so i gonna try a lower setting.

Just to give you an impression qp of 21 is saving alot of details and also the grain layer very nice, see Quicktime trailer encodes (especialy the long evan almighty one) my avg qp of 9 mins with the other implementation is 28 (with the letterbox) @ the moment @ 1080p most likely would be something as 25 @ 720p (with reduced grain due to resizing) so that's not really far and i think a nice result for 3 mbit 1080p @ 42 dB :)

PS: The problem is from the other implementation im working alot now with im not used such many finetuneing options, you just set the AQ and it goes of and the end result looks fine @ any bitrate so yes it seems adaptive (or very low set but high enough to give a visual enhancement).
For me it feels like that in x264 partition distrobution,deadzones and adaptive quantization do still need alot of tuneing but another question is should they be tuned based on a metric or visualy ?

[lexor]

Results with some tweaks:

2000kbps, without AQ:

[pic1 snip]

2000kbps, with new AQ:

[pic2 snip]

The old AQ algorithm has similar appearance, but because it doesn't raise the quantizer of complex blocks, the frames with tons of AQ, like this one, get drastically larger than with my algorithm, hurting the rest of the video considerably. Interestingly enough, the old AQ algorithm on max strength doesn't lower the quantizers nearly as much as the new one does... a good or bad thing depending on how you look at it.

I don't know if any changes were made since that post, but the new AQ in those 2 pics does a terrible job at preserving detail in light areas copared to no AQ first shot. New AQ is much blurier in areas with fine detail, enough so to negate any gain in dark areas. (imho)

[Dark Shikari]

I don't know if any changes were made since that post, but the new AQ in those 2 pics does a terrible job at preserving detail in light areas copared to no AQ first shot. New AQ is much blurier in areas with fine detail, enough so to negate any gain in dark areas. (imho)Can you point out specifically, maybe with some quick MS paint circling, where the problems are? I can then correlate this with the quantizers and try to fix the problem.

[lexor]

http://img222.imageshack.us/img222/8900/4l65bv7pi2.th.png (http://img222.imageshack.us/my.php?image=4l65bv7pi2.png)
as you get closer to the hand the ridges disappear (as you can see it happens in other areas as well, but not as much). This problem is relatively small in this sample, due to overwhelming darkness of the shot, but in my experience the worst shots that I always spend time trying to correct are just like that but with more light in the centre and deep dark all around. It gets even more noticeable when you watch it full screen.

[Dark Shikari]

http://img222.imageshack.us/img222/8900/4l65bv7pi2.th.png (http://img222.imageshack.us/my.php?image=4l65bv7pi2.png)
as you get closer to the hand the ridges disappear (as you can see it happens in other areas as well, but not as much). This problem is relatively small in this sample, due to overwhelming darkness of the shot, but in my experience the worst shots that I always spend time trying to correct are just like that but with more light in the centre and deep dark all around. It gets even more noticeable when you watch it full screen.
Its the same in the original footage, and in the non-AQ encode, at least thats what my eyes tell me.

The quantizer in that area is a good bit higher though.

[lexor]

ok on the scale (it looks like some scaled creature in the background to me, I can't really tell what it is) right bellow the ball (top half of marked area) the 3 first ridges closest to the hand disappear in new AQ shot. That is, the black vertical lines that form the ridges disappear and what's left is a uniform light blue glow. If you have problem seeing it (monitor contrast and such may play a role), just save the two pics in new folder, open in Window default Picture and Fax viewer, and click forward button repeatedly so it cycles.

[Dark Shikari]

ok on the scale right bellow the ball (top half of marked area) the 2 first ridges closest to the hand disappear in new AQ shot. That is, the black vertical lines that form the ridges disappear and what's left is a uniform light blue glow. If you have problem seeing it (monitor contrast and such may play a role), just save the two pics in new folder, open in Window default Picture and Fax viewer, and click forward button repeatedly so it cycles.
Yeah, they're very very faint in the original though; its probably something that wouldn't be as noticable when playing back the video rather than freezeframing.

If you have a problem with that kind of artifact you can try setting --qpmax appropriately, or you can just use less powerful AQ.

I could also try to find a more intelligent method of picking which blocks to raise the QP on and by how much.

[CruNcher]

lexor i doub't you spot the difference in motion as from the position it looks like a high motion scene where a cut is following shortly :d

[lexor]

lexor i doub't you spot the difference in motion as from the position it looks like a high motion scene where a cut is following shortly :d

You'd think so, but... I still have nightmares about encoding a horror movie for a friend (back in Xvid days) and it had a long pan shot in a library room with light in the centre and darkness on peripheral. The general blurriness of Xvid would blend the thin vertical shadows that formed between books lined up on shelves into each other, horrible effect. This seems to bring it back and motion is likely to conceal more ridges and blur them into smooth light blue, if anything.

It is the greater natural sharpness of h264 (or at least x264 implementation) that brought me over. Give me blocks over blur any day. (it always puzzled me that doom9's codec comparisons always said that sharpness was the same, when every single xvid vs x264 shot screamed vast sharpness difference to my eyes)

[Dark Shikari]

lexor i doub't you spot the difference in motion as from the position it looks like a high motion scene where a cut is following shortly :d
Yeah, here's the full clips:

AQ (http://tjhsst.edu/~jgarrett/AQ.mkv)

No AQ (http://tjhsst.edu/~jgarrett/NoAQ.mkv)

[CruNcher]

It is the greater natural sharpness of h264 (or at least x264 implementation)
x264 as a H.264 implementation doesn't give the highest sharpest Look & Feel yet.
Mainconcept is a tad sharper but Ateme is WOW ;) i really can say that for me Ateme is the XviD of H.264 in Look & Feel without all the old ASP problems ;) it would be really strange if XviD 2.0 AVC would also Look & Feel like x264 does now.

[woah!]

can we see some of the Ateme results then please as i dont think their encoder is available to try is it.

i keep hearing about how good it is but never see any proof as such..

[CruNcher]

woah look in the low bitrate thread i posted some results their http://forum.doom9.org/showthread.php?t=129200&page=3


can we see some of the Ateme results then please as i dont think their encoder is available to try is it.


actually it is with Nero Recode 2/3 (Atemes consumer Encoder is working in their with High Profile support) :)

@Dark Shikari
i tested now with useing --aq-strength 0.1 and --subme 2 but the top line problem is still their hmm,
but yeah that padding is also visible without any AQ @ all but only very slightly (actually i didn't really realized it's there before), it seems the AQ amplifies (sharpens - makes it more perceptable) it.
visualy it seems that the higher --aq-strenght the more sharper everything get's to high and the result oversharpens and artifacting occours :) (this can indeed change the Digital Cinema Look & Feel of x264 to a much more sharper one if correctly used (due to the better quantization it seems), but this padding problem in the 1st macroblock colum is definately not in the source @ all.

[DeathTheSheep]

Which version of Ateme are you referring to above ("Ateme is WOW")? The one currently in Nero or the old betas?

[*.mp4 guy]

The old Betas, they are... much less restricted then what is in nero. Generally when someone says Ateme, I think it is safe to assume they are talking about Ateme' Beta 3 encoder cli, usually when people are talking about the version in recode, they say nero, in my experience.

[CruNcher]

The old Betas, they are... much less restricted then what is in nero. Generally when someone says Ateme, I think it is safe to assume they are talking about Ateme' Beta 3 encoder cli, usually when people are talking about the version in recode, they say nero, in my experience.

Yep but that has nothing todo with the Look & Feel both have the same only precission and end quality is affected but the core with all it's optimizations and tuneings is the same, else all my bug reports and improvement sugestions would have been for the trashbin ;)

[IgorC]

More and more people saying about their preferement to Ateme beta encoder.
Encoder has lower ssim and opsnr results (not much) but it's better visually.Blocking,banding free and higher level of details.
I suspect Ateme used more realistic and efficient metrics than MC's amd x264's ssim optimizations. cwssim?

[Dark Shikari]

More and more people saying about their preferement to Ateme beta encoder.
Encoder has lower ssim and opsnr results but it's visually.
Blocking,banding free and higher level of details.
I suspect Ateme used more realistic and efficient metrics than MC's amd x264's ssim optimizations. cwssim?
x264 is PSNR, not SSIM optimized in my experience--one can easily tweak the quantizers to raise SSIM more (my original AQ algorithm).

Anyways can we seriously get this thread back on topic?

[Fishman0919]

I noticed some loss of detail in the edge of the table. Noticed it a little when playing the clips... but much more so in the stills.

AQ = http://img217.imageshack.us/my.php?image=aqvt7.png

No AQ = http://img170.imageshack.us/my.php?image=noaqhb1.png

[Dark Shikari]

I noticed some loss of detail in the edge of the table. Noticed it a little when playing the clips... but much more so in the stills.

AQ = http://img217.imageshack.us/my.php?image=aqvt7.png

No AQ = http://img170.imageshack.us/my.php?image=noaqhb1.png
Ah, I see what's happening... that area doesn't actually have much detail, but it has a HUGE standard deviation due to the black/white contrast. So the QP is lowered a lot when it shouldn't be.

I'll see if there's a way to fix that, nice catch!

[Daodan]

I think that image has too strong aq in it. With dark movies (riddick could be considered as such) strong aq just takes too much bits from the few bright areas. Aq 0.5 should be a max in my opinion for those.
As to get back to my tests, they were accurate before. There is clear bframe quality boost which makes the image more stable (otherwise there's a slight flickering in background grain instead of constant movement). So at least for that movies, there's a clear improvement. As said before, wouldn't mind a real build with it:thanks:.
offtopic: my tests also show --me imh +subme 8 combination is a downgrade rather then upgrade in fine grain preservation. It gets slightly more ringy and inconsistent in the frame (so not equally distributed).

[Sagittaire]

Ah, I see what's happening... that area doesn't actually have much detail, but it has a HUGE standard deviation due to the black/white contrast. So the QP is lowered a lot when it shouldn't be.

I'll see if there's a way to fix that, nice catch!

possible to write a complete patch for your AQ ... ?

[CruNcher]

I think that image has too strong aq in it. With dark movies (riddick could be considered as such) strong aq just takes too much bits from the few bright areas. Aq 0.5 should be a max in my opinion for those.
As to get back to my tests, they were accurate before. There is clear bframe quality boost which makes the image more stable (otherwise there's a slight flickering in background grain instead of constant movement). So at least for that movies, there's a clear improvement. As said before, wouldn't mind a real build with it:thanks:.
offtopic: my tests also show --me imh +subme 8 combination is a downgrade rather then upgrade in fine grain preservation. It gets slightly more ringy and inconsistent in the frame (so not equally distributed).

Yep that's a problem with the nature of b-frames they are sometimes to high compressed and then if the motion estimation has problems grain/noise gets destructed in them and these flickering happens, with AQ you have two effects that enhance this situation one is lower quantization @ p-frames so the next b-frames don't get so high quantinized as without masking :) and the other ofcourse the direct intereference of the partitions in the b-frame getting better balanced quantization, both is a plus and both enhances the visual quality with b-frames as finer grain/noise doesn't get blocked out as it does most times without masking.
The blurring of b-frames isn't really the problem @ low bitrates the blocking in b-frames is and especialy the more b-frames you use the more problematic it gets (referenced and amplified in the next b-frame) and mis predicted frames appear then creating somekind of warping effect (due to the mis predictions), visualy im trying to tweak this behaviour @ the moment in another implementation, and i think this is really a thing people should concentrate on improving in x264 as well (this is only really happening @ very low bitrates much lower then Dark Shikaris example tough, if you wan't to preserve a lossy grain layer).
The magic is if you can prevent such stuff from happening @ very low bitrates it should scale up with high ones, so you never will get those problems @ high bitrates and the grain/noise layer should get transparent faster. The problem is really low bitrate tuneing and here we should do more, tuneing x264 for ultra low bitrates in terms of visual stability as i said, those improvements scale up and won't lower the quality @ higher bitrates :)

So what i actually wanted to say is the more realisticly (without heavy visual problems) you can preserve the grain/noise layer @ low bitrates the sooner it get's transparent @ high bitrates (seems logic doesn't it) and that's where everything RC,ME,AQ,B-Frames should be tuned for :)

[Manao]

CrunCher : you are allowed to use dots & linebreaks, you know. I can't hold my breath long enough to understand a 10+ lines sentence...

[CruNcher]

Sorry i edited it ;)

[Gabriel_Bouvigne]

Yep that's a problem with the nature of b-frames they are sometimes to high compressed and then if the motion estimation has problems grain/noise gets destructed in them and these flickering happens, with AQ you have two effects that enhance this situation one is lower quantization @ p-frames so the next b-frames don't get so high quantinized as without masking :) and the other ofcourse the direct intereference of the partitions in the b-frame getting better balanced quantization, both is a plus and both enhances the visual quality with b-frames as finer grain/noise doesn't get blocked out as it does most times without masking.

What about a bit more editing?
Sample:
Yep that's a problem. With the nature of b-frames, they are sometimes too high compressed and then if the motion estimation has problems, then grain/noise gets destructed in them, and these flickering happens.
With AQ you have two effects that enhance this situation:
*one is lower quantization at p-frames, so the next b-frames don't get so high quantinized as without masking :)
*the other of course the direct intereference of the partitions in the b-frame getting better balanced quantization.
Both is a plus and both enhances the visual quality with b-frames, as finer grain/noise doesn't get blocked out as it does most times without masking.

[DeathTheSheep]

Woah, you two should get together sometime for an editing smash! (or CruNch, if you'll excuse the pun. ;))

Anyway, I just wanted to ask how this patch would be expected to perform on ultra-low-bitrate sources. Dark Shikari mentioned earlier that it wouldn't have much effect (or negative effect) on bitrates where "there will be blocking anyway." Is this still the case? I know that x264, even at some of the lowest streaming bitrates, blocks in dark areas and not on high-contrast bright ones...

[Dark Shikari]

Woah, you two should get together sometime for an editing smash! (or CruNch, if you'll excuse the pun. ;))

Anyway, I just wanted to ask how this patch would be expected to perform on ultra-low-bitrate sources. Dark Shikari mentioned earlier that it wouldn't have much effect (or negative effect) on bitrates where "there will be blocking anyway." Is this still the case? I know that x264, even at some of the lowest streaming bitrates, blocks in dark areas and not on high-contrast bright ones...At ultra-low bitrates, there are going to have to be blocks somewhere. Better to have them in dark areas than to totally block up the light areas in an attempt to save bits, IMO.

Very low bitrates are where all the PSNR/SSIM optimizations begin to shine--while at high bitrates there's enough bits to do AQ and such to vastly improve visual quality, at low bitrates every last bit of efficiency is needed.

[Dark Shikari]

Some updates on the AQ improvements.

I found that one major problem with my AQ algorithm is that on edges between flat areas and complex areas it can sometimes raise the quantizer considerably (especially if its a very pronounced edge, such as between black and video on a letterboxed video). This results in artifacting. What I did is have the standard-deviation function take the standard deviation of all 4 8x8 blocks in the macroblock separately and pick the flattest one. That one was used to decide the actual quantizer.

This gave pronounced improvements, eliminating loads of edge artifacting without any real negative consequences.

[Dark Shikari]

I did some encoding and have three encodes ready that were made with the same settings:

Latest Version of New Adaptive Quantization (http://tjhsst.edu/~jgarrett/AQ.mkv)

Old Adaptive Quantization (http://tjhsst.edu/~jgarrett/OldAQ.mkv)

No Adaptive Quantization (http://tjhsst.edu/~jgarrett/NoAQ.mkv)

Original video (raw MPEG-2 in MKV) (http://tjhsst.edu/~jgarrett/Raw%20Video.mkv)

Try them out and comment--I want to find all corner cases where the new AQ fails and fix them.

And for those of you whining about how much sharper Ateme is... try my AQ :p New executable will be up later.

The effect of my AQ at full-strength on metrics is approximately -2.2db PSNR and -7% SSIM on the source used here. It also results in a much lower percentage of skipped macroblocks.

[Dark Shikari]

I've replaced all the videos with 30-FPS ones since the 24 FPS ones were improperly IVTC'd. I also uploaded the clip of the original MPEG-2 for comparison.

It also appears that the grain retention isn't as good with this method because there are a number of places that its wasting bits where it shouldn't be. More updates to come.

[Daodan]

I only took a look at old vs new aq, and have to say there is something wrong with it. Probably Much too strong.
During the logo, with the rising sun, it blocks to hell (no wonder, with a qp of 37), then later on with the universal logo, the edges are really blurred and ringy. Unnaceptable for these bitrates (which are rather highish, frame qp ~16-17, should be close to transparent). Then moving later on to the real movie, again, some strong details are lost, while the benefit in the background is quite small.
So I'd say to try and put a high limit to the qp a mb can reach (based on average frame pq or smth), to stop those really visible blocks that happen in frames with too much black (the rising sun frames).

[Dark Shikari]

I only took a look at old vs new aq, and have to say there is something wrong with it. Probably Much too strong.
During the logo, with the rising sun, it blocks to hell (no wonder, with a qp of 37), then later on with the universal logo, the edges are really blurred and ringy. Unnaceptable for these bitrates (which are rather highish, frame qp ~16-17, should be close to transparent). Then moving later on to the real movie, again, some strong details are lost, while the benefit in the background is quite small.
So I'd say to try and put a high limit to the qp a mb can reach (based on average frame pq or smth), to stop those really visible blocks that happen in frames with too much black (the rising sun frames).Fortunately, the new AQ algorithm has been completely rewritten since I posted those samples.

It is now DCT-based, much like the original, though its algorithm is a lot more complicated and should work in many situations when the old AQ did not. It also takes into account CQMs!

It also doesn't kill quality nearly as much in certain blocks... plus its faster.

[cantonesejim]

Any news? this is very interesting to me.

[bob0r]

x264.697.fast-ref-search.01.aq-brdo.exe (http://mirror05.x264.nl/Dark/force.php?file=./x264.697.fast-ref-search.01.aq-brdo.exe) please test if AQ still works properly with this revision + added fast rev search patch.

@Dark Shikari
The patches seem to work fine with revision 697.
SSE3 shows up in all combinations of compiling, patch order that is.

[Dark Shikari]

x264.697.fast-ref-search.01.aq-brdo.exe (http://mirror05.x264.nl/Dark/force.php?file=./x264.697.fast-ref-search.01.aq-brdo.exe) please test if AQ still works properly with this revision + added fast rev search patch.

@Dark Shikari
The patches seem to work fine with revision 697.
SSE3 shows up in all combinations of compiling, patch order that is.
I'll test it when I get back to working on my AQ. Hopefully that'll be sooner rather than later.

[Razorholt]

x264.697.fast-ref-search.01.aq-brdo.exe (http://mirror05.x264.nl/Dark/force.php?file=./x264.697.fast-ref-search.01.aq-brdo.exe) please test if AQ still works properly with this revision + added fast rev search patch.

@Dark Shikari
The patches seem to work fine with revision 697.
SSE3 shows up in all combinations of compiling, patch order that is.

File doesn't exist :(

[bob0r]

File doesn't exist :(

Doh!
Fixed. :sly:

[ACrowley]

Somybody made AQ Tests with this Build now ?

I cant see any problems at the Moment

[Razorholt]

x264.697.fast-ref-search.01.aq-brdo.exe (http://mirror05.x264.nl/Dark/force.php?file=./x264.697.fast-ref-search.01.aq-brdo.exe) please test if AQ still works properly with this revision + added fast rev search patch.

@Dark Shikari
The patches seem to work fine with revision 697.
SSE3 shows up in all combinations of compiling, patch order that is.

I think it works, yes. Thank you for adding the AQ patch!

[Gromozeka]

But where your AQ added in standart build x264? :)

[J_Darnley]

No AQ patch has been included into the x264 SVN code yet you you can't get it in a 'standard' build yet. Dark Shikari has said that the AQ algorithm isn't good enough yet to be used much, he also stated that it was rewritten a few posts above your's, here (http://forum.doom9.org/showthread.php?p=1049016#post1049016). You can find the old AQ algorithm in Cef's builds linked to in the "daily builds" sticky.

[Gromozeka]

But films with AQ gives result better than films without AQ! Why to not add it in the standard version? A lot of time is necessary for improvement?

[Sharktooth]

false. AQ is usefull ONLY in dark/blue areas when there are visible blocks.
AQ actually LOWERS the image quality to avoid some blocking in certain areas.
Also the algo is inefficient coz it's not finetuned and it "exchange" bits even where it's not necessary.

[burfadel]

false. AQ is usefull ONLY in dark/blue areas when there are visible blocks.
AQ actually LOWERS the image quality to avoid some blocking in certain areas.
Also the algo is inefficient coz it's not finetuned and it "exchange" bits even where it's not necessary.

Does it do this with constant quality mode as well? I could understand it for constant/average bitrate to possibly lower the quality in other areas, but with constant quality wouldn't it leave those areas untouched and the overall affect just an improvement in the dark/blue areas?

If my assumption is correct, wouldn't it be beneficial to include it, but only have it enabled in CRF mode? Maybe a screen output during the encoding process should say 'CRF selected, AQ enabled' or something along those lines when enabled in CRF mode. When trying to enable in CBR/ABR mode, an output of 'CBR/ABR selected, AQ not valid for this mode'...

Just a thought :)

[Sharktooth]

i presume the frame quantizer will always be the same since AQ affects only the currently processed frame.

[Gromozeka]

I code in CRF mode and quality the image is equated to quality films without AQ, but with overestimated bitrate/
May be it is meaningful include official assembly but that AQ worked only in a mode crf?
Sorry? my rnglish is not good

[Unearthly]

I code in CRF mode and quality the image is equated to quality films without AQ, but with overestimated bitrate/
May be it is meaningful include official assembly but that AQ worked only in a mode crf?
Sorry? my rnglish is not good

Why does it need to be official? There are plenty of public builds that contain the AQ patch.

[Gromozeka]

Because if AQ it will be added in official Build she it will be added in unnoficial this:
http://www12.atwiki.jp/lunatilia/pages/70.html
And quality it will be better

[Sharktooth]

you didnt read what i wrote.

[Gromozeka]

I read your words, but my tests have shown that in a crf mode it gives the best result
See this:
http://forum.doom9.org/showthread.php?t=132175

[Sharktooth]

I've already read that and still my opinion about AQ is the same. It's an empirical "quick and dirty" workaround for some x264 blocking behaviour with "blue sky".
The actual AQ is not always needed. It also lowers the overall image quality if enabled and not needed... expecially with wrong settings.
As it is, it will never be commited to the x264 svn.
Maybe Haali or Dark Shikari can elaborate something useful that would be worth being "officialized".

[burfadel]

How does AQ lower the image quality in CRF mode? I thought it would only lower it in CBR/ABR modes? Thats assuming that the AQ patch is additive quality wise - that is, have a base CRF of say, 22, then the AQ to do its think on flat areas but still have the quality of all other areas at 22. By what you are saying, even in CRF mode it increases the quality in the flat areas, and decreases it in other areas to keep the quality as close to 22 (or whatever CRF selected)?

Hope that makes sense!

[Sharktooth]

AQ moves bits within a frame... it's not temporal...
IIRC once the frame quantizer is determined it subtracts bits from almost all blocks (partitions) to reassign them into dark/blue areas.
so, yes the average quantizer will be almost the same, but the blocks (partitions) are quantized differently...

[desta]

On a personal-use note, even an abr/file sized based encode will yield better visual results with AQ than one without, regardless of the amount of dark/blue areas.

I'm not talking quants here obviously - if you take AQ out of the encode then clearly the metrics will look better on paper than an encode with. But when used properly (ie, not overdone and for the sake of it) AQ will in most cases help to give an overall equal look of quality to the whole image.

I know I'm probably opening myself up to quite a bit of criticism by saying that, which is why I said "personal use". It's always going to be down to what each individual prefers.

[Dark Shikari]

so, yes the average quantizer will be almost the same, but the blocks (partitions) are quantized differently...Wrong... on CRF/CQP mode, AQ only lowers quants; it does not raise them.

[Sharktooth]

x264 encodes all blocks at the same quant.
what AQ does is exactly the opposite. it differentiate quants (within a frame) by "stealing" bits from non-blue/non-dark blocks to lower the quants of blue/dark blocks.
so basically it's really the opposite of what you're saying...

[Dark Shikari]

x264 encodes all blocks at the same quant.
what AQ does is exactly the opposite. it differentiate quants (within a frame) by "stealing" bits from non-blue/non-dark blocks to lower the quants of blue/dark blocks.
so basically it's really the opposite of what you're saying...But on CQP/CRF mode, it doesn't "steal bits"; it adds bits to certain blocks by lowering their quantizer, while not changing the quantizer of the blocks that aren't affected. As a result, the total bitrate rises.

There is nothing in the AQ code that raises quantizers.

[R3Z]

I dont see the issue in including Darks AQ method into the official builds. Its not like the option has to be turned on by default.

It might be a "quick and dirty" option, but its better than none at all.

[Dark Shikari]

I dont see the issue in including Darks AQ method into the official builds. Its not like the option has to be turned on by default.

It might be a "quick and dirty" option, but its better than none at all.My current method, involving RDO and bit cost, works pretty decently at the moment but I would much rather wait and get it implemented the way it should be (as part of ratecontrol) before making it public.

[burfadel]

My current method, involving RDO and bit cost, works pretty decently at the moment but I would much rather wait and get it implemented the way it should be (as part of ratecontrol) before making it public.

Ah ok! that fair enough!

Its also good to hear that it doesn't raise quantisers in CRF mode, I see other people were wondering that too. It makes perfect sense in ABR/CBR mode for it to do that, which would be counter-productive which is why I suggested it for CRF mode only!

Keep up the excellent work!

[Sharktooth]

i obviously didnt have a look at the Dark Shikari latest code :p