New Custom Quantization Matrices - EQM AVC Series. Last update: EQM AVC-HR 2005-08-01

[Sharktooth]

EQM AVC-HR rev.1
This matrix was made for medium-high bitrate backups with High-Profile AVC encoders that support custom matrices and 8x8 discrete cosine transform. It also reduces blocking.

Usage:
- high quality backups of DVD material at full resolution (anamorphic) and bitrates of at least 1300-1400kbps (for average motion and not noisy/old movies).
- non anamorphic/vertically resized encodes at bitrates of at least 1000kbps.

Standard version (JM, x264 & Encavc beta2-2 compatible):

INTRA4X4_LUMA =
 6, 9,13,19,
 9,14,20,27,
13,20,28,35,
19,27,35,42

INTRA4X4_CHROMAU =
 6,10,15,20,
10,16,21,27,
15,21,28,33,
20,27,33,42

INTRA4X4_CHROMAV =
 6,10,15,20,
10,16,21,27,
15,21,28,33,
20,27,33,42	

INTER4X4_LUMA =
 8,11,15,20,
11,16,21,27,
15,21,28,35,
20,27,35,42	

INTER4X4_CHROMAU =
 8,11,16,21,
11,17,22,27,
16,22,28,35,
21,27,35,42	

INTER4X4_CHROMAV =
 8,11,16,21,
11,17,22,27,
16,22,28,35,
21,27,35,42	

INTRA8X8_LUMA =
 6, 7, 8,10,12,14,16,18,
 7, 9,11,13,15,16,18,20,
 8,11,14,16,17,19,21,22,
10,13,16,18,20,22,24,26,
12,15,17,20,23,25,28,30,
14,16,19,22,25,29,34,38,
16,18,21,24,28,34,46,52,
18,20,22,26,30,38,52,72

INTER8X8_LUMA =
 8, 9,10,12,14,16,19,21,
 9,11,13,15,17,19,21,23,
10,13,16,18,20,22,23,25,
12,15,18,21,23,24,26,32,
14,17,20,23,25,27,33,40,
16,19,22,24,27,34,41,52,
19,21,23,26,33,41,53,64,
21,23,25,32,40,52,64,80

Download: EQM AVC-HR

EDIT: fixed to comply with JVT standard.

[Sharktooth]

*** Just a bump to make you know the HR matrix is out of beta.
More matrices will follow as soon as i finish internal testing.

thanks for your tests

[skal]

Hello Sharktooth,
could you elaborate a little on how you deduce your matrices?
("with my eyes" is ok as an answer;)
-Skal

[trbarry]

Hi Sharktooth -

When I wanted consistantly high quality and didn't care greatly about a precise bit rate I would often use 1-pass Xvid with a constant quant of 2, but with 2 b-frames, at q=3. Varyingly this would also be with one of your custom matrices. Other options to taste but mostly using more CPU to get better compression efficiency in a single pass. I tend to store things on external USB drives so a precise size is not that important. While I dislike wasting space it's often not worth another pass just to hit a arbitrary size target if quality demands more anyway.

Can you off hand recommend any sort of fairly high bit rate X264 equivalent set of fixed quant options?

Sorry if this has been covered before but I've been away for awhile and a search didn't turn up anything with these priorities. I don't yet have much feel for what I can get out of X264 (or Nero?) with fixed quants but you seem to work with these things a bit.

- Tom

PS - These are mostly HD caps, at 720p or 544p though I don't know if it matters here.

[Sharktooth]

Originally Posted by skal

Hello Sharktooth,
could you elaborate a little on how you deduce your matrices?
("with my eyes" is ok as an answer
-Skal

well with h264 i had to re-elaborate some of my theories on human eye perception but generally i "draw" a matrix around my initial idea, then i do a set of tests with a set (10 or more) of matrices (usually those are scaled from the "master" matrix with different factors and scaling formula) on 5 different sources.
Then i analyze the source sequencies and for every sequence i assign a score.
After the analysis phase i sum the scores and choose the matrix with the higest one.
This method was pretty fast for xvid, but since AVC is more complex it requires a lot more time...

Originally Posted by trbarry

Hi Sharktooth -

When I wanted consistantly high quality and didn't care greatly about a precise bit rate I would often use 1-pass Xvid with a constant quant of 2, but with 2 b-frames, at q=3. Varyingly this would also be with one of your custom matrices. Other options to taste but mostly using more CPU to get better compression efficiency in a single pass. I tend to store things on external USB drives so a precise size is not that important. While I dislike wasting space it's often not worth another pass just to hit a arbitrary size target if quality demands more anyway.

Can you off hand recom

Well, you can try with quants from 20 to 24, and you should have the "equivalent" of q2 (or q2+Bs) for xvid (depending on the CQM you used).mend any sort of fairly high bit rate X264 equivalent set of fixed quant options?

Sorry if this has been covered before but I've been away for awhile and a search didn't turn up anything with these priorities. I don't yet have much feel for what I can get out of X264 (or Nero?) with fixed quants but you seem to work with these things a bit.

- Tom

PS - These are mostly HD caps, at 720p or 544p though I don't know if it matters here.

Well, you can try with quants from 20 to 24, and you should have the "equivalent" of q2 (or q2+Bs) for xvid (depending on the CQM you used).

[trbarry]

Thx. That give me a useful place to start.

- Tom

[jellysandwich]

Is there a way to use custom quant matrices
with x264 vfw?

Edit: Also, is there one similar to MPEG for Xvid?
I like the sharpness that MPEG gives...

js

[yaz]

Originally Posted by Sharktooth

... you can try with quants from 20 to 24, and you should have the "equivalent" of q2 (or q2+Bs) for xvid (depending on the CQM you used).

according to my (very incomplete) tests x264 cq equivalent to xvid q2 is somewhat lower. it's somewhere q16-18. (xvid matrix: mpeg, x264 matrix: default, bvop(xvid):1/1/0)

Originally Posted by jellysandwich

is there one similar to MPEG for Xvid?
I like the sharpness that MPEG gives...

i would love that too. it's good to see that i'm not the very last fan (nerd?) of that matrix

the bests
y

[Sharktooth]

MPEG matrix is different from (and more bitrate hungry than) h263. The values i suggested are intender for a h263-like quant2 compression, also i specified those values depends on custom matrices (in both xvid and x264).

[TheBashar]

Hi Sharktooth,

I understand that your AVC matrices are intended for bitrates of around 1350kbps for 720x430 "average motion" movies. From following some of your XviD matrices development, I believe I recall that the matrices are "tuned" for the target bitrate/resolution ratio and perform sub-optimally on lower bitrate/resolution ratios.

Getting (slowly) to my question... will the matrices perform sub-optimally in regions of an encode where due to a default curve compression (qcomp = 0.6 in x264) bitrates are reduced because of slow/no motion detection? For instance, I may have an encode which averages 1350kbps but a particular scene may have only 800 kbps allocated due to it being a static scene. Will these matrices perform poorly in such regions solely becuase of the low bitrate or will they still perform as expected since in fact there is little or no motion?

The reason I ask is because I am considering unbridling the qcomp by significantly raising the ratetol from its default of 1.0.

Thanks for your time!

[Sharktooth]

Matrices are tuned for a certain compressibility. In regions where the codec assigns less bit, the compressibility is usually higher. So, in static scenes, even if the bitrate gets lower, it doesnt mean the codec is using higher quantizers. It's usually the opposite... However when i made AVC-HR i took into account the possibility to set the bitrate variability from "1" to "inf" and the HR matrix should behave correctly (at least in my tests it proved to do it).
It's all a matter of coefficients scaling... for example if the scaling is too "harsh" it will produce artifacts at lower bitrates (too much difference between quants).

[TheBashar]

Hi Sharktooth,

I really like your CQM. When I was trying to determing a good bitrate before I used your CQM, I had trouble with blocky dark backgrounds even at what I thought were too high bitrates. When I finally switched to your CQM, I found the blockyness was dramatically reduced. I'm guessing your CQM gives more bitrate in the dark color range than the standard one does.

So, all was well in the world. I have always been frustrated by my encodes being a tad too dark though. I recently stumbled onto this 9-04 thread about ColorMatrix, which corrects that (or the real problem rather). I tried it and in fact it really improved things. So, I started a reencode. Unfortunately, after my first reencode, I found the blockyness was back. Now, there are other differences as well: x264 cli rev 291B instead of 287A, substituted limitedsharpen instead of xsharpen.

I've had a hard time finding an old 287a x264 rrev to test with, so I thought I post here and ask if you think the ColorMatrix change could be responsible. Is there a chance that you tuned your CQM for the incorrect color spectrum that ColorMatrix is meant to correct? Not accusing or anything, just asking so I can narrow down the permutations I need to test.

Thanks!

[Sharktooth]

Well, limitedsharpen may be the cause. Sharpen filters usually lower the source compressibility in a significative manner. In this situation a lower bitrate matrix is needed but you can try rising the deblocking filter threshold by 1 or 2 to help compensate the compressibility loss.
However i dont use colormatrix coz i never experienced darkened or desaturated encodes so IMHO it's not a "tuning" problem.

[Chainmax]

This matrix will come in handy on my next encodes. I'm doing AVC backups of some high motion cartoon captures @1100kbps and 640x480 resolution, and with this matrix I might even be able to disable deblocking. Once I solve some issues with my current filterchain (SSIQ and aWarpSharp are causing green splotches on the sides of the picture) I'll try this and report back.

Sharktooth, how do you expect customs matrices to affect playback compatibility on future AVC capable standalones?

[Sharktooth]

Disabling deblocking is always a bad idea unless you're near saturation.
However, custom matrices are part of high profile and it will be played back only if the SAP will support it.

[Daodan]

Nice thing this matrix, exactly fit for my needs. One question though. I started encoding a movie (140 min movie) with 1800 kbps that I just finished encoding without a custom matrix. The 'problem' is, that the encode is 17 % faster now then before (it's only 30% completed now but the speed increase is obvious). I use the same settings as before so I could compare the results. This is something unexpected I would say.
Is it normal or did my computer start taking drugs?
Well, it just finished and it took 12 hours instead of 17.

[Manao]

Disabling deblocking is always a bad idea unless you're near saturation.

Deblocking gets disable automatically a long time before reaching saturation ( saturation for h264 is around ~20 mbits for D1, it can't be reached ).

Supporting CQM isn't complicated at all. It doesn't slow down the encoding process, so there's no objective reason not to support it. But now, I'm not a chip maker...

Edit : encoding with CQM doesn't slow down the encoding process.

[Kopernikus]

x264 has different optimized routines for quantizing. if a cqm has low entries a slightly slower routine ist used to prevent overflow.

[Manao]

But only if the quantizer gets really low. And low entries means higher bitrates --> high quantizers at same bitrates, so in the overall it compensates one another ( well, almost at least )

[Sharktooth]

Originally Posted by Manao

Deblocking gets disable automatically a long time before reaching saturation ( saturation for h264 is around ~20 mbits for D1, it can't be reached ).

sorry, wrong term... i meant "close to transparency" or a sufficient quality/bitrate to have blocks go away...