collection of available AVC custom quant matrices

[bond]

1) jvt
description: the jvt matrix, used as the default custom matrix by the avc reference software (comparable to the "mpeg" matrix of mpeg-4 asp), uses the same values in all intra4x4 matrices and the same values in all inter4x4 matrices:

#jvt

INTRA4X4_LUMA =
 6,13,20,28,
13,20,28,32,
20,28,32,37,
28,32,37,42

INTRA4X4_CHROMAU =
 6,13,20,28,
13,20,28,32,
20,28,32,37,
28,32,37,42

INTRA4X4_CHROMAV =
 6,13,20,28,
13,20,28,32,
20,28,32,37,
28,32,37,42

INTER4X4_LUMA =
10,14,20,24,
14,20,24,27,
20,24,27,30,
24,27,30,34

INTER4X4_CHROMAU =
10,14,20,24,
14,20,24,27,
20,24,27,30,
24,27,30,34

INTER4X4_CHROMAV =
10,14,20,24,
14,20,24,27,
20,24,27,30,
24,27,30,34

INTRA8X8_LUMA =
 6,10,13,16,18,23,25,27,
10,11,16,18,23,25,27,29,
13,16,18,23,25,27,29,31,
16,18,23,25,27,29,31,33,
18,23,25,27,29,31,33,36,
23,25,27,29,31,33,36,38,
25,27,29,31,33,36,38,40,
27,29,31,33,36,38,40,42

INTER8X8_LUMA =
 9,13,15,17,19,21,22,24,
13,13,17,19,21,22,24,25,
15,17,19,21,22,24,25,27,
17,19,21,22,24,25,27,28,
19,21,22,24,25,27,28,30,
21,22,24,25,27,28,30,32,
22,24,25,27,28,30,32,33,
24,25,27,28,30,32,33,35

when you want to use the jvt cqm with x264 you can do that via the following commandline:

--cqm jvt

2) q_matrix
description: the "q_matrix", coming with the avc reference software, uses the same values in all intra4x4 matrices and the same values in all inter4x4 matrices:

#q_matrix

INTRA4X4_LUMA =
 6,12,19,26,
12,19,26,31,
19,26,31,35,
26,31,35,39

INTRA4X4_CHROMAU =
 6,12,19,26,
12,19,26,31,
19,26,31,35,
26,31,35,39

INTRA4X4_CHROMAV =
 6,12,19,26,
12,19,26,31,
19,26,31,35,
26,31,35,39

INTER4X4_LUMA =
 9,13,18,21,
13,18,21,24,
18,21,24,27,
21,24,27,30

INTER4X4_CHROMAU =
 9,13,18,21,
13,18,21,24,
18,21,24,27,
21,24,27,30

INTER4X4_CHROMAV =
 9,13,18,21,
13,18,21,24,
18,21,24,27,
21,24,27,30

INTRA8X8_LUMA =
 6,10,13,16,19,24,26,28,
10,12,16,19,24,26,28,31,
13,16,19,24,26,28,31,33,
16,19,24,26,28,31,33,35,
19,24,26,28,31,33,35,37,
24,26,28,31,33,35,37,39,
26,28,31,33,35,37,39,42,
28,31,33,35,37,39,42,44

INTER8X8_LUMA =
 9,12,14,16,18,19,21,22,
12,13,16,18,19,21,22,24,
14,16,18,19,21,22,24,25,
16,18,19,21,22,24,25,27,
18,19,21,22,24,25,27,28,
19,21,22,24,25,27,28,30,
21,22,24,25,27,28,30,31,
22,24,25,27,28,30,31,33

when you want to use the q_matrix cqm with x264 you can do that via the following commandline:

--cqm4i 6,12,19,26,12,19,26,31,19,26,31,35,26,31,35,39 --cqm4p 9,13,18,21,13,18,21,24,18,21,24,27,21,24,27,30 --cqm8i 6,10,13,16,19,24,26,28,10,12,16,19,24,26,28,31,13,16,19,24,26,28,31,33,16,19,24,26,28,31,33,35,19,24,26,28,31,33,35,37,24,26,28,31,33,35,37,39,26,28,31,33,35,37,39,42,28,31,33,35,37,39,42,44 --cqm8p 9,12,14,16,18,19,21,22,12,13,16,18,19,21,22,24,14,16,18,19,21,22,24,25,16,18,19,21,22,24,25,27,18,19,21,22,24,25,27,28,19,21,22,24,25,27,28,30,21,22,24,25,27,28,30,31,22,24,25,27,28,30,31,33

3) q_matrix2
description: the "q_matrix2", coming with the avc reference software, uses the same values in all intra4x4 matrices and the same values in all inter4x4 matrices:

#q_matrix2

INTRA4X4_LUMA =
 7,16,22,24,
16,22,24,28,
18,22,27,33,
22,24,32,47

INTRA4X4_CHROMAU =
 7,16,22,24,
16,22,24,28,
18,22,27,33,
22,24,32,47

INTRA4X4_CHROMAV =
 7,16,22,24,
16,22,24,28,
18,22,27,33,
22,24,32,47

INTER4X4_LUMA =
13,15,17,18,
15,17,18,20,
17,18,21,22,
18,20,22,25

INTER4X4_CHROMAU =
13,15,17,18,
15,17,18,20,
17,18,21,22,
18,20,22,25

INTER4X4_CHROMAV =
13,15,17,18,
15,17,18,20,
17,18,21,22,
18,20,22,25

INTRA8X8_LUMA =
 7,13,16,18,22,22,24,28,
13,13,18,20,22,24,28,31,
16,18,22,22,24,28,28,32,
18,18,22,22,24,28,31,33,
18,22,22,24,27,29,33,40,
22,22,24,27,29,33,40,48,
22,22,24,28,32,38,47,57,
22,24,29,32,38,47,57,69

INTER8X8_LUMA =
13,14,15,16,17,17,18,19,
14,15,16,17,17,18,19,20,
15,16,17,17,18,19,20,21,
16,17,17,18,19,20,21,22,
17,17,18,19,21,22,22,23,
17,18,19,20,22,22,23,25,
18,19,20,22,22,23,25,26,
19,20,21,22,23,25,26,27

when you want to use the q_matrix2 cqm with x264 you can do that via the following commandline:

--cqm4i 7,16,22,24,16,22,24,28,18,22,27,33,22,24,32,47 --cqm4p 13,15,17,18,15,17,18,20,17,18,21,22,18,20,22,25 --cqm8i 7,13,16,18,22,22,24,28,13,13,18,20,22,24,28,31,16,18,22,22,24,28,28,32,18,18,22,22,24,28,31,33,18,22,22,24,27,29,33,40,22,22,24,27,29,33,40,48,22,22,24,28,32,38,47,57,22,24,29,32,38,47,57,69 --cqm8p 13,14,15,16,17,17,18,19,14,15,16,17,17,18,19,20,15,16,17,17,18,19,20,21,16,17,17,18,19,20,21,22,17,17,18,19,21,22,22,23,17,18,19,20,22,22,23,25,18,19,20,22,22,23,25,26,19,20,21,22,23,25,26,27

4) flat 16
description: not really a custom matrix, as its the matrix thats used when no custom quantisation matrix is used, but still listed for completeness:

#flat 16

INTRA4X4_LUMA =
16,16,16,16,
16,16,16,16,
16,16,16,16,
16,16,16,16

INTRA4X4_CHROMAU =
16,16,16,16,
16,16,16,16,
16,16,16,16,
16,16,16,16

INTRA4X4_CHROMAV =
16,16,16,16,
16,16,16,16,
16,16,16,16,
16,16,16,16

INTER4X4_LUMA =
16,16,16,16,
16,16,16,16,
16,16,16,16,
16,16,16,16

INTER4X4_CHROMAU =
16,16,16,16,
16,16,16,16,
16,16,16,16,
16,16,16,16

INTER4X4_CHROMAV =
16,16,16,16,
16,16,16,16,
16,16,16,16,
16,16,16,16

INTRA8X8_LUMA =
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16

INTER8X8_LUMA =
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16

to sum it up: iso/itu doesnt seem to think its worth the hassle to define different matrices for 4x4 luma and chromas

happy comparing!

attached the available cqm matrices coming with the avc reference encoder developed by mpeg plus soulhunters and sharktooths matrices:

[Soulhunter]

5) Soulhunters AVC v1

Description: My "Soulhunters v3 -> AVC" try... :P

#Soulhunters AVC v1

INTRA4X4_LUMA =
 8,12,14,16,
12,14,16,18,
14,16,18,24,
16,18,24,32

INTRA4X4_CHROMAU =
 8,12,14,16,
12,14,16,18,
14,16,18,24,
16,18,24,32

INTRA4X4_CHROMAV =
 8,12,14,16,
12,14,16,18,
14,16,18,24,
16,18,24,32

INTER4X4_LUMA =
 8,12,14,16,
12,14,16,18,
14,16,18,20,
16,18,20,24

INTER4X4_CHROMAU =
 8,12,14,16,
12,14,16,18,
14,16,18,20,
16,18,20,24

INTER4X4_CHROMAV =
 8,12,14,16,
12,14,16,18,
14,16,18,20,
16,18,20,24

INTRA8X8_LUMA =
 8, 8,10,12,13,14,15,16,
 8,10,12,13,14,15,16,18,
10,12,13,14,15,16,18,20,
12,13,14,15,16,18,20,22,
13,14,15,16,18,20,22,24,
14,15,16,18,20,22,24,26,
15,16,18,20,22,24,26,28,
16,18,20,22,24,26,28,32

INTER8X8_LUMA =
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16,
16,16,16,16,16,16,16,16

Bye

[bond]

Soulhunter, it would be great if you can write your matrices exactly the same way as i did, so
- no spaces in between the values, only ","
- using "INTRA4X4_LUMA =", "INTER8X8_LUMA =" aso

cause thats the way the cqm files are written for the reference software.
i think it would be great if there could be a standardised way of writing these files and the reference software files are useable for that i think

[yaz]

@soulhunter
you're a big inventor but would u, pls, drop some words about what would we expect from this matrix ? i don't think we have enough experience w/cqm in avc (at least i haven't, sorry for those who has)
thx
y

[Soulhunter]

Originally Posted by yaz

@soulhunter
...what would we expect from this matrix ?

- It quantizes intra-frames stronger than inter-frames
- It drops high frequencys stronger than low ones
- It treats the luma and the chroma equally
- It wants to be tested by you... ^^


Bye

[Soulhunter]

6) Soulhunters AVC v2

- It quantizes intra-frames and inter-frames equally
- It drops a lot high frequencys in the 4X4 chroma
- It generally drops a lot high frequencys
- It is a HVS style attempt... ^^;

#Soulhunters AVC v2

INTRA4X4_LUMA =
 8,12,14,16,
12,14,16,18,
14,16,18,24,
16,18,24,32

INTRA4X4_CHROMAU =
 8,14,16,24,
14,16,24,32,
16,24,32,64,
24,32,64,128

INTRA4X4_CHROMAV =
 8,14,16,24,
14,16,24,32,
16,24,32,64,
24,32,64,128

INTER4X4_LUMA =
 8,12,14,16,
12,14,16,18,
14,16,18,24,
16,18,24,32

INTER4X4_CHROMAU =
 8,14,16,24,
14,16,24,32,
16,24,32,64,
24,32,64,128

INTER4X4_CHROMAV =
 8,14,16,24,
14,16,24,32,
16,24,32,64,
24,32,64,128

INTRA8X8_LUMA =
 8, 9,10,12,13,14,15,16,
 9,10,12,13,14,15,16,24,
10,12,13,14,15,16,24,32,
12,13,14,15,16,24,32,48,
13,14,15,16,24,32,48,56,
14,15,16,24,32,48,56,64,
15,16,24,32,48,56,64,96,
16,24,32,48,56,64,96,128

INTER8X8_LUMA =
 8, 9,10,12,13,14,15,16,
 9,10,12,13,14,15,16,24,
10,12,13,14,15,16,24,32,
12,13,14,15,16,24,32,48,
13,14,15,16,24,32,48,56,
14,15,16,24,32,48,56,64,
15,16,24,32,48,56,64,96,
16,24,32,48,56,64,96,128

Bye

[Sharktooth]

It will be hard to beat the "standard" quantization with a custom matrix...
It's like ASP h263 vs CQMs... h263 wins most of the times even if visual quality is inferior...

[Joe Fenton]

This thread should probably be a sticky.

[lexor]

Originally Posted by Joe Fenton

This thread should probably be a sticky.

it shouldn't because the moment it becomes sticky it will gain too much of an "official" status which would lead many people to believe that matrices presented here are better than default. Which hasn't been shown by any reasonably involved visual comparison.

when a bunch of people do that and show visual difference thorough clips or screenshots then we can start a sticky with those "better" matrices.

P.S. After writing this I'm sort of ashemed of not participating in visual comparison they had in XVid forum for CQM's

[Soulhunter]

A CQM sticky in the XviD forum would be nice... XD


Bye

[Soulhunter]

@ bond

Btw, is it possible to add separate scaling for 8x8 chroma ???

Would help to make a real HVS style CQM... ^^


Bye

[Sharktooth]

No, there's no 8x8 chroma quantization in h.264.

[Sharktooth]

i confirm what revgen said. 700kbps is too low for EQM AVC-HR (unless the source is supercompressible o.r the resolution is lowered)

[leowai]

The usage of the EQM AVC-HR cqm is mentioned in the download page (2005-08-01).
http://forum.doom9.org/showpost.php?...92&postcount=1

EQM AVC-HR rev.1
This matrix was made for medium-high bitrate backups with High-Profile AVC encoders that support custom matrices and 8x8 discrete cosine transform. It also reduces blocking.

Usage:
- high quality backups of DVD material at full resolution (anamorphic) and bitrates of at least 1300-1400kbps (for average motion and not noisy/old movies).
- non anamorphic/vertically resized encodes at bitrates of at least 1000kbps.

Standard version (JM, x264 & Encavc beta2-2 compatible)

Probably cqm users need more information when using the cqm. I think it's encouraged that the author could follow what sharktooth did. Just provide more detail information & recommandations about the usage of their cqm on the download page, i.e.:
- Movie resolution (640x480 up to 720x480)
- Interlaced* or Progressive video?
*Sorry, i'm not familiar to cqm, is there any cqm for interlaced video? (where is the "I'm with stupid" icon? I can't find it!!)
- Recommended bitrates (1000kbps)
- Recommended movie type (Anime, TV & etc)

[Oline 61]

What matrix should I use for high bitrate (3905 kbit/s) backups of live action DVD movies? Should I just use JVT?

[Oline 61]

bump?
Should I try without any matrix?

[kotrtim]

INTRA4X4_LUMA =
10,12,14,16,
12,14,16,18,
14,16,18,20,
16,18,20,20

INTRA4X4_CHROMAU =
13,14,15,16,
14,15,16,18,
15,16,18,20,
16,18,20,20

INTRA4X4_CHROMAV =
13,14,15,16,
14,15,16,18,
15,16,18,20,
16,18,20,20	

INTER4X4_LUMA =
10,12,14,16,
12,14,16,18,
14,16,18,20,
16,18,20,20

INTER4X4_CHROMAU =
13,14,15,16,
14,15,16,18,
15,16,18,20,
16,18,20,20		

INTER4X4_CHROMAV =
13,14,15,16,
14,15,16,18,
15,16,18,20,
16,18,20,20	

INTRA8X8_LUMA =
10,10,11,12,12,13,14,15,
10,11,12,13,13,15,15,16,
11,12,12,14,15,15,16,17,
12,13,14,15,15,16,17,18,
12,13,15,15,16,17,18,19,
13,15,15,16,17,18,19,19,
14,15,16,17,18,19,19,20,
15,16,17,18,19,19,20,20

INTER8X8_LUMA =
10,10,11,12,12,13,14,15,
10,11,12,13,14,14,15,16,
11,12,12,14,14,15,16,17,
12,13,14,14,15,16,17,18,
12,14,14,15,16,17,18,19,
13,14,15,16,17,18,19,20,
14,15,16,17,18,19,20,20,
15,16,17,18,19,20,20,20

try this?
No matter how much you tweak the default flat16, it is always a drop in psnr...
This is just simply cut and paste from xvid's SixOfNine, merging with soulhunter_v1's 4x4s .....decresing all the values so that no. 20 is the
max...

soulhunter's v2 has some problems, it tends to so some ugly red spot on a clean blue sky with crf 26, well, i tried this sixofnine port, it does solve this problems... still need testing

[Soulhunter]

Red spot on a clean blue sky?


Bye

[kotrtim]

picture order
1. original
2. soulhunter v2
3. eqm_avc_hr
4. jvt
5. soulhunter/sixofnine merge

look carefully, the top left corner (North East)
I'm using --nf

besides this, soulhunter v2 will produce errors like one big red area on a clean blue sky, tried eqm, no such errors, i will try to reproduce.

EDIT: sorry, checked eqm, same error, even worst 2 spot, one clear, one faint...jvt does not have such errors, which is very annoying

EDIT2 : Tested with more different settings, the above encodes are all
default settings (except crf 26 , cqm, nf).. I tried subme 6 and it help to make the spot less noticible and also shifted the spot to another location, subme 6 + no fast pskip solved all the problems, it's more like a x264 bug, not a cqm problem, but of course if default settings is used with default matrix, no such problems

So for those who use cqm, subme 6 + no fast pskip is a must, even better if trellis 2 is used

[IgorC]

Originally Posted by kotrtim

...So for those who use cqm, subme 6 + no fast pskip is a must, even better if trellis 2 is used

If I'm not mistaking x264 doesn't use fast skip when cm is enable.