x264 not honoring cqm coefficients

[jnelson02]

Recently, I tried to run an encode using the same quant matrix as the source, but the resulting matrix extracted from my encoded file is different than the one specified by the cqm.

Here's my command string:

"C:\Program Files\megui\tools\x264\x264.exe" --crf 20.0 --level 4.1 --ref 4 --mixed-refs --no-fast-pskip --bframes 3 --b-adapt 2 --b-pyramid --weightb --direct auto --deblock -3:-2 --subme 9 --psy-rd 1.1:0 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 30000 --vbv-maxrate 38000 --me umh --merange 24 --threads auto --thread-input --aq-mode 0 --sar 1:1 --cqmfile "D:\Matrices\Matrices\the_one.cfg" --aud --progress --no-dct-decimate --no-psnr --no-ssim --output

I used h264_parse to extract the original quant matrix (the_one.cfg):

INTRA4X4_LUMA =
8,13,20,28,
13,20,28,32,
20,28,32,37,
28,32,37,42

INTRA4X4_CHROMAU =
10,14,20,28,
14,20,28,32,
20,28,32,37,
28,32,37,42

INTRA4X4_CHROMAV =
16,16,16,16,
16,16,16,16,
16,16,16,16,
16,16,16,16

INTER4X4_LUMA =
8,13,20,28,
13,20,28,32,
20,28,32,37,
28,32,37,42

INTER4X4_CHROMAU =
10,14,20,24,
14,20,24,27,
20,24,27,30,
24,27,30,34

INTER4X4_CHROMAV =
16,16,16,16,
16,16,16,16,
16,16,16,16,
16,16,16,16

INTRA8X8_LUMA =
8,10,13,16,18,23,25,27,
10,11,16,18,23,25,27,29,
13,16,18,23,25,27,29,31,
16,18,23,25,27,29,31,33,
18,23,25,27,29,31,33,36,
23,25,27,29,31,33,36,38,
25,27,29,31,33,36,38,40,
27,29,31,33,36,38,40,42

INTER8X8_LUMA =
9,13,15,17,19,21,22,24,
13,13,17,19,21,22,24,25,
15,17,19,21,22,24,25,27,
17,19,21,22,24,25,27,28,
19,21,22,24,25,27,28,30,
19,22,24,25,27,28,30,32,
22,24,25,27,28,30,32,33,
24,25,27,28,30,32,33,35

When I check the encoded file, this is what it get:

pic_scaling_matrix_present_flag: 1
Pic Scaling List[0] Present Flag: 1
delta: 0
scaling list[0][0]: 8
delta: 5
scaling list[0][1]: 13
delta: 0
scaling list[0][2]: 13
delta: 7
scaling list[0][3]: 20
delta: 0
scaling list[0][4]: 20
delta: 0
scaling list[0][5]: 20
delta: 8
scaling list[0][6]: 28
delta: 0
scaling list[0][7]: 28
delta: 0
scaling list[0][8]: 28
delta: 0
scaling list[0][9]: 28
delta: 4
scaling list[0][10]: 32
delta: 0
scaling list[0][11]: 32
delta: 0
scaling list[0][12]: 32
delta: 5
scaling list[0][13]: 37
delta: 0
scaling list[0][14]: 37
delta: 5
scaling list[0][15]: 42
Pic Scaling List[1] Present Flag: 1
delta: 2
scaling list[1][0]: 10
delta: 4
scaling list[1][1]: 14
delta: 0
scaling list[1][2]: 14
delta: 6
scaling list[1][3]: 20
delta: 0
scaling list[1][4]: 20
delta: 0
scaling list[1][5]: 20
delta: 4
scaling list[1][6]: 24
delta: 0
scaling list[1][7]: 24
delta: 0
scaling list[1][8]: 24
delta: 0
scaling list[1][9]: 24
delta: 3
scaling list[1][10]: 27
delta: 0
scaling list[1][11]: 27
delta: 0
scaling list[1][12]: 27
delta: 3
scaling list[1][13]: 30
delta: 0
scaling list[1][14]: 30
delta: 4
scaling list[1][15]: 34
Pic Scaling List[2] Present Flag: 0
Pic Scaling List[3] Present Flag: 1
delta: 0
scaling list[3][0]: 8
delta: 5
scaling list[3][1]: 13
delta: 0
scaling list[3][2]: 13
delta: 7
scaling list[3][3]: 20
delta: 0
scaling list[3][4]: 20
delta: 0
scaling list[3][5]: 20
delta: 8
scaling list[3][6]: 28
delta: 0
scaling list[3][7]: 28
delta: 0
scaling list[3][8]: 28
delta: 0
scaling list[3][9]: 28
delta: 4
scaling list[3][10]: 32
delta: 0
scaling list[3][11]: 32
delta: 0
scaling list[3][12]: 32
delta: 5
scaling list[3][13]: 37
delta: 0
scaling list[3][14]: 37
delta: 5
scaling list[3][15]: 42
Pic Scaling List[4] Present Flag: 1
delta: -8
scaling list[4][0]: 8
scaling list[4][1]: 8
scaling list[4][2]: 8
scaling list[4][3]: 8
scaling list[4][4]: 8
scaling list[4][5]: 8
scaling list[4][6]: 8
scaling list[4][7]: 8
scaling list[4][8]: 8
scaling list[4][9]: 8
scaling list[4][10]: 8
scaling list[4][11]: 8
scaling list[4][12]: 8
scaling list[4][13]: 8
scaling list[4][14]: 8
scaling list[4][15]: 8
Pic Scaling List[5] Present Flag: 0
Pic Scaling List[6] Present Flag: 1
delta: 0
scaling list[6][0]: 8
delta: 2
scaling list[6][1]: 10
delta: 0
scaling list[6][2]: 10
delta: 3
scaling list[6][3]: 13
delta: -2
scaling list[6][4]: 11
delta: 2
scaling list[6][5]: 13
delta: 3
scaling list[6][6]: 16
delta: 0
scaling list[6][7]: 16
delta: 0
scaling list[6][8]: 16
delta: 0
scaling list[6][9]: 16
delta: 2
scaling list[6][10]: 18
delta: 0
scaling list[6][11]: 18
delta: 0
scaling list[6][12]: 18
delta: 0
scaling list[6][13]: 18
delta: 0
scaling list[6][14]: 18
delta: 5
scaling list[6][15]: 23
delta: 0
scaling list[6][16]: 23
delta: 0
scaling list[6][17]: 23
delta: 0
scaling list[6][18]: 23
delta: 0
scaling list[6][19]: 23
delta: 0
scaling list[6][20]: 23
delta: 2
scaling list[6][21]: 25
delta: 0
scaling list[6][22]: 25
delta: 0
scaling list[6][23]: 25
delta: 0
scaling list[6][24]: 25
delta: 0
scaling list[6][25]: 25
delta: 0
scaling list[6][26]: 25
delta: 0
scaling list[6][27]: 25
delta: 2
scaling list[6][28]: 27
delta: 0
scaling list[6][29]: 27
delta: 0
scaling list[6][30]: 27
delta: 0
scaling list[6][31]: 27
delta: 0
scaling list[6][32]: 27
delta: 0
scaling list[6][33]: 27
delta: 0
scaling list[6][34]: 27
delta: 0
scaling list[6][35]: 27
delta: 2
scaling list[6][36]: 29
delta: 0
scaling list[6][37]: 29
delta: 0
scaling list[6][38]: 29
delta: 0
scaling list[6][39]: 29
delta: 0
scaling list[6][40]: 29
delta: 0
scaling list[6][41]: 29
delta: 0
scaling list[6][42]: 29
delta: 2
scaling list[6][43]: 31
delta: 0
scaling list[6][44]: 31
delta: 0
scaling list[6][45]: 31
delta: 0
scaling list[6][46]: 31
delta: 0
scaling list[6][47]: 31
delta: 0
scaling list[6][48]: 31
delta: 2
scaling list[6][49]: 33
delta: 0
scaling list[6][50]: 33
delta: 0
scaling list[6][51]: 33
delta: 0
scaling list[6][52]: 33
delta: 0
scaling list[6][53]: 33
delta: 3
scaling list[6][54]: 36
delta: 0
scaling list[6][55]: 36
delta: 0
scaling list[6][56]: 36
delta: 0
scaling list[6][57]: 36
delta: 2
scaling list[6][58]: 38
delta: 0
scaling list[6][59]: 38
delta: 0
scaling list[6][60]: 38
delta: 2
scaling list[6][61]: 40
delta: 0
scaling list[6][62]: 40
delta: 2
scaling list[6][63]: 42
Pic Scaling List[7] Present Flag: 0

My INTER4X4_CHROMAU matrix is truncated to a flat 8 matrix, and my INTER8X8 is ignored. I've read that cqm files must be JM-compatible, but I don't have a clue what that means.

I'm relatively new to x264 encoding. I've tried using notepad and Megui's AVC cqm editor, with no success. If anyone can point me in the right direction, I'd appreciate it.

[akupenguin]

My INTER8X8 is ignored
x264 used your inter8x8luma matrix, it just didn't need to code it in the bitstream, because your matrix was equal to the inter8x8luma JVT matrix, which is the fallback/default if you use any cqms at all (as opposed to flat16 which is the default if you don't use any cqms).

My INTER4X4_CHROMAU matrix is truncated to a flat 8 matrix
That's a bug in your decoder (or perhaps just in its diagnostic output... did you check the decoded frames?). The first coef in the scaling list is predicted to be 8. If it's coded to be 0 (i.e. delta=-8), then that means "use the appropriate JVT matrix". (x264 couldn't omit the scaling list entirely like with inter8x8luma, because inter4x4chroma fallsback to inter4x4luma, which had already been coded to some other value.) Your decoder apparently parsed the coef value of 0 to mean "end of scaling list, set all remaining coefs to the current prediction value", which is what it means for any coef beyond the first.

BTW, I assume you also know that x264 doesn't support different matrices for U and V, since you didn't complain about that.

Is this all excessively complex? you bet. But if I wanted to save on complexity in this facet of the standard, I'd do it by not supporting CQM at all.

[jnelson02]

Thanks akupenguin for that very detailed response. I didn't know about the different matrices for U and V. I was just trying to see how the quant matrix of the source compared to the flat 16 matrix. I thought it was being modified, but it just turns out that I didn't understand what was happening.
Thanks for the info on the JVT fallback as well. Looks like I've got a lot to learn about x264 and cqms. Seems that the matrix I was trying to use, is very close to the standard JVT matrix. This makes me wonder if h264_parse, which I used to extract and check the matrices, is giving me the right info.