Custom Quant Matrices comparison

[bond]

i made a comparison of the currently available cqm for avc on my favourite matrix1 clip (smith interrogating morpheus, lobby shootout)

i encoded the clip with x264 r342 and the following settings:
x264 --pass 2 --bitrate 770 --bframe 3 --b-pyramid --ref 5 --mixed-refs --filter -5:-5 --8x8dct --analyse all --weightb --progress -o output.mp4 input.avsthe bitrate was 770kbps, the resolution was 720x288 (resized with lanczos, no prefiltering). so a typical 1 CDR DVD backup situation

the following matrices were tested:
- flat 16 (the default matrix of avc, as used in main profile)
- jvt (the standard alternative matrix of avc of high profile (comparable to the "mpeg" matrix in asp))
- q_matrix (a cqm coming with the reference encoder)
- q_matrix2 (also coming with the reference)
- Sharktooth's EQM AVC-HR (mainly focusing on high bitrates)
- Soulhunter's AVC v1
- Soulhunter's AVC v2
- *.mp4 guy's Hybrid-AVC-LR

(you can download all of them here (http://forum.doom9.org/showthread.php?t=96159))


subjective comparison

for this purpose i used the avscompare tool

and the winner is...

1) Soulhunter's AVC v2
2) flat 16
Sharktooth's EQM AVC-HR rev.1
Soulhunter's AVC v1
3) jvt
q_matrix
q_matrix2
*.mp4 guy's Hybrid-AVC-LR

my comments on the matrices:
- soulhunter's avc v2 outperformed the standard flat matrix in as good as all scenes sharpness-wise (without showing more blockyness). it also was better regarding the treatment of walls and smoke.
in a few closeup face scenes flat was slightly better, but in more other scenes it was the other way round. it definitely had problems and performed worse in the highmo scene were lots of parts of the walls are flying around where it showed clearly less details than flat

- flat 16, the default avc matrix, definitely performed very good and was only outperformed by soulhunter's avc v2. so it doesnt really hurt using it
the big problem the flat 16 matrix has was with the treatment of walls where it performed worse than every other matrix i tested

- sharktooth's matrix clearly handled the walls better than flat
it performed worse than flat on the scene where neo is standing still in front of the fast moving weapon shelf. also flat retained more details in the "flying wall pieces" scene
regarding smoke and closeup faces the retained details varried, sometimes flat being better sometimes not, so i would say on average they are on par

- soulhunter's avc v1 matrix performed pretty much the same as sharktooth's. it handled the walls better than flat, varrying results regarding closeup faces and it also had problems with the flying wall pieces compared to flat. additionally it performed worse on smoke than flat
it was on par with flat on the "neo in front of moving shelf" scene

- jvt like all the others too performed better on walls than flat, but was worse on all other scenes, except on one closeup face scene and it performed the same way as flat on smoke
thinking about that this is the alternative matrix of the avc specs it performed really poorly

- q_matrix also performed better on walls and was the same than flat on smoke
for the rest it was worse on all other scenes and clearly worse in the flying wall pieces scene

- q_matrix2 performed slightly better than q_matrix on a few face closeup scenes, but still was all in all worse than flat
it handled walls better than flat and also could keep up on smoke

- *.mp4 guy's Hybrid-AVC-LR also couldnt keep up with flat sharpnesswise. in fact i would say it was the matrix with the smoothest picture (maybe being more suited for really low bitrates and not medium bitrates as tested here?). it also didnt handle the smoke very well
on the walls it performed better than flat


"objective" comparison

here the avg psnr values calculated by x264:

flat 16: 44.091
q_matrix2: 44.079
hybrid_avc_lr: 44.042
soulhunter avc v2: 43.958
q_matrix: 43.951
jvt: 43.933
soulhunter avc v1: 43.909
eqm_avc_hr: 43.843the only thing these values tell me tough, once more giving clearly different result than my eyes, is that

PSNR sucks!!!!

to hell with these objective results, people should feel offended already when even reading the label indicating that psnr should give results that are "objective"

</flame>

;)

[IgorC]

PSNR sucks!!!!

Thank you, bond. I thought only a few guys (including me ) think so.

[Soulhunter]

Thanks for taking the time... ^__^


Bye

[virus]

PSNR sucks!!!!
...that's why it has been used as a tool for measuring a codec's coding efficiency(1) by researchers all over the world - including the ones who developed all the MPEG standards - for the last 20 years...

Of course, saying that PSNR sucks because it cannot evaluate what your thoughts are is just like saying that all boats suck since you cannot park them in the road in front of your house.

------------------------------------------------------------------------
(1) the ability to reproduce a source with a given distortion measure with the lowest number of bits. Nothing to do with psychovisual optimizations.

[Revgen]

@Bond

If your willing, would you be interested in testing MP4 Guy's matrix?

It should be included in Sharktooth's build.

It's supposed to be optimized for 1 cd backups.

[DarkZell666]

@virus :

I think you'll agree with me that PSNR is a plain "mathematical deviation from original source" value, not an "ugliness/beatifulness factor"

1) a given scene can have extremely low PSNR but still "look" (as in "be percieved) pretty much the same as the original
2) an other scene can have high PSNR and look awfull

This explains why many people say PSNR sucks (which from a perceptual point of view is a bit true of course).

The only reason all those "big brained scientific" people use PSNR is to have a numeric value to use, but we all know that numbers are precisely only numbers and that human eyes see what they _can_ see (not what the numbers say), this is why matrices and HVS tuning exists, and blablabla (same old debate as usual really ;))

Lets put it another way : scientifics need objective mathematical proof but the audience needs a subjectively clean picture, and you can't translate subjective quality into an integer number (nor an floating point number for that mater ;))
--

As for the matrices I've never used any yet (I though the results I had until now were satisfying), I'll try them right away and try to sqeeze those anime episodes into 50MB (getting closer :P) Any sort of special matrix for anime and/or for VERY low bitrate ? (i'm talking about ~230kbps here ^^) I'll have a try with *.mp4 guy's too =)

[Sagittaire]

flat 16: 44.091
q_matrix2: 44.079
soulhunter avc v2: 43.958
q_matrix: 43.951
jvt: 43.933
soulhunter avc v1: 43.909
eqm_avc_hr: 43.843


... and your conclusion is PSNR sucks ... lol

1) yours encoding are high quality encoding ... certainely ~q23 for average.

2) max difference quality is only 0.25 dB. In pratice it's very hard for eyes to notices quality difference like it's difficult to see difference between for example 1000 kbps and 1050 kbps encoding.

3) Your eyes say "flat 16 (uniforma matrix) is better than JVT Matrix" but JVT expert group say "our matrix is visually very better that flat matrix". Your subjective comparison is true for your eyes and only for your eyes.

4) Like for ASP, PSNR is unable to say "Matrix X is better Matrix Y" ... and certainely your eyes too ... ;)


Finaly if I have choice I make encoding with flat matrix because:
- Seem to have very good subjective quality (in all condition and for all eyes)
- best objective quality (for all condition)
- better compatibility for soft/hardware decoder

In french : CQFD

[Sagittaire]

Finally IMO:

I think that "classical CQM construction" is very useful for encoding only without inloop because the principal HVS tunning (higher relative degradation for high frequencies and lower relative degradation for low frequencies for example) are not the same with inloop and without inloop : deblocking is in fact global optimisation and is good for gradiant representation (low frequencies) in most case.

Must be very interessing to see results with for example inverse "classical matrix construction" (higher relative degradation for low frequencies and lower relative degradation for high frequencies for example) with high deblock level ...

If for your eyes flat matrix is best with deblock = 0, certainely that JVT matrix will be better for your eyes with deblock = - 3 or somethink like that. Deblock + CM combinaison is very complex ... and we can use too custum deblock matrix with encavc from Ateme ... :D

[bond]

"max difference quality is only 0.25 dB" which people normally cant differentiate? than i wonder why i have been able to rank the matrices in the subjective part easily in three categories with my eyes only, and if i would even look more closely i am sure that i could also rank them more clearly

my eyes are pretty good (no glasses or anything) and i have the opinion that humans are indeed able to spot quality differences and especially sharpness differences and that the perception of sharpness isnt really affectable by subjective opinions
i think that x264 is so good that it leaves hardly any big issues open that could really be influenced by the personal opinion on what looks better (eg the old smoother image vs. more blocks discussion), at least in the clip i tested

at the "but mpeg says, the pros use it" stuff:
i never read that jvt claims that the jvt matrix is better than the flat one and i doubt that they would make such a statement, cause its surely not true always (as my eyes showed me), cause jvt really sucked here
apart from that if it would be true why didnt they specify it as the default matrix in main profile...

so all i can say is i 100% trust my eyes (of course) and i dont trust psnr at all.
if someone has another opinion i would recommend him/her to try visual comparisons him/herself (avscompare is a great tool for that if you want to compare encodes highly zoomed in and frame-by-frame)
of course i know its damn much work, but if you want to know how things look for your eyes you cant come around using your eyes ;)

-------

so anyone having an opinion about the cqm tested here? own tests? after all the topic of this thread is the performance of cqms and not psnr :rolleyes:

[*.mp4 guy]

Finally IMO:
Must be very interessing to see results with for example inverse "classical matrix construction" (higher relative degradation for low frequencies and lower relative degradation for high frequencies for example) with high deblock level ...
<Evil Grin> I'll make one.... if you promise to use it for all your encodes ;) . On a more serious note Mpeg4-AVC already has trouble with flatish areas a matrix like this would most likely turn the picture into block soup.

[Sagittaire]

In fact it's very strange ... but perhabs that AVC's DCT don't work like ASP's DCT ... ???

1) You seem don't like inloop ... you choose deblocking = -5. In blocking vs sharp fight you seem to prefer more sharp but more blocking too. It's your personnal choise and I can't discut that. But for the majority visually the good range is [-2 , 0] and more for animates ...

2) All the custum matrix tested are classical HVS tunning (higher relative degradation for high frequencies and lower relative degradation for low frequencies) but better low frequencies = less blocking and worst high frequencies = more ringing ...

For inloop you choose one way and for CM you choose opposite way. I think that inloop and CM are not independant setting. If you want make good subjectif test you must use all CM with for example -6 , -4 , -2 and -0 for deblocking strenght ...

but perhaps that my reasoning is false ... ???

[*.mp4 guy]

Sense you insist....

Sagittaires' AVC CQM

INTRA4X4_LUMA =
128,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTRA4X4_CHROMAU =
128,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTRA4X4_CHROMAV =
128,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTER4X4_LUMA =
128,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTER4X4_CHROMAU =
128,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTER4X4_CHROMAV =
128,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTRA8X8_LUMA =
128,78,64,52,44,36,30,26,
78,64,52,44,36,30,26,21,
64,52,44,36,30,26,21,18,
52,44,36,30,26,21,18,17,
44,36,30,26,21,18,17,17,
36,30,26,21,18,17,17,16,
30,26,21,18,17,17,16,16,
26,21,18,17,17,16,16,15

INTER8X8_LUMA =
128,74,56,48,40,34,28,24,
74,56,48,40,34,28,24,20,
56,48,40,34,28,24,20,18,
48,40,34,28,24,20,18,17,
40,34,28,24,20,18,17,17,
34,28,24,20,18,17,17,16,
28,24,20,18,17,17,16,16,
24,20,18,17,17,16,16,15

[Manao]

You can't make the 4x4 dct ring. The 8x8 can ring, however.

If you want make good subjectif test you must use all CM with for example -6 , -4 , -2 and -0 for deblocking strenght ...There's some truth in that statement.

[bond]

ok i now added mp4 guys cqm, sorry for the results :( ;)

and i compared soulhunter v2 with nerodigital avc and i have to say nd avc (main profile) is still better clearly than x264
its sharper (in face closeups and high motion (flying wall pieces) and also handles walls clearly better (which might indicate that the wall handling of x264 could be enhanced a lot even with flat without needing a cqm)

edit: i now also compared x264 and nd avc to xvid and of course both were clearly better than xvid, except in two scenes: neo standing in front of moving shelf and flying wall pieces


There's some truth in that statement.yeah and i also should test multiple streams and bitrates and resolutions and best would be to do it on full 2 hours encodes...

man, you guys dont seem to do visual comparisons often, cause than you would know how much time it takes ;)

[*.mp4 guy]

The results are rather what I expected. it is aimed more twards producing a pleasing picture at low bitrates then retaing details.

[Sagittaire]

Sense you insist....

Sagittaires' AVC CQM

INTRA4X4_LUMA =
128,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTRA4X4_CHROMAU =
128,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTRA4X4_CHROMAV =
128,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTER4X4_LUMA =
128,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTER4X4_CHROMAU =
128,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTER4X4_CHROMAV =
128,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTRA8X8_LUMA =
128,78,64,52,44,36,30,26,
78,64,52,44,36,30,26,21,
64,52,44,36,30,26,21,18,
52,44,36,30,26,21,18,17,
44,36,30,26,21,18,17,17,
36,30,26,21,18,17,17,16,
30,26,21,18,17,17,16,16,
26,21,18,17,17,16,16,15

INTER8X8_LUMA =
128,74,56,48,40,34,28,24,
74,56,48,40,34,28,24,20,
56,48,40,34,28,24,20,18,
48,40,34,28,24,20,18,17,
40,34,28,24,20,18,17,17,
34,28,24,20,18,17,17,16,
28,24,20,18,17,17,16,16,
24,20,18,17,17,16,16,15


hmmm ... your dct construction is strange ... perhabs CM more like that with simple linear inverse construction. If you prefer inloop = N, try this matrix perhabs with inloop = N + 2 ...


INTRA4X4_LUMA =
16,20,18,16,
20,18,16,14,
18,16,14,12,
16,14,12,10

INTRA4X4_CHROMAU =
16,20,18,16,
20,18,16,14,
18,16,14,12,
16,14,12,10

INTRA4X4_CHROMAV =
16,20,18,16,
20,18,16,14,
18,16,14,12,
16,14,12,10

INTER4X4_LUMA =
16,20,18,16,
20,18,16,14,
18,16,14,12,
16,14,12,10

INTER4X4_CHROMAU =
16,20,18,16,
20,18,16,14,
18,16,14,12,
16,14,12,10

INTER4X4_CHROMAV =
16,20,18,16,
20,18,16,14,
18,16,14,12,
16,14,12,10

INTRA8X8_LUMA =
16,22,21,20,19,18,17,16,
22,21,20,19,18,17,16,15,
21,20,19,18,17,16,15,14,
20,19,18,17,16,15,14,13,
19,18,17,16,15,14,13,12,
18,17,16,15,14,13,12,11,
17,16,15,14,13,12,11,10,
16,15,14,13,12,11,10,10

INTER8X8_LUMA =
16,22,21,20,19,18,17,16,
22,21,20,19,18,17,16,15,
21,20,19,18,17,16,15,14,
20,19,18,17,16,15,14,13,
19,18,17,16,15,14,13,12,
18,17,16,15,14,13,12,11,
17,16,15,14,13,12,11,10,
16,15,14,13,12,11,10,10

[Sharktooth]

bond, my matrix produces much less blocks than flat even at mid bitrates (btw i consider 770 a pretty low bitrate). details are retained since you need a lower filtering for the same bitrate but if you used -5:-5, everything should be a blockfest...

[Sagittaire]

bond, my matrix produces much less blocks than flat even at mid bitrates (btw i consider 770 a pretty low bitrate). details are retained since you need a lower filtering for the same bitrate but if you used -5:-5, everything should be a blockfest...

... and IMO I think that JVT matrix are only used for high definition with low quant and parhabs with inloop desactived.

CM for ASP are in practice always used for blocking fight and we can't use the same HVS property with AVC because AVC can use inloop. Finally Flat 16 + inloop is probaly a good HVS combinaison ...

@ *.mp4 guy

My "simple linear inverse construction matrix" seem to work visually very good and with very high PSNR ... lol

[*.mp4 guy]

@ *.mp4 guy

My "simple linear inverse construction matrix" seem to work visually very good and with very high PSNR ... lol
Well I misunderstood you, I thought you were arguing for an "inverse HVS" style matrix, which of course wouldn't work well. And your right about the matrix you made, it would work reasonably well. Though I would guess that it's not so great for lower bitrates.

[Sagittaire]

Well I misunderstood you, I thought you were arguing for an "inverse HVS" style matrix, which of course wouldn't work well. And your right about the matrix you made, it would work reasonably well. Though I would guess that it's not so great for lower bitrates.

And why not ... ???

HVS properties for ASP are not HVS properties for AVC ... If you want know you must try !

try with that ... pehabs that it's good ... :)


INTRA4X4_LUMA =
16,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTRA4X4_CHROMAU =
16,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTRA4X4_CHROMAV =
16,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTER4X4_LUMA =
16,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTER4X4_CHROMAU =
16,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTER4X4_CHROMAV =
16,64,40,30,
64,40,30,24,
40,30,24,16,
30,24,16,16

INTRA8X8_LUMA =
16,78,64,52,44,36,30,26,
78,64,52,44,36,30,26,21,
64,52,44,36,30,26,21,18,
52,44,36,30,26,21,18,17,
44,36,30,26,21,18,17,17,
36,30,26,21,18,17,17,16,
30,26,21,18,17,17,16,16,
26,21,18,17,17,16,16,16

INTER8X8_LUMA =
16,74,56,48,40,34,28,24,
74,56,48,40,34,28,24,20,
56,48,40,34,28,24,20,18,
48,40,34,28,24,20,18,17,
40,34,28,24,20,18,17,17,
34,28,24,20,18,17,17,16,
28,24,20,18,17,17,16,16,
24,20,18,17,17,16,16,16