Adaptive Quantization and Custom Matrices

[dotBL0t]

Hi All!!! Just a little, dumb question:
Is it safe to use adaptive quantization when using custom matrices or is it better to disable it? :confused:

[Dark Shikari]

Its safe, I see no reason why not.

[shon3i]

Its safe, I see no reason why not.
because some guru from forum said earlier that the use of AQ and the CQM-together is not a good idea for quality. However, I use all the time together Prestige and AQ and always gets better results together than separately.
The only thing is, Prestige bitrate-hungry, if bitrate sufficient quality will be good

[dotBL0t]

Ok thanks a lot! I was not sure if it could interfere with the quantization restrictions set by the custom matrix.

Edit:

@shon3i:
Someone also said that Prestige matrix sucks. How do you feel with it?

[Sagekilla]

The thing is, CQM's are kind of pointless now that we have so many of the psychovisual algos. AQ really nullifies the need for having a CQM, for one. Also, prestige is a bad matrix to use because of how its setup. You end up getting these small annoying artifacts in the image. There's a thread somewhere that describes why it's a bad matrix to use, just use search.

[Ranguvar]

The thing is, CQM's are kind of pointless now that we have so many of the psychovisual algos. AQ really nullifies the need for having a CQM, for one. Also, prestige is a bad matrix to use because of how its setup. You end up getting these small annoying artifacts in the image. There's a thread somewhere that describes why it's a bad matrix to use, just use search.

+1

OP, I recommend doing some small tests and seeing what looks better to you, though. Everyone's eyes are different :)

[Sagekilla]

Indeed, but I think everyone should be able to agree that Prestige is not a matrix you want to use for compresssing at optimal bitrates. It eats up too much bits and looks funky at the same time. The original thread for the Prestige matrix doesn't have the pictures up anymore, but if you look at this (http://forum.doom9.org/showthread.php?p=989214#post989214) one you can see how it works.


Friends don't let friends use prestige for any sane encoding ;)

[G_M_C]

I thought is was best used if you wanted to create some special effect, like atrificially adding grain (using the matrix) giving the feel of an "older print". At least, thats the way it seemed to be used in the original Prestige-movie.

[Sagekilla]

It's not meant for artificially adding grain -- That's what we have AddGrainC() and such for ;) It's used for grain retention, but the way it works it artifacts like crazy and eats up a lot of bitrate. Now though, you'd be better off using no matrix and using a higher bitrate or lower crf with --psy-rd 1:1 and --aq-strength 1.0. x264 using it's psychovisual enhancements will retain detail better than a CQM will, tbh.

[Terranigma]

x264 using it's psychovisual enhancements will retain detail better than a CQM will, tbh.

But with a cqm like the prestige, you'd get even more grain retention at the same bitrate. If you don't believe, then do some tests. The main reason why i personally use cqm's is for fades, because the default flat matrix sucks at fades even with psy-rdo unless you increase the strength to undesirable results.

[Sagekilla]

At the same bitrate yes, but if you're using something like crf then prestige gobbles up bitrate fast. You still have to deal with the zig-zag artifacts that prestige introduces, too.

[Terranigma]

You still have to deal with the zig-zag artifacts that prestige introduces, too.

I've never noticed those artifacts. :)
And regardless, the prestige matrix is still being used by some internal hd groups out there even with all the advancements thats been made, so it can't be all that bad. ;)

[Sagekilla]

I suppose it depends on the bitrate you encode at as well ;) I personally stopped using it after it increased my bitrate usage by over 40% and introduced artifacts which I could notice. But, there's the beauty of all this: We have a choice :) I prefer a clean, detailed looking image, without grain or noise. So I don't use matrices for grain retention.

If you want extreme grain retention, then Prestige has it's place. But I don't think it's the best way to go.

[Sharktooth]

psy-rdo/psy-trellis is even better...
so, no need for a CQM for grainy sources. just use psy-rdo 1.0 (default) and set psy-trellis to value between 0.5 and 1.0.
prestige also looks like crap with EVERY source, except if you use a much higher bitrate... in that case it looks good on grainy stuff. but as i said, psy-rd/psy-trellis combo with default matrix, look even better than prestige in every combination with other features.

[Terranigma]

but as i said, psy-rd/psy-trellis combo with default matrix, look even better than prestige in every combination with other features.

Care to make some screenshots/clips to back that up? And make sure to include fades.

[dotBL0t]

I made some tests and I noticed that prestige matrix is not so bad if used with low deadzones settings (e.g. 10 - 5 or 6 - 6) and I didn't notice any artifact (or at least far less than psy ones at same bitrate). Unfortunately i deleted the clips used for comparison but I will make other tests as soon as i can and I'll post the results here if you like.

[Terranigma]

I made some tests and I noticed that prestige matrix is not so bad if used with low deadzones settings (e.g. 10 - 5 or 6 - 6) and I didn't notice any artifact (or at least far less than psy ones at same bitrate). Unfortunately i deleted the clips used for comparison but I will make other tests as soon as i can and I'll post the results here if you like.


Yes, that'd be great. :)

[Sagekilla]

Remember, because of how prestige works it inherently kills compression heavily.

[shon3i]

psy-rdo/psy-trellis is even better...Well that combo produce much ringing for my taste, i think better combo now is --fgo + presitge or --psy-rd 0.8,0, better grain retention and much less ringing. For BD9 backups and 1080p resolution prestige matrix are just fine.

I've never noticed those artifacts.
And regardless, the prestige matrix is still being used by some internal hd groups out there even with all the advancements thats been made, so it can't be all that bad. Agree, PQ on this releases is much better than others.

[Sagekilla]

shon3i: In that case just adjust the setting more :) Lower psy settings will make sure it won't produce as heavy of an effect. A small amount of psy-rd + psy-trellis can help.

[*.mp4 guy]

Psy rdo/fgo only affect spatial/temporal bitrate distribution directly, and indirectly effect freqency distribution of bitrate, through the metrics they use.

Aq effects spatio temporal distribution of bitrate, but bases this distribution on metrics that are effected by frequency distribution and intensity.

Matrices act as a direct way to alter the bitrate distribution of spatial/temporal-delta frequencies, with no direct effect on spatial or temporal location of bitrate distribution, though certain spatial and temporal features will indirectly be affected.

Psy trellis directly effects which frequencies are retained, and discarded (as a cqm) but does so in a nonlinear fashion (not like a cqm) and does not effect how much compression is aplied to the retained frequencies in relation to each other (unlike a cqm).

All psy optimizations show a general tendency to shift bitrate away from the most bitrate hungry areas of the image (primarily transients) the difference is the exact mechanism by which this is done. This loss of fidelity in transient areas is what causes these methods to cause ringing.

All metric based methods (everything but cqms) are technically introducing non-linearity into the compression process, nonlinear system tend to outperform similar linear systems, but can be prone to "catastrophic failure" whereas linear systems are almost always extremely dependable.

Idealy psy aq, and psy trellis would be integrated with a psy cqm system however psy cqm systems are dificult to implement unless the standard was written with them in mind (allowing cqm switching per frame, or macroblock, perhaps using a list of precomputed cqms for each gop and switching between them on a frame/mb/region basis). Unfortunately, computerised cqm generation is notoriously bad, making the lack of support within established standards a moot point, and explaining the large amount of pre made cqms for various standards, situations, and uses.