New Custom Quantization Matrices - EQM AVC Series. Last update: EQM AVC-HR 2005-08-01

[Sharktooth]

EQM AVC-HR rev.1
This matrix was made for medium-high bitrate backups with High-Profile AVC encoders that support custom matrices and 8x8 discrete cosine transform. It also reduces blocking.

Usage:
- high quality backups of DVD material at full resolution (anamorphic) and bitrates of at least 1300-1400kbps (for average motion and not noisy/old movies).
- non anamorphic/vertically resized encodes at bitrates of at least 1000kbps.

Standard version (JM, x264 & Encavc beta2-2 compatible):
INTRA4X4_LUMA =
6, 9,13,19,
9,14,20,27,
13,20,28,35,
19,27,35,42

INTRA4X4_CHROMAU =
6,10,15,20,
10,16,21,27,
15,21,28,33,
20,27,33,42

INTRA4X4_CHROMAV =
6,10,15,20,
10,16,21,27,
15,21,28,33,
20,27,33,42

INTER4X4_LUMA =
8,11,15,20,
11,16,21,27,
15,21,28,35,
20,27,35,42

INTER4X4_CHROMAU =
8,11,16,21,
11,17,22,27,
16,22,28,35,
21,27,35,42

INTER4X4_CHROMAV =
8,11,16,21,
11,17,22,27,
16,22,28,35,
21,27,35,42

INTRA8X8_LUMA =
6, 7, 8,10,12,14,16,18,
7, 9,11,13,15,16,18,20,
8,11,14,16,17,19,21,22,
10,13,16,18,20,22,24,26,
12,15,17,20,23,25,28,30,
14,16,19,22,25,29,34,38,
16,18,21,24,28,34,46,52,
18,20,22,26,30,38,52,72

INTER8X8_LUMA =
8, 9,10,12,14,16,19,21,
9,11,13,15,17,19,21,23,
10,13,16,18,20,22,23,25,
12,15,18,21,23,24,26,32,
14,17,20,23,25,27,33,40,
16,19,22,24,27,34,41,52,
19,21,23,26,33,41,53,64,
21,23,25,32,40,52,64,80
Download: EQM AVC-HR (http://www.webalice.it/f.corriga/x264/eqm_avc_hr.cfg)

EDIT: fixed to comply with JVT standard.

[Sharktooth]

*** Just a bump to make you know the HR matrix is out of beta.
More matrices will follow as soon as i finish internal testing.

thanks for your tests :)

[skal]

Hello Sharktooth,
could you elaborate a little on how you deduce your matrices?
("with my eyes" is ok as an answer;)
-Skal

[trbarry]

Hi Sharktooth -

When I wanted consistantly high quality and didn't care greatly about a precise bit rate I would often use 1-pass Xvid with a constant quant of 2, but with 2 b-frames, at q=3. Varyingly this would also be with one of your custom matrices. Other options to taste but mostly using more CPU to get better compression efficiency in a single pass. I tend to store things on external USB drives so a precise size is not that important. While I dislike wasting space it's often not worth another pass just to hit a arbitrary size target if quality demands more anyway.

Can you off hand recommend any sort of fairly high bit rate X264 equivalent set of fixed quant options?

Sorry if this has been covered before but I've been away for awhile and a search didn't turn up anything with these priorities. I don't yet have much feel for what I can get out of X264 (or Nero?) with fixed quants but you seem to work with these things a bit.

- Tom

PS - These are mostly HD caps, at 720p or 544p though I don't know if it matters here.

[Sharktooth]

Hello Sharktooth,
could you elaborate a little on how you deduce your matrices?
("with my eyes" is ok as an answer;)
-Skal
well with h264 i had to re-elaborate some of my theories on human eye perception but generally i "draw" a matrix around my initial idea, then i do a set of tests with a set (10 or more) of matrices (usually those are scaled from the "master" matrix with different factors and scaling formula) on 5 different sources.
Then i analyze the source sequencies and for every sequence i assign a score.
After the analysis phase i sum the scores and choose the matrix with the higest one.
This method was pretty fast for xvid, but since AVC is more complex it requires a lot more time...

Hi Sharktooth -

When I wanted consistantly high quality and didn't care greatly about a precise bit rate I would often use 1-pass Xvid with a constant quant of 2, but with 2 b-frames, at q=3. Varyingly this would also be with one of your custom matrices. Other options to taste but mostly using more CPU to get better compression efficiency in a single pass. I tend to store things on external USB drives so a precise size is not that important. While I dislike wasting space it's often not worth another pass just to hit a arbitrary size target if quality demands more anyway.

Can you off hand recomHi Sharktooth -

When I wanted consistantly high quality and didn't care greatly about a precise bit rate I would often use 1-pass Xvid with a constant quant of 2, but with 2 b-frames, at q=3. Varyingly this would also be with one of your custom matrices. Other options to taste but mostly using more CPU to get better compression efficiency in a single pass. I tend to store things on external USB drives so a precise size is not that important. While I dislike wasting space it's often not worth another pass just to hit a arbitrary size target if quality demands more anyway.

Can you off hand recommend any sort of fairly high bit rate X264 equivalent set of fixed quant options?

Sorry if this has been covered before but I've been away for awhile and a search didn't turn up anything with these priorities. I don't yet have much feel for what I can get out of X264 (or Nero?) with fixed quants but you seem to work with these things a bit.

- Tom

PS - These are mostly HD caps, at 720p or 544p though I don't know if it matters here.
Well, you can try with quants from 20 to 24, and you should have the "equivalent" of q2 (or q2+Bs) for xvid (depending on the CQM you used).mend any sort of fairly high bit rate X264 equivalent set of fixed quant options?

Sorry if this has been covered before but I've been away for awhile and a search didn't turn up anything with these priorities. I don't yet have much feel for what I can get out of X264 (or Nero?) with fixed quants but you seem to work with these things a bit.

- Tom

PS - These are mostly HD caps, at 720p or 544p though I don't know if it matters here.
Well, you can try with quants from 20 to 24, and you should have the "equivalent" of q2 (or q2+Bs) for xvid (depending on the CQM you used).

[trbarry]

Thx. That give me a useful place to start.

- Tom

[jellysandwich]

Is there a way to use custom quant matrices
with x264 vfw?

Edit: Also, is there one similar to MPEG for Xvid?
I like the sharpness that MPEG gives...

js

[yaz]

... you can try with quants from 20 to 24, and you should have the "equivalent" of q2 (or q2+Bs) for xvid (depending on the CQM you used).according to my (very incomplete) tests x264 cq equivalent to xvid q2 is somewhat lower. it's somewhere q16-18. (xvid matrix: mpeg, x264 matrix: default, bvop(xvid):1/1/0)
is there one similar to MPEG for Xvid?
I like the sharpness that MPEG gives...i would love that too. it's good to see that i'm not the very last fan (nerd?) of that matrix ;)

the bests
y

[Sharktooth]

MPEG matrix is different from (and more bitrate hungry than) h263. The values i suggested are intender for a h263-like quant2 compression, also i specified those values depends on custom matrices (in both xvid and x264).

[TheBashar]

Hi Sharktooth,

I understand that your AVC matrices are intended for bitrates of around 1350kbps for 720x430 "average motion" movies. From following some of your XviD matrices development, I believe I recall that the matrices are "tuned" for the target bitrate/resolution ratio and perform sub-optimally on lower bitrate/resolution ratios.

Getting (slowly) to my question... will the matrices perform sub-optimally in regions of an encode where due to a default curve compression (qcomp = 0.6 in x264) bitrates are reduced because of slow/no motion detection? For instance, I may have an encode which averages 1350kbps but a particular scene may have only 800 kbps allocated due to it being a static scene. Will these matrices perform poorly in such regions solely becuase of the low bitrate or will they still perform as expected since in fact there is little or no motion?

The reason I ask is because I am considering unbridling the qcomp by significantly raising the ratetol from its default of 1.0.

Thanks for your time!

[Sharktooth]

Matrices are tuned for a certain compressibility. In regions where the codec assigns less bit, the compressibility is usually higher. So, in static scenes, even if the bitrate gets lower, it doesnt mean the codec is using higher quantizers. It's usually the opposite... However when i made AVC-HR i took into account the possibility to set the bitrate variability from "1" to "inf" and the HR matrix should behave correctly (at least in my tests it proved to do it).
It's all a matter of coefficients scaling... for example if the scaling is too "harsh" it will produce artifacts at lower bitrates (too much difference between quants).

[TheBashar]

Hi Sharktooth,

I really like your CQM. When I was trying to determing a good bitrate before I used your CQM, I had trouble with blocky dark backgrounds even at what I thought were too high bitrates. When I finally switched to your CQM, I found the blockyness was dramatically reduced. I'm guessing your CQM gives more bitrate in the dark color range than the standard one does.

So, all was well in the world. I have always been frustrated by my encodes being a tad too dark though. I recently stumbled onto this 9-04 thread (http://forum.doom9.org/showthread.php?s=&threadid=82217) about ColorMatrix, which corrects that (or the real problem rather). I tried it and in fact it really improved things. So, I started a reencode. Unfortunately, after my first reencode, I found the blockyness was back. Now, there are other differences as well: x264 cli rev 291B instead of 287A, substituted limitedsharpen instead of xsharpen.

I've had a hard time finding an old 287a x264 rrev to test with, so I thought I post here and ask if you think the ColorMatrix change could be responsible. Is there a chance that you tuned your CQM for the incorrect color spectrum that ColorMatrix is meant to correct? Not accusing or anything, just asking so I can narrow down the permutations I need to test.

Thanks!

[Sharktooth]

Well, limitedsharpen may be the cause. Sharpen filters usually lower the source compressibility in a significative manner. In this situation a lower bitrate matrix is needed but you can try rising the deblocking filter threshold by 1 or 2 to help compensate the compressibility loss.
However i dont use colormatrix coz i never experienced darkened or desaturated encodes so IMHO it's not a "tuning" problem.

[Chainmax]

This matrix will come in handy on my next encodes. I'm doing AVC backups of some high motion cartoon captures @1100kbps and 640x480 resolution, and with this matrix I might even be able to disable deblocking. Once I solve some issues with my current filterchain (SSIQ and aWarpSharp are causing green splotches on the sides of the picture) I'll try this and report back.

Sharktooth, how do you expect customs matrices to affect playback compatibility on future AVC capable standalones?

[Sharktooth]

Disabling deblocking is always a bad idea unless you're near saturation.
However, custom matrices are part of high profile and it will be played back only if the SAP will support it.

[Daodan]

Nice thing this matrix, exactly fit for my needs. One question though. I started encoding a movie (140 min movie) with 1800 kbps that I just finished encoding without a custom matrix. The 'problem' is, that the encode is 17 % faster now then before (it's only 30% completed now but the speed increase is obvious). I use the same settings as before so I could compare the results. This is something unexpected I would say.
Is it normal or did my computer start taking drugs?
Well, it just finished and it took 12 hours instead of 17.

[Manao]

Disabling deblocking is always a bad idea unless you're near saturation.Deblocking gets disable automatically a long time before reaching saturation ( saturation for h264 is around ~20 mbits for D1, it can't be reached ).

Supporting CQM isn't complicated at all. It doesn't slow down the encoding process, so there's no objective reason not to support it. But now, I'm not a chip maker...

Edit : encoding with CQM doesn't slow down the encoding process.

[Kopernikus]

x264 has different optimized routines for quantizing. if a cqm has low entries a slightly slower routine ist used to prevent overflow.

[Manao]

But only if the quantizer gets really low. And low entries means higher bitrates --> high quantizers at same bitrates, so in the overall it compensates one another ( well, almost at least )

[Sharktooth]

Deblocking gets disable automatically a long time before reaching saturation ( saturation for h264 is around ~20 mbits for D1, it can't be reached ).
sorry, wrong term... i meant "close to transparency" or a sufficient quality/bitrate to have blocks go away...

[akupenguin]

But only if the quantizer gets really low. And low entries means higher bitrates --> high quantizers at same bitrates, so in the overall it compensates one another ( well, almost at least )
The precision needed (and thus the version of quantization function used) depends on the entries in the CQM, not on the QP.

[Kopernikus]

But only if the quantizer gets really low. And low entries means higher bitrates --> high quantizers at same bitrates, so in the overall it compensates one another ( well, almost at least )

From the Mailinglist:


MMXEXT32 qaunt2x2dc can be used if cqm >= 2.
MMXEXT32 qaunt4x4dc can be used if cqm >= 4.
MMXEXT32 quant4x4 can always be used.
MMXEXT32 qaunt8x8 can be used if cqm >= 2.

MMXEXT16 qaunt2x2dc can be used if cqm >= 4.
MMXEXT16 qaunt4x4dc can be used if cqm >= 4.
MMXEXT16 quant4x4 can be used if cqm >= 4.
MMXEXT16 quant8x8 can be used if cqm >= 7.

MMX15 qaunt2x2dc can be used if cqm >= 7.
MMX15 qaunt4x4dc can be used if cqm >= 7.
MMX15 quant4x4 can be used if cqm >= 7.
MMX15 quant8x8 can be used if cqm >= 11.

And Sharktooth Matrix contains entries < 7, so the faster MMX15 routines are not used.

[Manao]

Then I'm completely wrong. Strange, I would have thought it was working as the decoding process.

But then, nothing prevents you from multiplying all values by three in Sharktooth matrices , and adapting deblocking accordingly :)

[akupenguin]

But then, nothing prevents you from multiplying all values by three in Sharktooth matrices, and adapting deblocking accordingly :)
That kills RD, since the lambdas depend only on QP, not CQM. Also confuses MB skip detection.

[Manao]

Your remark could apply to any CQM ( edit : which differs a bit too much from the flat one ). Perhaps lambda / skip decision could be made according to matrices coefficients ( normalizing thresholds / lambdas by the sum of the coefficients or something alike )

[jackiehcs]

Is this CQM suitable for anime? and also suitable for encoding with SAR?

[bond]

and also suitable for encoding with SAR?sure, cqm has nothing to do with sar

[Manao]

Actually, you can take into account the SAR by tweaking the CQM, because horizontal and vertical frequencies won't behave in the same way. So theorically, CQM makes SAR usage even more suitable. Theorically only though ( creating the proper matrix is another matter, quite complicated )

[Sharktooth]

Default quantization or flat matrices should be more appropriate for anime.

[ChrisBensch]

I'm doing 720p encodes of captured HDTV material. I'm using the Sharktooth full x264 builds and x264-MeGUI using the HQ-Slow preset at about 6Mbps. Is this matrix something that would benefit me? Or is there something else I should possibly be looking at? I don't have any issues with my encodes, but am always on the lookout for improvements.

[Sharktooth]

Well, the general rule is to use low bitrate matrices with HD sources but 6mbp is relatively high for h.264.
So yes, it may be worth a try. Post your findings though:)

[ChrisBensch]

Well, honestly, 6Mbps was just a guess, I didn't want to post a "what is a good bitrate for x264 at 720p" but since you brought it up...what might a good starting point be? The sources are all typical movies, sources are 1920x1080@29.970. I resize (and deinterlace if necessary) to 720p@23.976fps. I honestly have no idea where to begin for bitrates for these HD encodes.

[Sharktooth]

1500kbps is a good starting point for sources with good compressibility at 1280x720 @ 23.976fps.
Higher bitrates are needed if noise or grain is present or if compressibility is lower (action movies for example).
Obviosly that's subjective... but artifacts on HD stuff are much less visible.
Use a quality preset (like HQ-Slower for MeGUI) or if encoding time is a problem just rise the bitrate and use a faster preset.

[ChrisBensch]

I didn't realize that I could get away with such a low bitrate for HD content! Well, that'll be good, I tried the bits/pixel calculator on the bitrate tab of the newest GKnot and it showed I'd need a minimum of 3500Kbps to get the 0.190bits/pixel. I guess it's a bit outdated for x264? Either way, I'm stoked, now how about for DVD->x264 (my Emily Rose DVD is 720x288@23.976)?

[redfordxx]

minimum of 3500Kbps to get the 0.190bits/pixel.In general, the smaller resolution, the higher bits/pixel needed, for by downsizing you get more detail per compressed block. At least for clean sources.
Against this rule can go the fact, that with downsizing from HD to SD you can accidentally get rid of some film grain, which increases the compressibility.

As always it depends on source. I personally do not see any special reason to squeeze a full length movie 1280x??? smaller than 1 DVD size.

For instance one of my test:
wmv3 6.5Mbps 1440x820ANA -->Quite heavy denoising and deblocking --> 1280x544 --> x264 4200kbps
IIRC the avg quant was some around/below 20 with flat (not sure) w/o deblocking filter.

[redfordxx]

Sharktooth,
you have obviously a lot of testing with your matrix behind.
Do you have any experience with your matrix, on which quantizer (and below) the blocking disappears (or other artifacts --- ringing, nonsharpness). With the filter deblock off if you know --- I have to encode the HD with the filter of, otherwise the playback eats too much CPU.

[Sharktooth]

It really depends on the source.

[Adub]

Sharktooth, are you developing anymore AVC quantization matrices, like you said in your first post? I don't mean to be nosy or anything, I just use your matrix alot and I am curious if there is anything in development.

[Sharktooth]

Yes, im still experimenting with matrices. Im trying to fine tune HR and LR is in preliminary testing phase... but that's not as easy as for mpeg4 ASP:)
however i think i'll rename them... both.

[Adub]

Great! I can't wait to test these babies out!

[juerginst]

:D Great!!!

[IndieRockSteve]

1500kbps is a good starting point for sources with good compressibility at 1280x720 @ 23.976fps.
Higher bitrates are needed if noise or grain is present or if compressibility is lower (action movies for example).
Obviosly that's subjective... but artifacts on HD stuff are much less visible.
Use a quality preset (like HQ-Slower for MeGUI) or if encoding time is a problem just rise the bitrate and use a faster preset.

Interesting. I want to do 1080i->1080p encodes of my HDTV captures, I'm less concerned with file size than I am with quality though. Whats a good bitrate to start at for this? Which of the matrices in your sig should I try with my encodes?

thanks! (and thanks for all your posts on this forum, lots of good reading!)

[Manao]

If you're not concerned by the filesize, use a one pass crf encoding. Try value from 18 to 26, and choose the biggest value whose quality you still find acceptable. Then stick to that crf for all your encoding.

[Romeo_by]

Does anybody can advice a source info about creating custom matrices or may be explain basicly how to do it?

[Audionut]

Second result from a custom search.
http://forum.doom9.org/showthread.php?t=54147&highlight=matrix

[RickA]

I know this is an old thread and I do not mean to resurrect it. I just wanted to thank you for this matrix Sharktooth. Tried to PM you this instead but it said your message box was full up.

What a great matrix you have created here. It really helped clear up the 'blockiness' I was getting in dark scenes of low detail. Such as the intro to Alien as they are spelling out the letters and panning across the planet. Even with Constant Quality of 17 and a sufficiently high bitrate I still had ugly blocky artifacts before using this matrix. I realize h264 has a tendency to go blocky in low detail and low lit areas, but EQM AVC-HR really cleared those up for me.

Since this matrix works so well on darkly lit low detail movies I take it that it is also fine to use on well lit low detail movies... as a precautionary measure? I'm new to h264, been playing with MeGUI and AutoMKV for a few weeks. My goal is to put my dvds on the computer hard disk for playback via tv and home sound system - a video jukebox if you will. A file size reduction of @30% and near source quality is what I am aiming for.

Once again, sorry for posting in an old thread. I debated whether to start a new thread just to say 'Thankyou' for the work you have done.

Cheers,
Rick

[Sharktooth]

thanks for your kind words.
as the description says, EQM AVC-HR requires high bitrate and tends to force the codec to assign more bits in certain areas.
this behaviour has been discussed with the x264 devs and is not optimal, but indeed it helps a lot to avoid blocking.
yes, it works good even on "lit" details and if you feel confortable with how it looks, then feel free to use it but remember it needs more bits, that means if you encode in CRF or CQ it will take more disk space than normal.

[RickA]

Greetings Sharktooth,

Thanks for responding to my EQM AVC-HR post. I am not fully sure on what you mean when you said 'if you feel comfortable with how it looks' would you please expand more on that? My newbie eyes have only detected a reduced blockiness of h264 in dark, low detailed areas. I have just started testing with your matrix. Is there something else I am missing with using your matrix and should be aware of? You also mention the matrix behavior and h264 as not being optimal. Are you referring to compressing the file size, or picture quality?

As far as the increased file size by using your matrix that is not as much of a concern for me as keeping video quality as close to source as possible. I am basically making a movie jukebox by putting my dvd's onto computer for playback via tv and home sound system. If I get near 30% file size reduction I am happy.

Thanks again.

Cheers,
Rick

[mpiper]

Sharktooth,

I just downloaded your custom matrices, but being new to x264, I don't want to randomly use them with no idea of their purpose.

Can you possibly provide us with a simple description of suggested uses? Something like min/max bitrate, anime/clean/grainy source, fastmotion w/ high detail, details vs. banding, etc.

Thank you in advance!
Mike

[ToS_Maverick]

read two posts above yours:

thanks for your kind words.
as the description says, EQM AVC-HR requires high bitrate and tends to force the codec to assign more bits in certain areas.
this behaviour has been discussed with the x264 devs and is not optimal, but indeed it helps a lot to avoid blocking.
yes, it works good even on "lit" details and if you feel confortable with how it looks, then feel free to use it but remember it needs more bits, that means if you encode in CRF or CQ it will take more disk space than normal.


for even more information, start at the beginning of this thread.

and there's another thread about matrices, if you're eager to read:
http://forum.doom9.org/showthread.php?t=117041

[henryho_hk]

Do we have a "compressibility test" for MPEG4-AVC?

[Selur]

would it make sense?

[henryho_hk]

Somehow, unless we have some general rules describing terms like "high bitrate", "difficult materials", etc. for MPEG4-AVC (say, with respect to x.264). I am asking myself questions like "how would this movie look like if I encode it into xxxMB using x.264?" I think something like a compressibility test would help.

[Sharktooth]

compressibility test is like for xvid. except you have to use CRF 18 istead of CQ2.
also the avc matrices are really becoming obsolete and useless (at least for x264) since VAQ and Psy-RDO,

[henryho_hk]

If we are doing a sampled compressibility test, do we need to discard the frames of "artificial" scene changes?

[Sharktooth]

as a rule of thumb it would be better to discard them but the theoretical right way is just rising the patial comp.test compression value/percent by e few points to compensate the difference of partial encode scenechanges number vs the final encode scenechanges effective or approximated number.

[xxthink]

Would you like to give some idear or papers about how to train a custom quantization matrix?

[Sharktooth]

http://en.wikipedia.org/wiki/Quantization_(image_processing)#Frequency_quantization_for_image_compression
how to "train" depends on what you want to obtain once you got "how" quantization matrices work.