Understanding a Quantization Matrix

[crusty]

Ok Folks, here is an explanation of what a matrix really does.

With Xvid you can not only use different built-in Quantization Matrices, like H.263 and Mpeg, but you can also use custom matrices either made by yourself or someone else.

Since few people actually understand what a matrix does, I will try to give an explanation here that is possible to understand even if you're not a math expert.

You will all have heard about macroblocks by now. These are the 16x16 or 32x32 blocks of which a single mpeg4-frame is composed of.
These macroblocks are comprised of 4 8x8 blocks that are grouped together. These 8x8 blocks form the basis of MPEG-4 compression.

Instead of being composed of proper pixels, like a normal bitmap or a film still, a block is more like a representation of a complex formula that tries to mimic the content of the original picture as best as it can.

The Human eye is much more sensitive to changes in the brightness, or Luminance than it is in color changes.
So mpeg4 uses a type of color space in it's file structure that assigns less bits to color changes than it does to brightness changes.

A 8x8 block is not made up of pixels but it is made up of a single value which represents the average brightness (or color) value, and all the remaining values are mathematical representations of the amount of variation from this average value for the whole block.
To put it in other words, you have one basic mean, or average value, and all the variation, or detail of the picture , is represented by the end result of a certain complex formula.
The other places in the 8x8 block, which we invariably but inaccurately call pixels, represent different types of detail, and especially the variation of this detail.

Let's take a look at one block:

X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X

The first place in the upper left corner represents the average, or mean value. So if the whole block is dark brown or light red on average, this place says so.
Going from this place to the right or down, we get representations of the amount of variation from this value.
Now this is hard to grasp, so pay attention.
When going from left to right, or from top to bottom, the higher the amount of detail gets.
If you take the original picture of 8x8 bits, the amount of detail is transformed into values depending on a certain frequency at which the detail is present.
Finer detail is represented by higher frequencies.
So, the further you go to the right in the block, the higher the frequency (or the finer the detail).
Let's take three examples:
An 8x8 picture with one big iron bar in it has little detail, so it has a very low frequency.
An 8x8 picture with four broomsticks in it has some detail, so it has a medium frequency.
An 8x8 picture with rain, a cornfield or hair in it, has lots of detail in it and so it has a very high frequency.
As you can guess by now, the values in a block represent both the horizontal and vertical frequency in the original picture.
The formula that does the translation or transformation from detail to frequency is called Discrete Cosinus Transformation or DCT.

So when looking at a block in the way we have just gotten to understand, we can see the next things:


Average brightness and color of the whole block
I
I Low frequency (bigger)detail
I I
I I Medium frequency (normal)detail
I I I
I I I Higher Frequency (fine)
I I I detail
I I I I
X--X--X--X--X--X--X--X
X--\--X--X--X--X--X--X
Low Frequency detail--- X--X--\--X--X--X--X--X
X--X--X--\--X--X--X--X
Medium Frequency detail X--X--X--X--\--X--X--X
X--X--X--X--X--\--X--X
X--X--X--X--X--X--\--X
High Frequency detail---X--X--X--X--X--X--X--A
I
I
Mathematical representation of the finest
detail, both horizontally and vertically

EDIT: Finally used the code tag...

Ok, now we understand how a block is built.
So how come Quantization matrices into play?

A Quantization matrix (QM from now on)looks something like this:

08 16 19 22 26 27 29 34
16 16 22 24 27 29 34 37
19 22 26 27 29 34 34 38
22 22 26 27 29 34 37 40
22 26 27 29 32 35 40 48
26 27 29 32 35 40 48 58
26 27 29 34 38 46 56 69
27 29 35 38 46 56 69 83

Now there is actually a rather complex proces behind this, but I'll try to describe it in a simple manner:
Every value in the QM is the threshold for the DCT detail-to-frequency translation.
All detail below the threshold will not be regarded as detail and will NOT be encoded. It will simply be discarded.
Now you can understand why Xvid is a so called lossy codec.
It throws detail away. The amount of detail thrown away is determined by the QM.
As you can see, the farther to the right and the farther to the bottom the higher the threshold gets.
The finer the detail, the higher the detail has to stand out of the rest of the picture to be encoded and not thrown away.

So say you have a picture of a girl with long blond hair standing in front of a very light gray wall and behind two prison bars. (don't we all love to see that now) :D
Off course it's hard to portray that in an 8x8 picture but bear with me for a minute.

The blond girl and the wall will have little contrast between them, so the difference between the average values for brightness and contrast and the maximum values will not be that much.
The values will be lower and will not go that often over the threshold. This would mean less difference that has to be encoded and the picture will have high compressability.
If the wall had been black, the contrast would be much higher and the difference between the average value (sort-of-grey) and the extremities (blond and black) would be much higher. So much more values would have gone over the threshold and would have to be encoded. So higher contrasting scenes result in lower compressability, which we already know offcourse.
Now the two black prison bars in front of the girl are detail, all be it not very fine. So they get a low frequency and they will be encoded if their difference from the average goes over the threshold.
So assuming they're not blond, they will be encoded. :cool:
Now the girl has hair, and the texture of hair is off course very fine. So the hair has a lot of detail and gets a very high frequency.
As you can see in our QM, the threshold for high frequencies is much higher than for low frequencies. So the fine detail (the hair) has to differ much more from the average values to be encoded.
So unless the difference is very high, which in this case we assume isn't, the details of the hair won't be encoded.
The matter would be different if she has something like coloured streaks in her hair which would increase the contrast.

So the end result is that the finer the detail, the bigger the contrast of this detail needs to be from the average values of the picture, to be encoded.
This is offcourse on a per-block basis and not on a picture as a whole, which generally consists of more than one 8x8 block. Let's hope you can see through the simplification.

Now you can understand why some matrices soften the picture, like H.263, while others like Mpeg produce a sharper picture.
The values in one QM simply give finer detail a lower threshold and are therefore more likely to encode finer detail, at the price of compressability.
You can also see that the QM that I took as an example isn't a very high compressability matrix; the values are rather low in general.

Some other points:
-End credits usually have very little detail, so you could design a QM especially for this, with VERY high compressability.

-A heavy compression matrix simply ups all the finer detail values so less of that is compressed.

-You could design specific matrices for specific type of content.
You could design matrices especially for sci-fi space adventures, Anime and animation movies, and jungle scenes.

-If you know the exact frequency of interlacing artifacts, you could up their threshold to filter them out!

-Same might work for other types of noise and artifacts.

-I don't know if the one QM is meant for both luminance and color information, I assume it does, but separate QM's for luminance and color would produce higher tweakability. (Don't know if it would be Mpeg4-compliant though, or if it's possible at all).

--------------------------------------------------------------------
Well that's it folks!
Hope I didn't make too many errors, and please correct me if and where I'm wrong.

Cheers,
Crusty

EDIT:
Edited some spelling errors
Removed part about Modulated QM
Reedit: altered macroblock to block. A macroblock is 4 8x8 blocks grouped together.

[mf]

Try using the [code] tag for nonproportional text.

[Acaila]

Modulated QM uses different matrices for different types of scenes within one clip. For different parts of the clip, it takes the QM with the highest compression for that particular clip.
- Modulated uses the MPEG matrix when a frame's quantizer is <=3, and the H.263 matrix when the quantizer is >=4.

- New Modulated HQ uses the exact opposite, so the H.263 matrix when a frame's quantizer is <=3, and the MPEG matrix when the quantizer is >=4.

So it's not as smart as you might have hoped, but it's still quite effective.

[crusty]

Ok, removed that part...
Rest of it is OK?

[Acaila]

As far as I know everything else is exactly correct :).

[snowbeach]

Very good explanation crusty! Maybe I should add this to my guide! If I am allowed? :D

[MrBunny]

Originally posted by crusty
Some other points:
-End credits usually have very little detail, so you could design a QM especially for this, with VERY high compressability.

As Syskin says in this post: http://forum.doom9.org/showthread.php?s=&postid=305827#post305827 changing quant types (in this case to a new quant matrix for the credits) is not MPEG-4 compliant, though it is xvid compliant if on an i-frame. Just a little something to note.

Other than that, it's a really nice comprehensive guide. You also might want to make a quick reference to quantization error (from quantization rounding). It doesn't have much to do with compressibility, but might affect image quality.

Keep up the good work :)

[crusty]

@Snowbeach:
Permission granted :D :D
Have fun with it.

MrBunny
You also might want to make a quick reference to quantization error (from quantization rounding). It doesn't have much to do with compressibility, but might affect image quality.

I would if I'd understand it. I know that rounding exists but I don't know the effect. :confused:

BTW, is your first name Energizer ? :D

[MrBunny]

From what I recall of quantization and inverse quantization:

Quantization: Quantized coeff = Floor(DCT coeff/QM coeff)
Inverse Quantization: Reconstructed coeff = (Quantized coeff + 0.5) * QM coeff

The problem arises with the rounding error involved in the floor and using the midpoint value of each coefficient to reconstruct.

Consider the DCT coefficients 75, 88, and 99 with a QM coefficient of 25. Each will result in a quantized coeffient of 3 and each will reconstruct to 87.5. As you can see, there is significant error.
However, I don't visualize well in the frequency domain and don't know how much nor exactly the type of effect it would have in the spacial domain (after iDCT).

I guess at least one of the lessons that should be taken away is that you should be really careful with putting really high coefficients in a QM, since large coeffs will result in large error (when the QM fails to zero that particular coefficient). If you do want to simply zero a certain number of the highest frequency values, you might want to have a look at the avisynth based DCTfilter (http://forum.doom9.org/showthread.php?s=&threadid=38539 and http://forum.doom9.org/showthread.php?s=&threadid=45695) then use a "normal" quant type.

I think quantization error is one of the reasons QMs look like they do (smaller values for low freq and increasingly larger for higher freq). You need to have the most accurate reconstruction possible of the most important coefficients (thus small quants/low error for low freq), whereas it is less important for the less important high freq coeffs (high quants for high freq with hopes of zeroing).

I hope this isn't too confusing ;)

[Acaila]

@MrBunny:

IIRC JPEG uses flooring, but MPEG uses rounding after quantization. Mainly because rounding to nearest integer results in a smaller quantization error.

...is that you should be really careful with putting really high coefficients in a QM, since large coeffs will result in large errorBut high values in the matrix will drive more coefficients to zero. Because quantization matrix values are 8-bit values, I believe the highest you can put in there is 256. If you want a certain coefficient to become zero, why shouldn't you just put 256 in that position?

[MrBunny]

@Acaila

I said to be careful about doing it, not to not do it at all :)
And I was thinking in terms of JPEG compression, I was unaware that MPEG handled quantization differently.

Can DCTed coefficients be larger than 256 (Or i just 128 if it rounds)? If it can, then even a QM w/ 256 coefficient isn't guaranteed to zero. In that case I still think that using the DCTfilter to zero coefficients is a better alternative than using a custom quant matrix. Otherwise, putting 256 should be sufficient.

[Acaila]

Yes in theory DCT coefficients can easily be larger than 256, however except for the top-left DC coefficient almost all other AC coefficients will be quite low in practice. So I don't see anything wrong in using that value in a matrix, however since I've never tried it out myself I could be wrong of course :).

[crusty]

So higher QM values will result in more variation thrown away in the first place, but will work with a bigger margin of error when it doesn't thrown away variation.
But as long as you use the higher values only for finer detail, it really doesn't matter that much now does it?
I don't know for sure, but I assume that higher rounding errors are less of a problem for finer detail.
The end effect would be that finer detail, if allowed through by the QM value, would simply have a bigger luminance error.
Is the human eye less sensitive for the luminance of fine detail?

The only type of scenes that I can think of that would be affected, off the top of my head, would be dark skies with stars in them or 'proper' space scenes.
This is because a real night sky with stars in them has high contrast fine detail.
The stars are basically points, but there is a difference in luminance that is very easily spotted. They vary easily from the average value, which would be the mean value of the sky (very dark blue), so they would easily get above even a high treshold, but they would then have a pretty large margin of error in luminance.
If you wanted to portray this type of scene realistically, you would have to find out the DCT frequency of the stars and then lower the QM value to allow for a smaller margin of error.
Otherwise the stars will look washed out and you might loose the 'twinkling' effect that a real-life night sky has.

Mind you, DVD's use mpeg2 compression which also uses DCT and macroblocks, so unless they specifically used a smaller QM value for the conversion from movie to DVD, this type of scene will look washed out on DVD anyway.
Also, most 'artificially' created (Special Effects) space scenes don't come close to realistically portraying stars. When astronauts first went up into space, they found the stars to be very bright and to have none of the 'twinkling' effect whatsoever, which offcourse is created by the atmosphere.
But in space the stars are so bright compared to the darkness of space that no TV or monitor can get close to portaying this kind of contrast.

Another thing;
I think that especially for anime movies many of the QM values could be higher, because contrast is usually much higher. I don't know for sure because I have never done an anime movie before, but I'll do Akira probably somewhere this month and I'll try it with different custom matrices.
I suspect a QM with lower values to low frequencies and higher to high frequencies will give a better result. And because anime has in general much sharper contrast than real-life footage, all the values could probably be higher to start with.
It might also be possible to find a 'sweet spot' for each different anime movie or serie because one movie could use finer lines than the other.
To all: Please try custom matrices based on this and report the effect.


Question to developers:
Is mosquito noise around sharp edges created by using too high frequencies for these edges?
I can imagine a very bright thin line getting a higher frequency than a very bright thick line, ergo the macroblock with a thin line getting additional thin lines in decoding because of the higher frequency used.
Is this correct or is this just plain nonsense?
If it's correct, then I assume that some form of postprocessing filter could be made to detect this type of noise and remove it, not just on Xvid, but on all Mpeg4.

[mf]

Originally posted by crusty
Question to developers:
Is mosquito noise around sharp edges created by using too high frequencies for these edges?
I can imagine a very bright thin line getting a higher frequency than a very bright thick line, ergo the macroblock with a thin line getting additional thin lines in decoding because of the higher frequency used.
Is this correct or is this just plain nonsense?
If it's correct, then I assume that some form of postprocessing filter could be made to detect this type of noise and remove it, not just on Xvid, but on all Mpeg4.
Hah! finally I can engage in this conversation as well. Ringing around sharp edges is created by dropping coefficients.

[crusty]

Dropping coefficients ?

Could you elaborate on that ?

[MrBunny]

I believe he means zeroing coefficients (as we've been calling it).

It makes sense. Edges are typically not very "average" with respect to its surroundings. Thus it is not well covered by the DC coefficient and other lower freq components and generally requires a number of the higher freq components to represent it correctly. When some of these components get zeroed, the edge gets eroded (less sharp, less defined and unsettles the surrounding area) resulting in the mosquito noise we all hate.

[Didée]

After so much elaborating about which cells of the quantization table correspond to which frequencies, and to what kind of image detail, I feel the need to remind of this:

All of the above written is generally correct for intra-frames, resp. intra-blocks, only.
In I-frames the actual image gets coded, and therefore all the frequency stuff is directly related to image detail.

Now, in the usual way we all encode mpeg-4, most of the video stream (~ 98%) is P- and/or B-frames. And for these, its a quite different story.
What gets DCT'ed is the image after the motion compensation, and therefore it is not possible to directly draw the conclusion "fine detail" -> "high frequency", or the other way round. For example, it is perfectly possible that you have a block with pretty fine detail, but that block gets very nicely catched by ME, so the result that gets DCT'ed consists only out of low frequencies!

Because of that, the relation between quantization coefficients and image detail is not of that kind like our straight imagination would suggest. Keep that in mind!

Another point in this context:
For the above reasons, it makes a big difference if you "zero-out" some frequencies by either the encoder's quantization table (so: after ME), or by Tom's DCTfilter (before ME).
Zeroing by DCTfilter will directly remove image detail, whereas zeroing by encoder's matrix will remove the differences that are left over by ME. That are two different pairs of shoes!


Regards

Didée

[kilg0r3]

So what do we learn from this regarding the relation between an intra cel and the corresponding pframe cell?

After all there are matrices that distinguish between these two. Other, however, have the same value for I-and P-cells. Any general suggestion what might be a wiser way to go?

[crusty]

MrBunny:
When some of these components get zeroed, the edge gets eroded (less sharp, less defined and unsettles the surrounding area) resulting in the mosquito noise we all hate.

So, if higher QM values for higher frequencies results in less accurate edge compression and more mosquito noise, I suggest the following to the developers:
Insert some form of 'edge detection algorithm' between DCT and Quantization, at least for the higher frequencies. That way, if normal picture content is below the threshold, it will not be encoded, but if there is an edge in the picture, the resulting DCT value would get an additional bonus, so it would go over the treshold easier, therefore reducing noise.
It would probably hurt compressability a bit, but it would result in less mosquito noise. It would be part of the encoding process only, so it would not break mpeg4 compatability.
Call it 'DCT edge detection' or something and allow the user to adjust it's 'DCT edge bonus'.
The question is offcourse whether the improvement in quality would counter the cost in bitrate, but there's only one way to find out....program it and let us test it.
Nic, SysKin, Koepi and others....any thoughts on this?

Didée:
Zeroing by DCTfilter will directly remove image detail, whereas zeroing by encoder's matrix will remove the differences that are left over by ME. That are two different pairs of shoes!

This leads me to conclude that, since Qpel works with smaller motions than Hpel, Qpel will be less efficient when using high QM values, because the effect of Qpel will be nullified by the high thresholds of the QM. The minimum use of bits by Qpel will remain the same tho.
Am I right in this?

What gets DCT'ed is the image after the motion compensation, and therefore it is not possible to directly draw the conclusion "fine detail" -> "high frequency", or the other way round. For example, it is perfectly possible that you have a block with pretty fine detail, but that block gets very nicely catched by ME, so the result that gets DCT'ed consists only out of low frequencies!

Isn't it so that when a pictures content is catched by ME, the corresponding macroblock doesn't contain ANY texture bits, but just motion vectors?

Kilg0r3:
After all there are matrices that distinguish between these two. Other, however, have the same value for I-and P-cells. Any general suggestion what might be a wiser way to go?

If DCT in P- and B-frames generally results in low frequencies, low settings for high frequencies seem to me to have little or no influence to the end result. Maybe a higher threshold for these could lead to higher compressability without too much quality loss.
Also, if high QM values for high frequencies seem to counter the effect of Qpel, I would suggest not using Qpel with high compression matrices, since the amount of bits requiered by the Qpel technique may not be countered by the beneficial effect of Qpel.
The higher rounding errors created by high QM values would then also make Qpel less interesting.

So, to sum up:
1-higher QM values will lead to bigger rounding errors, nullifying subtle luminance variations in Keyframes.
2-Qpel 'might' be useless with high QM values for high frequencies. Don't use Qpel with high compression QM's, at least not with high compression P/B-frame QM's.
3-Using high values for high frequencies in B/P-frame QM could improve compressability. Lowering the high frequency QM values in I-frames could be used to add bits to fine detail in keyframes, possibly creating better values for the B/P-frame process to work with.
4-using the same QM for both Inter- and Intra-frames is less efficient than using different QM's, because of the difference in method used.
Any discussion on these points would be greatly appreciated.

Also, does anybody have the QM's for DivX 3 Low-motion and Fast-motion?
I wonder if they are any different.
It would also be very interesting to see the effects of different QM's on low and high-motion content.

Too bad I'm trying to flush the content of my harddrive because I'm installing a new motherboard this week....otherwise I would be experimenting with this stuff all day long. :D
Only 30 movies to go.....;)

[Acaila]

This leads me to conclude that, since Qpel works with smaller motions than Hpel, Qpel will be less efficient when using high QM values, because the effect of Qpel will be nullified by the high thresholds of the QM. The minimum use of bits by Qpel will remain the same tho.
Am I right in this?
I don't know why you brought QPel into this, for it has next to nothing to do with quantization. QPel increases the sensitivity of Motion Estimation. Once that process is finished the motion estimated frame is subtracted from the current frame with the result being a residual frame. It in on this residual frame that quantization is applied. QPel works before QM, and so there's no way that high QM could nullify the effect of QPel (more like the other way around if anything).
QPel helps to catch more motion, so it decreases the size of the residual frame, and thus also decreases the amount of effect of the QM. If ME were to work for 100% the residual frame would be empty and no quantization (of the residual frame anyway) could take place.

3-Using high values for high frequencies in B/P-frame QM could improve compressability.And that's exactly what all matrices already do...
And since using too high values cause ringing as mentioned before, I believe the standard matrices are already quite well made. The only reason I could see anyone using a custom QM is for increasing the amount of detail/size of a video (since for higher compression you could just as well use a higher quant, no need for a custom matrix).

Also, does anybody have the QM's for DivX 3 Low-motion and Fast-motion?As far as I know the two flavors of DivX 3 have identical quantization matrices. The only difference is the quant ranges that they can use. I believe low-motion could use anything 2-31, while the high-motion codec could only use 4-31, but I'm not sure if these ranges are correct.

[crusty]

I don't know why you brought QPel into this,
I figured that since they both work with the precision of motion, there has to be some sort of interaction.
And that's exactly what all matrices already do...And since using too high values cause ringing as mentioned before, I believe the standard matrices are already quite well made.
Offcourse they are, but there certainly are certain types of film that might improve with a tweaked QM, like cartoons.

What do you think of the edge detection in the codec I suggested?

[manono]

Hi Acaila-

while the high-motion codec could only use 4-31

It's even worse than that-the lowest quant that Hi-Mo uses is 5. Minor point though, in your informative series of posts in this thread.

[Acaila]

What do you think of the edge detection in the codec I suggested?For one thing it would have to be done before DCT because edges no longer exist in the transformed frame (I think). All spatial information is gone after DCT, only frequency remains; high frequency can just as well mean noise or edges, there's no way to know after the DCT.
Giving a bonus to egdes sounds to me very much like sharpening, and that's not always a good idea, it either creates ringing or increases edge enhancement (->halos). But other than that I have no idea if it would be a good idea in general.

Maybe some real developer can say if it's possible/worthwhile.

Ps. Thanks Manono :)

EDIT: Yes, edge detection using the DCT coefficients (or another transform for that matter) seems possible but from what I have read it's very complicated too.

[Didée]

Originally posted by Acaila
And since using too high values cause ringing as mentioned before, I believe the standard matrices are already quite well made. The only reason I could see anyone using a custom QM is for increasing the amount of detail/size of a video (since for higher compression you could just as well use a higher quant, no need for a custom matrix).[/B]
Well, let me bore you with my points in using custom matrices.

The standard matrix is well designed as an all-purpose-matrix. It delivers reasonable results with all quality ranges. But obviously, it is not at all optimized for either hi-quality or hi-compression scenarios. For both, a good custom one will fit better.
On the hi-quality side, we already had the discussion "better quality at max quality" (here (http://forum.doom9.org/showthread.php?s=&threadid=40588&highlight=better+quality+max)). Quant-2 doesn't deliver super-high quality, and quant-1 ... you know.
Moreover, I think that the bigger coeffs for the high frequencies are mostly there to deal with noise, not with image detail. (Well, intended or not, it IS like that.) But if you have a well-denoised picture, then your interest in keeping high frequencies will raise quickly, proportional to the ratio of win/loss forced by keeping them. Hah, what a sentence ;)

Furthermore, there is an additional benefit for 2-pass scenarios:
The codec can scale the bitrate much smoother.
Usually, for really good quality, we're still aiming for an average quantizer of '3' or better - at least for P-frames. Right?
Now, with the standard matrix we have that "big jump" in filesize, or bitrate, between quant-2 and quant-3. Therefore, when the codec is working with only q2 and q3, the bitrate is not distributed smoothly: one expensive q2 equals to many cheap q3's, and the quality improvement from those seldom q2's is hard to notice. There should be something between q2 and q3 ...

ATM, I use the following matrix almost all the time. Scales well with all high-bitrate to better-medium-bitrate scenarios - but only for well-denoised material. I always use PixieDust(2|3), seldom with a little caliming-down the noise beforehand by light Convolution3D or Fluxsmooth. Undot() of course.
Since attaching seems down again, I'll do a little more typing. Perhaps you will, too.
# "SixOfNine"

08 10 11 12 12 13 14 15 10 10 11 12 12 13 14 15
10 11 12 13 13 15 15 16 10 11 12 13 14 14 15 16
11 12 12 14 15 15 16 17 11 12 12 14 14 15 16 17
12 13 14 15 15 16 17 18 12 13 14 14 15 16 17 18
12 13 15 15 16 17 18 19 12 14 14 15 16 17 18 19
13 15 15 16 17 18 19 19 13 14 15 16 17 18 19 20
14 15 16 17 18 19 19 20 14 15 16 17 18 19 20 20
15 16 17 18 19 19 20 20 15 16 17 18 19 20 20 20
- This' matrix quantizer range q2-q8 eaquals about the standard matrix range 'q1.4'-'q4.5'
- q3 is a tad better as q2(standard)
- q4 is between q2 and q3 (std)
- q5 is a little weaker as q3 (std)
- ...

For the same bitrate range, the codec simply has more quantizers available for scaling.


Regards

Didée

[JimiK]

@crusty
I agree with Acaila, that imho there is no use for detecting edges. Even if would be possible, what benefit would you have? You would know there is an edge, and now? This is not meant to sound rude.
Now I always thought a bit different about quantizer matrices. Why do they have higher coefficients for higher frequencies (lower right of the matrix)? Well, I thought it is, because you cannot notice a loss in detail that easily. That's why you can quantize it more. It's dependend on the human visual system. Ever noticed macroblocks in evenly colored areas? Of course that's because the whole block gets the same value which is another problem. But why do we see these blocks, even though the difference is not big (blue block in the sky next to another blue block with almost the same color). That's because the human eye notices differences in a colorgradient much easier than differences between two completely different colors.
So it's not possible to quantize low frequencies even more, you would easily notice. You can of course quantize high frequencies less and get more details and maybe less mosquito noise, but you would have to pay with a much higher need of bitrate, while it's quite hard to see the difference to the other picture.
Again (as I always say): I don't now much about QM's, so maybe that's wrong, but I think I read that somewhere.
@Didee
Thats interesting, with the more even distribution. But there is only a slight difference between the quantization of low to high frequencies (ratio 2:1) which would leed to the problem (bitrate) I just mentioned. Strange, didn't you say you put LOTR on one CD with decent quality? And another point: doesn't this matrix cause problems with ffdshow? As far as I know, it does not like values below 16.
Best regards,
JimiK

[vinouz]

Theoretically you could notice edges on the DCT, as it produces a fair amount of odd coefficients. Computing pow(odd*odd) - pow(even*even) could maybe give a serious hint about edges. And then the corresponding matrix could be one special for edges (I once before talked of it).

Odd armonics code for a square signal (multiplied by their order's inverse). The only thing is I don't know how this pure square 'signature' resists to noises and geometric distortions. But sure it's good to take a little look at it.

[Acaila]

@Didee:

JimiK made some valid points there. I agree with him that the coefficients should not be scaled so linearly as you've done in your matrices, because high frequencies are harder to notice than low. With those matrices you rely almost solely on quantizers to scale the bitrate, while I believe you should rely on both QM and quantizers. What I mean with that is that a matrix with higher values in the bottom-right and lower in the top-left should give better scaling (with higher quality possible at quant 2 and less filesize distance between quants, see below). Once a frame gets encoded with higher quantizers you don't want to loose (relatively) just as many low frequencies as high ones because the low have a lot more impact on visual quality.

Also ffdshow indeed has problems with values less than 16, however as far as I can tell it only has problems with the top-left value being lower than 16. Whenever I put a 16 in the top-left corner everything played just fine no matter how low the other values were.

Something else I've noticed is that VHQ doesn't seem to work very well (lower PSNR) on custom matrices compared to H.263 and MPEG. Can anyone confirm this?

The matrices I would recommend:

Intra: Inter:
08 09 10 11 12 13 14 15 16 08 10 12 14 16 18 20
09 10 11 12 13 14 15 16 08 10 12 14 16 18 20 22
10 11 12 13 14 15 16 17 10 12 14 16 18 20 22 24
11 12 13 14 15 16 17 18 12 14 16 18 20 22 24 27
12 13 14 15 16 17 18 19 14 16 18 20 22 24 27 30
13 14 15 16 17 18 19 20 16 18 20 22 24 27 30 33
14 15 16 17 18 19 20 20 18 20 22 24 27 30 33 36
15 16 17 18 19 20 20 20 20 22 24 27 30 33 36 40
Filesize and PSNR are distributed as below (high to low):

Matrix Quantizer
Mine 2
H.263 2
Mine 3
Mine 4
H.263 3
Mine 5
Mine 6
H.263 4
Mine 7
Mine 8
H.263 5
As you can see this gives better bitrate scaling just like Didee suggested and the option for better quality at quant 2. It also doesn't scale low and high fequencies linearly because I don't think that's right.

[ssaga]

Acaila's QM has a very interesting feature, the number on the same
line are the same, does it have any special meaning?

Another question, since the right site and the bottom site both mean higher frenquency, what's the difference between the numbers on the same diagonal? (such as (3,5) vs (5,3))

[Acaila]

Acaila's QM has a very interesting feature, the number on the same
line are the same, does it have any special meaning?
I like symmetry :).

Another question, since the right site and the bottom site both mean higher frenquency, what's the difference between the numbers on the same diagonal? (such as (3,5) vs (5,3))From left to right is increasing horizontal frequency. From top to bottom is increasing vertical frequency.

[crusty]

JimiK:
Ever noticed macroblocks in evenly colored areas? Of course that's because the whole block gets the same value which is another problem.
Well, I read in one thread that Chroma Motion Estimation is designed especially for less color-blocking in almost evenly colored spaces.
I don't know exactly which thread it was, try using the search.

Acaila:
For one thing it would have to be done before DCT because edges no longer exist in the transformed frame (I think). All spatial information is gone after DCT, only frequency remains; high frequency can just as well mean noise or edges, there's no way to know after the DCT.
Well I already knew that it would have to happen before DCT, perhaps I was not clear enough on that, no problem.

Giving a bonus to egdes sounds to me very much like sharpening, and that's not always a good idea, it either creates ringing or increases edge enhancement

Well the idea was really to decrease compression artifacts.
Giving a bonus to edges does seem like sharpening on first sight, but since we would be using it for edge enhancing anyway, that's basically what we would be doing. For normal or oversharpened movies this would not come in handy, but it would be good for movies that are too soft. I personally like sharpness and crispness a lot.
And if you think about it, it's only the REAL edges, determined by the to-be-programmed edge enhancement, that get the bonus. So you wouldn't be adding noise or artifacts, just enhancing the real footage.

But I was thinking a bit more about what would really be usefull, and I figured something else.
Maybe edges don't need a bonus on the current frequency like this:
Edge detection --> determines bonus for DCT coefficient --> DCT --> DCT coefficent gets bonus added --> final value goes trough QM......
Which was what I first had in mind...
But this would indeed increase the visibility of the edges but would not reduce the noice.
Instead I am now thinking about assigning edges a LOWER frequency based on an edge detection bonus than they would have gotten without the edge detection.
If mosquito noise is created by the codec thinking there is TOO MUCH detail (ergo, noise) than there really is, then assigning a lower frequency would reduce the noise now, wouldn't it?
So instead of the edge detection bonus upping the DCT value for the determined frequencies' treshold, it would now instead assign the macroblock a different frequency, based on a more precise estimate of the real content, created by the edge detection algorithm.
So it would instead look more like this:

Edge detection --> assigning extra (internal to codec) value to edges -->
DCT algorithm -->assigning different frequency to macroblock determined by internal value --> QM

It would probably be hella slow, but you might get right of a lot of noise :D
Just some food for thoughts...

Vinouz:
And then the corresponding matrix could be one special for edges
Wouldn't that entail more than one QM per frame, and is that a) possible and b) Mpeg4 compliant?

Didée:
The codec can scale the bitrate much smoother.
Could you elaborate a bit more on that? I'm not entirely sure on how that would be affected by a different QM.
For the same bitrate range, the codec simply has more quantizers available for scaling.
So you're saying that by lowering the QM values, you give the codec more freedom to allocate bits when needed and when not? How good do you find this effect in practice?
I always use PixieDust(2|3)
Slightly off-topic:
Pixiedust created artifacts with me when using it at resolutions lower than 576xXXX. Have you experienced the same or haven't you used Dust with that low resolutions?
/slightly off-topic.

Acaila:
Also ffdshow indeed has problems with values less than 16, however as far as I can tell it only has problems with the top-left value being lower than 16. Whenever I put a 16 in the top-left corner everything played just fine no matter how low the other values were.
So what DOES happen when you use a value less than 16? Does it refuse to play or do you get artifacts?

Also @ Acaila:
I see that your QM for P/B-frames has twice the values for the higher frequencies than your I-frame QM. This would result in your B/P-frames having a bigger bias to low frequencies.
Compared to the standard H263 or mpeg QM, did you adjust the B-frame settings (Q-ratio, offset, treshold etc) to compensate for different end result created by your custom QM?

Acaila:
From left to right is increasing horizontal frequency. From top to bottom is increasing vertical frequency
So from left to right it goes something like: 1 prison bar, 2 prison bars, 4 prison bars, etc etc.
And from top to bottom it goes something like: 1 stairstep, 2 steps, 4 steps, etc etc.
Right?

Also it would not always be prudent to use a symmetrical QM, based on the type of footage.
Say you've got a jungle scene. Most of this would be trees and leefs.
I'd say that would have a bias towards vertical objects, ergo a lot of horizontal frequencies (think about the texture on the trees).
And if you've got a scene with a lot of stairs in it, it would be a lot of horizontal objects, so a lot of vertical frequencies.
Offcourse these are normally just small clips in an entire movie, but I hope it proves the point. :D

[Acaila]

So what DOES happen when you use a value less than 16? Does it refuse to play or do you get artifacts?It's like pieces of the frame stay put while others start moving around. All I know is it's ugly :).

Compared to the standard H263 or mpeg QM, did you adjust the B-frame settings (Q-ratio, offset, treshold etc) to compensate for different end result created by your custom QM?
I haven't tested B-frames yet with that matrix. It's mostly theoretical, I haven't done any extensive testing with it, but it seemed to perform great on what few tests I have done.

Also it would not always be prudent to use a symmetrical QM, based on the type of footage.
Say you've got a jungle scene. Most of this would be trees and leefs.
I'd say that would have a bias towards vertical objects, ergo a lot of horizontal frequencies (think about the texture on the trees).
And if you've got a scene with a lot of stairs in it, it would be a lot of horizontal objects, so a lot of vertical frequencies.
Offcourse these are normally just small clips in an entire movie, but I hope it proves the point. The trees create vertical frequencies, the leaves create horizontal frequencies. So you see things are not as simple as that :). Some scenes in a video might have more horizontal other scenes might have more vertical frequencies. So the best matrix is one that can accomodate for both types equally. That is why I always make my matrices symmetrical. If we could have something that creates QM's on the fly optimized for each frame we wouldn't go through such trouble to make a good QM, now would we? We can only have one QM, so it must be optimal for as many different scenes as possible, hence symmetrical.

As for your custom edge enhancement theory I'm still not convinced it is actually useful or will even have a different effect from sharpening for that matter. Edge detecting during filtering: very useful. Edge detection during encoding: doubtful. I believe it's not the codec's job to reduce noise (unless it is done with preprocessing filters).
That doesn't mean it might not be useful, just that this is my opinion.

[Didée]

Back from some testing. However, there is much more to test and compare than available time :(

I took 1500 frames of a 5%-snip through chapter 2 and 3 of LOTR Fellowship SEE, with a snipsize of 100 frames (4s, PAL).

Treatment was

crop(8,80,-8,-80).undot().LanczosResize(640,272).undot()
\ .ConvertToYUY2().pixiedust(limit=2).ConvertToYV12()

XviD (14052003-1) settings were basicly default, +chroME +qpel +chromaOptimizer. And custom quants, of course ...
Destination was 9200 kB, what was 75% of the 1st-pass with standard-mpeg.

Originally posted by trbarry, some time ago
Ah, a table!


| | PSNR PSNR PSNR | MeanAbsDev |
| 1st-pass | min avg max | (avg) | q-distrib.
============================================================================
Standard | 12542 kB | 41.6922 45.5240 50.1203 | 0.9432 | q2:541
| | | | q3:956
| | | | q4:4
----------------------------------------------------------------------------
SixOfNine | 20544 kB | 41.5405 45.5355 49.9382 | 0.9384 | q3:388
| | | | q4:970
| | | | q5:138
| | | | q6:5
----------------------------------------------------------------------------
Acaila | 19438 kB | 40.9498 45.0723 50.1547 | 0.9900 | q3:466
| | | | q4:906
| | | | q5:128
| | | | q6:1
Acaila,
my matrix is not so flat as you may think. In fact, SixOfNine was originally designed to behave very much like the standard matrix, just to get a differently scaled quantizer range in the end.
I also tested around with bigger values in the critical upper-left cell some time back. But I got the impression of flat areas becoming more blocky by that strategy, and so I dropped the idea again. Also it has its pro's in reducing the raised bitrate. Erh, what?
And I don't forced to let ffdshow decode my videos ... it is sufficient as postprocessor only.

Too tired for analyzing now ... perhaps tomorrow with more test results.

Good night, gentlemen,
please try to not post more than 2-3 additional pages on this thread in the next 24 hours, okay? (Crusty!! Carry on!)

Didée

[LoKi128]

Well, since we are talking about matrices here, I have a question.

When encoding an anamorphic source... would a matrix with higher detail in the horizontal axis provide better quality? Most matrices that I've seen are pretty even in both the horizontal and vertical... but in an anamorphic encode it is important to keep the higher horizontal frequencies, since sqishing the image in essense shifts any lower frequencies up.

[Defiler]

Acaila:
Ouch, the first pass is huge with your matrix. :)

[MrBunny]

Originally posted by Defiler
Acaila:
Ouch, the first pass is huge with your matrix. :)
There isn't an issue with large first pass sizes. In a two pass situation, a large first pass will simply cause a higher average quantizer. However, since the Acaila and Six of Nine QMs have relatively small coefficients compared to others QMs, even if the average quantizer is larger (as shown to be true in Didée's table), the actually value used for quantization is still comprable to "normal" QMs.

and back a few posts...
Originally posted by crusty
Wouldn't that entail more than one QM per frame, and is that a) possible and b) Mpeg4 compliant?
As I'd mentioned in a post in the previous page, Syskin talks about switching QMs on the fly in this post: http://forum.doom9.org/showthread.p...5827#post305827

Keep up the interesting discussion (Sorry if we overwhelm you Didée :D )

[Acaila]

@Didee:

I just finished a dozen tests on a 5000 frame Collosseum scene from Gladiator and my findings support yours for 100%, so I won't post them here. In 2-pass my matrix had a (0.5 dB) lower PSNR than the standard or SevenOfNine matrices on all occasions.

But it looked nice in theory :D.

I've also tried some variations on your SevenOfNine matrix, but no matter what I tried, PSNR always dropped. So I believe you've got quite a good matrix there ;).
Substituting the top-left value 8 with a 16 (in the interframe) to make it ffdshow compatible didn't seem to harm quality at all and PSNR only dropped a little for chroma, not luma. So I see no harm in doing that if anyone wants to use that matrix and uses ffdshow to decode.

Originally posted by LoKi128
When encoding an anamorphic source... would a matrix with higher detail in the horizontal axis provide better quality? Most matrices that I've seen are pretty even in both the horizontal and vertical... but in an anamorphic encode it is important to keep the higher horizontal frequencies, since sqishing the image in essense shifts any lower frequencies up.Honestly I have no idea. But I don't think it makes much difference.

If only we had a way to see the DCT coefficients before they get quantized...I bet that would help a lot in creating a good matrix.

[Defiler]

Originally posted by Acaila
If only we had a way to see the DCT coefficients before they get quantized...I bet that would help a lot in creating a good matrix. Yes, and it would also be nice to be able to enter values in the matrix, and instantly see the changes previewed on an I-frame. That would be extremely cool.
By the way.. I know the large first pass is irrelevant, because what matters is the actual coefficient, not the quantizer value itself. I just had to up the disk quota in that folder to make room for the first pass. Heh.

[Defiler]

Also.. are there a pair of tools that can:
1. Create XviD-compatible QM presets from their text equivalent.
2. Load a saved QM into the Custom page via the command-line?

It would be nice to be able to schedule a large set of test encodes, rather than having to load and save manually between each one.

[crusty]

Acaila:
I haven't tested B-frames yet with that matrix.
It looks like B-frames would be compressed even more than normal with this QM, so it might produce a lower quality end result if B-frames are enabled.

If we could have something that creates QM's on the fly optimized for each frame we wouldn't go through such trouble to make a good QM, now would we? We can only have one QM

Why? I want more !! :D :D
No seriously, even if it would not be mpeg4 compliant, if we could have more QM's in one movie, it would probably improve quality at the same compression level or improve compression at the same quality level. That's what modulated QM did do, even if the modulation decision was as dumb as hell.

I believe it's not the codec's job to reduce noise
Well it's the codec's job to reduce size, and noise adds size. So reducing noise, especially when it's created by the compression itself, would be the codec's responsibility.
Reducing artifacts created by the compression proces is different from reducing noise already present in the original footage. It's the difference between preserving the original picture and altering the original picture.
The process I suggest is to preserve the original picture, while improving compressibility.

Didée:
please try to not post more than 2-3 additional pages on this thread in the next 24 hours, okay? (Crusty!! Carry on!)
Working on it...processing....processing...please stand by :D

MrBunny, I couldn't find the thread you posted.
In a two pass situation, a large first pass will simply cause a higher average quantizer.
You mean, because a large first pass tells the codec that the end result would be bigger than allowed if it used a lower quantizer, right? Just to get this straight...

I was just thinking....
When I started encoding a few years ago I had a really hard time encoding the movie '2010: The year we made contact' end I ended up using both the DivX 3 Fast-motion and Low-motion codecs.
It's a space movie and on the whole it looked better with the Low-motion codec but there where two or three scenes of fast motion that looked blocky.
I encoded the whole movie with both codecs and then used a tool (cannot remember it's name) that would merge the best parts of both.
The end result was very good.

Now I also have used different QM's recently inside one movie, by using a Heavy Compression matrix on the end credits and using H263 or modulated on the main movie.
It may not be mpeg4 compliant, but it works like a charm.
So using different QM's in one clip is quite possible.

This leads me to the following idea (you're probably guessing it already):

Encode the entire movie with several different QM's and then use a tool to switch between the best looking/best compressed parts and merge those. Then you would have a modulated end result.
And with this process you can easily alter the algorithm for choosing between the QM's without altering the codec itself, as it's just a standalone tool.
This way you could use up to a zillion different QM's in one clip, depending on how you make the decision. So you could automatically use different QM's for end credits, low motion and fast motion scenes and dark and light content etc etc.

Any thoughts on this?

[Defiler]

Can you merge two different AVI files that use different custom XviD quantizer matrices? I've never tried.

By the way.. B-frames work fine with Acaila's matrix. At least, they did for me.

[Acaila]

You're not the first to think of this :), DCTune (http://ise.stanford.edu/class/psych221/98/dctune/yuke/index.htm) already does per-frame QM optimization during encoding. Even though the program seems to be gone by now, they did prove it was possible. But I bet it required a lot of horsepower as well :D.

Doing it with an external program adds some extra difficulties. I-frames wouldn't be a problem, but P-/B-frames require a reference frame as you all know. Normally this reference frame is a decode version of a previously encoded frame on which motion estimation is performed.
If you take two videos, each encoded with a different matrix, this will mean that all frames that are used as reference will be different between those two videos. If you take an I-frame out of the first video and combine it with the following P-frame of the second video, the P-frame would act on other data than it was created for and the result would be a total mess.

The only way that I can see your idea can work, is if by analysing multiple videos you could output a logfile that says with which QM to encode each frame based on which would result in the smallest frame. During encoding this logfile would then be used to encode the video with the appropriate QM.
But faster would be to just calculate multiple matrices during encoding and pick the one that results in the smallest frame. But would have to be incorporated into a future codec someday.

[crusty]

If you take two videos, each encoded with a different matrix, this will mean that all frames that are used as reference will be different between those two videos. If you take an I-frame out of the first video and combine it with the following P-frame of the second video, the P-frame would act on other data than it was created for and the result would be a total mess.

Well a quick workaround for that would be to only cut those parts with keyframes on the same frames.

Say movie 1 has keyframes on these:
1 112 229 300 330 400 567 1000.....
And movie 2 has keyframes on these:
1 96 229 300 334 401 567 1000.....
You see that it would be possible to swap out sections 1-229, 1-300, 229-300, 1-567, 229-567, 300-567 etc etc ...
Nothing difficult here, really.

The only way that I can see your idea can work, is if by analysing multiple videos you could output a logfile that says with which QM to encode each frame based on which would result in the smallest frame. During encoding this logfile would then be used to encode the video with the appropriate QM.

This would be indeed be a better but far more complex manner. Even more so because you would have to take into account the I/B/P-frame decision and the use of different QM's for I and B/P-frames.
So, if you did an analysis like that on each frame, you would have to take into account whether or not the current frame would be a I-frame, in which case the calculations aren't that complex, or whether it would be an B/P-frame, in which case you have to recalculate the preceding frames up to the I-frame used for reference. This would be both inefficient and far more cpu intensive.
If you just calculated all frames as I-frames, you would ignore the effects of a separate B/P-frame QM. It would be far simpler tho.
But faster would be to just calculate multiple matrices during encoding and pick the one that results in the smallest frame.
True. That's why I suggested to put it in the codec in the first place. :D
But the crude way I mentioned above, splitting only when I-frames match, would be far easier to implement.
It would allow people to modulate QM on a much more intelligent basis, than just the current Quantizer-based decision. It would also allow modulation on more than 2 QM's provided you have the cpu and time to spare, and you would be able to use other QM's than just h263 and Mpeg.

[MrBunny]

Originally posted by crusty

MrBunny, I couldn't find the thread you posted.

You mean, because a large first pass tells the codec that the end result would be bigger than allowed if it used a lower quantizer, right? Just to get this straight...
Sorry, I don't know what happened to the url, must have copy&pasted it wrong.
http://forum.doom9.org/showthread.php?s=&postid=305827#post305827

Originally posted by crusty

I was just thinking....
When I started encoding a few years ago I had a really hard time encoding the movie '2010: The year we made contact' end I ended up using both the DivX 3 Fast-motion and Low-motion codecs.
It's a space movie and on the whole it looked better with the Low-motion codec but there where two or three scenes of fast motion that looked blocky.
I encoded the whole movie with both codecs and then used a tool (cannot remember it's name) that would merge the best parts of both.
The end result was very good.

Now I also have used different QM's recently inside one movie, by using a Heavy Compression matrix on the end credits and using H263 or modulated on the main movie.
It may not be mpeg4 compliant, but it works like a charm.
So using different QM's in one clip is quite possible.

This leads me to the following idea (you're probably guessing it already):

Encode the entire movie with several different QM's and then use a tool to switch between the best looking/best compressed parts and merge those. Then you would have a modulated end result.
And with this process you can easily alter the algorithm for choosing between the QM's without altering the codec itself, as it's just a standalone tool.
This way you could use up to a zillion different QM's in one clip, depending on how you make the decision. So you could automatically use different QM's for end credits, low motion and fast motion scenes and dark and light content etc etc.

Any thoughts on this?
Ahhh, the old manual MM4 days. I remember using a tool called ProjectDivX for it, but your point is well taken. Like that tool, the QM/cutscene points would need to be per i-frame as not to break Xvid Compatibility (much for the reasons Acaila described).

I personally would love a tool like ProjectDivX just for cutting and pasting between two xvid files. Sometimes I enjoy tweaking a scene or two (or many more...) that the curve just doesn't deal with right. Using it for clips with different QMs would also be an option. Multiple QM xvid files are possible as long as they are cut @ i-frames, I've done one myself. But that was before I know about MPEG-4 compatibility issues, and right now I prefer to stay MPEG-4 compliant, even at the cost of a little quality.

Sigh...new post as I was writing this :D
My major problem with auto-selection of QM by any algorithm, is that it can't "see" what the end result is. It's quite possible that matrix A is results in slightly smaller framesizes than matrix B, but encoding with matrix A looks really ugly relative to B. I think such an algoritm would have to be very carefully written.

[Acaila]

Well a quick workaround for that would be to only cut those parts with keyframes on the same frames.No, you misunderstood me. What I meant was that a frame encoded with one matrix cannot serve as a reference for the following frame when that frame expects a reference frame encoded with some other matrix.

Suppose you have one video encoded with H263 and another with the MPEG matrix. If you take an I-frame from the first video and put a P-frame from the second after it, the P-frame will not have the correct data because it expects an I-frame encoded with MPEG, but gets one with H263.
The fact that a frame is encoded with either H263 or MPEG makes it look completely different and mismatched I-frames like that will corrupt all P- and B-frames referencing it.

So it's simply not possible to cut&paste a video together to get the most efficient matrices. Not without decoding-reencoding anyway.

[Defiler]

This sounds like a job for... KLUDGE!
http://sourceforge.net/projects/kludge

[crusty]

No, you misunderstood me. What I meant was that a frame encoded with one matrix cannot serve as a reference for the following frame when that frame expects a reference frame encoded with some other matrix.

Ok, I see. But that doesn't change the workaround because you can still cut on keyframes, because they have no reference.

[mf]

Originally posted by Acaila
The only way that I can see your idea can work, is if by analysing multiple videos you could output a logfile that says with which QM to encode each frame based on which would result in the smallest frame. During encoding this logfile would then be used to encode the video with the appropriate QM.
But faster would be to just calculate multiple matrices during encoding and pick the one that results in the smallest frame. But would have to be incorporated into a future codec someday.
Why smallest frame? If I have a matrix that destroys lots of detail, it will almost always create the smallest frames, and will "win" the decision over all the other matrices I might have. I'd say something like PSNR coupled to size, but then again, PSNR would probably always be related to size (since in this stage all you're doing is reducing coefficients, which will always kill detail and thus reduce PSNR), so you'd have to involve some psychovisual mathematics in it.

[Acaila]

I followed the smallest frame scenario because that is what crusty originally mentioned when he brought that idea up. But you're right, something based on an optimal rate-distortion curve would probably be preferable. That way you try to achieve the highest PSNR possible while still keeping the same total bitrate as the original video.

[mf]

Well ok in the 2nd pass it's doable :). But at constant quant a decision would be harder.

[Acaila]

In the 2nd pass? We were talking about an external program to cut up videos that were already finished (which would always be suboptimal compared to encoder based optimization, but that's beside the point). Whether that video was created with 2-pass or 1-pass is no longer relevant at that point.