Encoding gurus, I wish to learn!

[Losko]

Hi everyone, I've been reading at these pages for a long time and now I want to post my question.
I often manage with DVD authoring (in a way, I have fun with it, but my eye aiming to quality) so I am aware of video and audio encoding issues. I really appreciate Hank work (I use HCenc as my favourite encoder since version 0.22!) but also I look with great expectations at x262 developing.
Now, after reading these pages, a question raises about encoding quality.
Aside from the encoder you can choose, the quantization matrix (QM) has a relevant role in order to get more quality with the same filesize (or the same quality in a smaller file).
[I have surfed the web a lot, I'm quite familiar with math and matrixes, so now I think I now that a QM is]
I'm aware there are lots of QMes out there, but the encoding pros providing them often don't write a single line to explain why their QM is good or why it performs better than others.
So, which one you have to choose? How am I supposed to choose among them?
Have I to encode my movie several times with different QMes in order to evaluate their quality?
And when I will choose the QM which offers the best quality at my eyes, will I use that QM forever?
And, aside from my simple eyes, are there objective tools able to show the quality (via graphs or others...)?
In short, is choosing the QM a trial and error procedure? Or is it a slow refinement?
(in other words, when I get a final encoding from a QM, does this result just say if that QM is good/no_good? Else, will I be able to choose a better QM starting from that encoding?)
Thank you all.

[Dark Shikari]

The primary purpose of QMs in MPEG-2 is to compensate for the horrible lack of precision in the MPEG-2 quantizer range at high qualities when the exponential quantizer scale isn't used.

Other than that, I consider them mostly useless.

[Lyris]

I've encoded a few DVD titles for the US market (using CCE SP2 and SP3, for what it's worth) and honestly, the Quantization Matrix settings are some of the few that I have never found a use for. In some difficult scenes, in rare cases, I've found that using an alternative matrix or stepping down to 8-bit precision has brought a technically higher SSIM result, but the end result, to my eye, doesn't ever look obviously better - just slightly different.

I'm aware there are lots of QMes out there, but the encoding pros providing them often don't write a single line to explain why their QM is good or why it performs better than others.

I would also like to hear explanations like this. Speaking truthfully, I am not mathematically inclined. I'd love to learn more about what goes into designing these matrixes and why people choose the values they do. Unfortunately, I also get the idea that most of these people are re-encoding DVDs, and are just trying to use the same matrix that was used to encode the first time, which isn't terribly useful.

[manono]

You use a matrix that produces an average quant within the values you want. If you're serious about it you have several from which to choose in order to achieve that goal with one of them. HCEnc has a good selection (along with a bunch of crap ones). Most of the included CCE ones are junk, although encoding with the Standard Matrix set up for Q-Matrix Switching can be a good idea. In addition, when shrinking a DVD9 to a DVD5 I have a low-bitrate matrix to encode any extras I might want to keep, in order to free up bits for the main movie. I have no idea what Dark Shikari said in his first sentence, but I know I disagree with his second sentence.

Originally Posted by Lyris

I've encoded a few DVD titles for the US market (using CCE SP2 and SP3, for what it's worth) and honestly, the Quantization Matrix settings are some of the few that I have never found a use for.

So, you're using the crummy default Standard Matrix? Then you're obviously not working for any major DVD production house because they all gave up that matrix years ago. No major Hollywood movie released on DVD ever uses CCE's Standard Matrix. One can learn a lot just by collecting and studying the matrices they do use. Of course, when shrinking DVD9s to DVD5s, one usually can't use those very good matrices because the average quants will wind up too high and you'll get artifacting, including mosquito noise, color smearing, and maybe even macroblocking.

In some difficult scenes, in rare cases, I've found that using an alternative matrix or stepping down to 8-bit precision has brought a technically higher SSIM result

And we all know what SSIM results are worth. As I understand it, one should almost never use anything but 10-bit precision. Again, look to the good Hollywood DVDs. The movies may be crap, most of them, but those guys do know how to encode MPEG-2 video, for the most part.

I also get the idea that most of these people are re-encoding DVDs, and are just trying to use the same matrix that was used to encode the first time, which isn't terribly useful.

I don't get that idea but, if true and if reencoding for DVD5, it's not such a good idea.

[mikenadia]

How to compare the two following process for DVD9 to DVD5 encoding?
1- Average bitrate with mild denoising and MPEG matrix and average quant=6
2- Same bitrate with overdenoising ( just to improve compressibility) and Fox matrix and average quant=6

When using 2, any limit to overdenoising , meaning that what are the circumstances when the lower bitrate matrix will do a better job that the filter ( with very strong parameters).

Thanks in advance.

[Lyris]

Just to clarify Manono, are you talking from the perspective of recompressing already compressed DVDs, or going from broadcast tape formats (or similar) to DVD?

Again, look to the good Hollywood DVDs.

The what?
Most of the Hollywood DVDs I've seen are lowpass filtered into oblivion and are hardly good from a compression standpoint either. Some of the "giveaway" DVDs that come bundled with the BDs I buy now are a joke, and that's not me being defensive.

The movies may be crap, most of them, but those guys do know how to encode MPEG-2 video, for the most part.

Can you give me some examples of Hollywood DVDs you think are good?

So, you're using the crummy default Standard Matrix?

Yes. The results are anything but crummy. As I said, using anything else has never produced visibly better results.

I'd love to know more about what alternative matrixes you'd recommend, and in what circumstances. Can you suggest a new Q-Matrix I should use, so we can do a test encode with the default CCE matrix and also one with your suggestion? I would like to see the improvement you speak of. I can send you a small uncompressed sample of the source material if you like. Curious to learn about your reasoning.

[Dark Shikari]

Originally Posted by manono

You use a matrix that produces an average quant within the values you want.

This is not exactly true, or at least it's somewhat misleading.

If you halve all the values in a quantization matrix, it is mathematically EXACTLY THE SAME as if you halved the average quantizer. That is, a video with a quant matrix of all "8" at quantizer 4 will look identical to a video with a quant matrix of "16" at quantizer 8.

But in real encoding, there is a difference -- MPEG-2, by default uses a linear quantizer scale. Therefore, if your average quantizer is "2", the encoder is picking between 1, 2, and 3 (roughly) for its quantizers. This gives it very little precision to work in, because 1 is twice as precise as 2: there's nothing in between 1 and 2 for it to pick. By comparison, in H.264, there would be 5 values in between 1 and 2, and if it used the MPEG-2 exponential quantizer scale, there would be 7.

This is doubly important when using a good MPEG-2 encoder with adaptive quantization support, like HCEnc.

If all MPEG-2 encoders used the exponential scale, we wouldn't need CQMs for any purpose, except perhaps lowpassing the video to make bad DVDs.

[kolak]

Most Hollywood DVDs are encoded with old Sony Vizaro encoder, Toshiba encoder or Cinemacraft Xtream (or even SP2/SP3).
First 2 are not as good as Xtream, because they are very old.
Xtream/SP3 use Adaptive GOP an Adaptive QM and I can't see any other encoder which can be better for film source. Both will use many QM over whole encode and keep switching them from scene change to scene change or even per GOP.

I would never show Hollywood DVD as a reference one. They just have very good sources and if you add eg. Xtream encoder it comes as very good encode even at default settings.


Andrew

[manono]

Originally Posted by Lyris

Just to clarify Manono, are you talking from the perspective of recompressing already compressed DVDs, or going from broadcast tape formats (or similar) to DVD?

I'm not in the 'biz', if that's what you're asking. My sources are DVDs, all kinds and qualities of retail DVDs. I'm reencoding them for DVD5.

Most of the Hollywood DVDs I've seen are lowpass filtered into oblivion and are hardly good from a compression standpoint either.

Yes, I know that, and there are reasons they're forced to do that. One of the major complaints about DVD is the low max bitrate allowed. In addition, often the studios don't even use that max because there are many players out there that won't play compliant DVDs smoothly with the max bitrate cranked up. So you get Warner Home Video setting their max's at 7500. As far as that goes, Criterion is the best because they often allow their max video bitrates to go well over 9000 (and they don't use the Standard Matrix either). Nor do they, as far as I know, low-pass filter their stuff. In addition the studios often try and squeeze in too many DD 5.1 audio tracks which forces the max video bitrate even further down. Or they'll try and put in too many useless extras, compromising the quality of the main movie by forcing down the average bitrate. My guess is these things are forced upon the encoding engineers by a bunch of know-nothings, and the encoders can only work around these constraints. The upshot of all that is, without smoothing the video they're liable to get artifacts, particularly in the complex scenes, and sometimes they do anyway in spite of their best efforts.

The results are anything but crummy. As I said, using anything else has never produced visibly better results.

There are reasons why the studios don't use it and Dark Shikari touched on one of them when he said, "Therefore, if your average quantizer is "2", the encoder is picking between 1, 2, and 3 (roughly) for its quantizers. This gives it very little precision to work in, because 1 is twice as precise as 2: there's nothing in between 1 and 2 for it to pick. The use of the Standard Matrix for a film on a DVD9 can give you average quants that are too low. I know that seems counterintuitive, but it's true. In addition, the whole point of choosing the proper quantisation matrix is to allow the detail removal to take place in the higher frequencies (the lower right of the intra and non-intra matrices), where it's less noticeable. By using the Standard Matrix, or even the MPEG Matrix (as CCE calls them), you're getting unnecessarily high numbers, the 16s, 17s, 18s, etc., in the lower frequencies (the upper left) where the detail removal is much 'cruder', much more obvious. Compare the upper left numbers from those 2 matrices with those from a matrix such as the Fox1 available in HCEnc.

Can you give me some examples of Hollywood DVDs you think are good?

The Fox DVDs are pretty good. It's not a Hollywood studio, but Criterion produces about the best DVDs of all, I think. Except for them dropping to video at chapter stops a lot of the time, which can mess with a lot of DVD players.

I'd love to know more about what alternative matrixes you'd recommend, and in what circumstances.

I know a bunch but, as I mentioned, it depends on the final average quant. You certainly have a lot more leeway if encoding for DVD9 and don't go overboard with the audio tracks and the extras. As I mentioned earlier, and as kolak reenforced, if you don't want to mess with matrices at all, just use the Standard Matrix but at the same time allow for Q-Matrix Switching. Then, when all done, go back and check how often the Standard Matrix is actually used, as opposed to the ones with the numbers halved or quartered. If you're lucky, you may not see it at all, or only during the most complex of scenes. Make sure and run an extra pass or two, to allow the matrix switching to be optimized. If you want to try, HCEnc has the good Fox1 matrix (aka Fox Home Theater), which Fox used to use but now, for the most part, they use an even better one. If your file size allows for it (the final average quants are low enough, maybe 4-6 or so), it'll create very nice results.

[Lyris]

If your sources are existing DVDs, that is a very different scenario to encoding from an actual master, which might go some way in explaining our differences of opinion.

Nor do they, as far as I know, low-pass filter their stuff

They do - at least sometimes. I've seen plenty of Criterion titles with the trademark "wrinkles" and lack of high frequency content.

I found Criterion's output - like most other "studios" - highly variable. A lot of people fall into the trap of thinking that all studios do everything in-house, but it's not the case. I've seen nice discs from Criterion as well as surprisingly bad ones, and they come from different facilities. With that said, I always loved using "The Rock" as a test disc (what an usual title for them to release). Granted it was lowpass filtered but the encoding quality was excellent, at least for the time. I haven't checked it out lately.

I'll do an encode with this Fox matrix and compare to CCE SP3's defaults.

[Lyris]

Well, I just tried the CCE default "Segment" setting and then transplanted the "FOX1" values from HCEnc. I would struggle to pick one result as being better than the other. If anything, the FOX1 matrix looks slightly worse, but this might as well just be a coincidence.

On the other hand, I find that the Quantizer Characteristics settings (Activity, Residual, Luma) are what make a bigger difference to the image (and, obviously bitrate most of all). I also re-ran the test with CCE SP3's Adaptive Quantization Matrix option disabled.

Note: images sharpened for the web to make differences more obvious.

Source


CCE SP3 Default Matrix. AAQM on.


Fox1 matrix. AAQM on.


CCE SP3 Default Matrix. AAQM off.


Fox1 matrix. AAQM off.


Just to reiterate my position: I have never seen changing the QM settings produce a "Wow, look at that!" result, so I choose to spend my time erasing defects in the master, pre-processing, and also correctly setting SP3's Activity/Residual/Luma settings. I would still like to understand why some people spend so much time with QM and I would like to see some cases where it has been worthwhile for them. Surely it can't all be theory?

Settings in all cases were 6mbps average (2/8 min/max), 9 passes, new VAF created each time, NTSC with 2:3 Pulldown. Picture settings were completely flat (no LPF) except for some slight roll-off for Horizontal chroma (my reasoning: this telecine transfer has fine coloured grain which can't be fully reproduced by stepping down 4:2:0, and filtering chroma results in less "coloured compression artefacts" in the difficult scenes). Quant characteristics were 20/40/70 (Activity/Residual/Luma). Motion estimation Normal, Picture Type Progressive.

Your thoughts? I'd just love to see a practical example, that's all.

[manono]

Originally Posted by Lyris

I found Criterion's output - like most other "studios" - highly variable. A lot of people fall into the trap of thinking that all studios do everything in-house, but it's not the case.

That's true, sometimes they're using masters supplied by the rights owners. They're pretty easy to tell, though.

I'll do an encode with this Fox matrix and compare to CCE SP3's defaults.

I don't use SP3, but the other CCEs have a low-pass filter on by default. Surely you're turning it off?

[Lyris]

Yes, absolutely. I can't stand lowpass filtered video when it's avoidable. It is the single worst encoding practice that was accepted with DVD, IMHO.

As for the Criterion comment - that was more to do with encoding than masters. They deserve serious praise for actually making new masters when one is needed (usually...)

[Dark Shikari]

Using a CQM with extremely large values in the high frequency positions is basically equivalent to lowpass filtering. If you don't like lowpass filtering, you shouldn't be using such CQMs.

[manono]

Originally Posted by Dark Shikari

Using a CQM with extremely large values in the high frequency positions is basically equivalent to lowpass filtering. If you don't like lowpass filtering, you shouldn't be using such CQMs.

Sure, but one doesn't always have a choice. Especially if compressing a difficult-to-compress movie down to a DVD5. You can filter in the script, in the encoder (low-pass filter) or in the matrix used. mikenadia was asking about something similar earlier in the discussion. Me, I prefer to do it in the matrix and consider it the least of the three 'evils'. By using a 'better quality' matrix you might open the door to all kinds of artifacts and poorer results, in my opinion.

Originally Posted by Lyris

As for the Criterion comment - that was more to do with encoding than masters.

I didn't phrase that right. Sometimes they're not only supplied the masters but some ready-for-DVD encode. I presume, because the encoding settings are so peculiar and so different from the usual Criterion practices. That's what I should have said when I mentioned they're easy to spot.

[kolak]

Originally Posted by Lyris

Settings in all cases were 6mbps average (2/8 min/max), 9 passes, new VAF created each time, NTSC with 2:3 Pulldown. Picture settings were completely flat (no LPF) except for some slight roll-off for Horizontal chroma (my reasoning: this telecine transfer has fine coloured grain which can't be fully reproduced by stepping down 4:2:0, and filtering chroma results in less "coloured compression artefacts" in the difficult scenes). Quant characteristics were 20/40/70 (Activity/Residual/Luma). Motion estimation Normal, Picture Type Progressive.

Your thoughts? I'd just love to see a practical example, that's all.

With SP2/SP3 there is almost no point of changing QM, because with AQM=ON encoder will change you matrix anyway. Quite often at the end default matrix became FOX one

You may try to lower Activity to 4 or go much higher eg. 48.
With some noisy sources SP3 tries to hard and when it can't preserve all details it does blocking (like on your sample grabs). It means it's to sensitive for all smallest noise, so you can try to rise Activity setting. In the same time it may start producing mosquito noise when Activity is to high

Andrew

[Losko]

Mhh... I see you figured out two main different scenarios when talking about encoding:
* encoding from HQ video source (maybe HD content?)
* re-encoding MPEG2 material
Why are these so different? Is source so relevant to affect workflow?
What have you to take care?

[Lyris]

Most existing MPEG2 encodes seen on commercial DVDs are not terribly good (and that's me being diplomatic).

[*.mp4 guy]

Cqm's are useful for compensating for source oddities in general. And are pretty much required for compressing something that has been lowpassed, because a dct of a lowpassed image will still contain some high frequencies (some of which will be encoded, at pretty much any bitrate that looks decent) which contribute very very little to decode quality and nothing to decode detail levels.

Cqm's are also helpful for psyops in encoders that don't actively optimize for grain/detail retention at the quantization/rdo level, aq alone is not enough in some cases.

To expand on compensating for source oddities, for example sometimes a source is a bit soft (and that's how it is supposed to be, no lowpassing required), flat matrices often don't look that great in such situations.

I have to state that I don't use anything but hcenc anymore, and its quite possible that the way other encoders such as cce are set up, they don't get any benefit from matrices, there are plenty of ways this can happen that don't necessarily mean "matrices don't do anything useful".

Basically, cqm's are for making sure that you have a roughly equal amount of all of the undesirable artifact types in your output, and that you don't have one specific thing that jumps out as worse then everything else. Unsurprisingly, if you have a perfect source, and an encoder with aggressive psyops built in, cqm's are not terribly useful.

[Biggiesized]

I know this is old, but how did you acquire the Fox Home Theater CQMs?