Quantization matrices and adaptive quantization

[merlinhd]

Hi,

In my last encodings I've been trying a few custom quantization matrices - Didée and Sharktooth's - with different results. The thing is that there's just too many matrices and too little information is given about them.

My question is: what bitrates are Didée's matrices typically good for? I usually encode aiming at bitrates between 1500 and 1800 - depending on file size. What matrices are typically recommended for these bitrates?

On the other hand, I was wondering whether it's possible that there is a conflict between HVS matrices - such as Sharktooth and Didée's - and adaptive quantization, since both techniques seem to aim at the same, that is, taking bits from dark/bright areas to dedicate them to other areas more noticeable for the human eye. Is it maybe better to use a non-HVS matrix if adaptive quantization is activated?

Thanks for any hints.

[Sharktooth]

For what concerns my matrices i've made a thread with informations about bitrates, resolution, etc. : http://forum.doom9.org/showthread.php?s=&threadid=83125
The xvid adaptive quantization option should not interfere with any custom matrix except, maybe (but im not sure), at very low bitrates.

[akupenguin]

Originally posted by merlinhd
I was wondering whether it's possible that there is a conflict between HVS matrices - such as Sharktooth and Didée's - and adaptive quantization, since both techniques seem to aim at the same, that is, taking bits from dark/bright areas to dedicate them to other areas more noticeable for the human eye.
Custom quant matices adjust the balance between low and high frequency details. (coarse approximation: blocking vs ringing). They have nothing to do with light or dark, so there is no conflict with "adaptive quantization".

[merlinhd]

Sharktooth: Yes, I read the information about your matrices, but I didn't find any about Didée's.

The problem is that I'm getting quite soft pictures in the movies I'm compressing, and I was wondering whether the problem could be here.

[Didée]

Originally posted by merlinhd
The problem is that I'm getting quite soft pictures in the movies I'm compressing, and I was wondering whether the problem could be here.
The problem could be anything from the phase of the moon to the butterfly effect. Bitrate 1500~1800 tells us nothing, since we know nothing about your source and involved software.
(Real world example: Using SixOfNine, I can encode "source A" at 1150 kbps / 704*528 in a breeze. Encoding "source B", same resolution, I'd have a hard time to somehow get below 2000 kbps.)

So:

- what kind of source? DVD, DV, LD capture, ...
- do you use filtering? If yes, what kind of?
- what's the input / output resolution?
- what XviD settings are you using?


Shot in the blue: If it's nothing else, and overall image quality is good without blocking, only too soft , then activate qpel.

[merlinhd]

All the filtering I'm using is:


Crop.
BicubicResize. I don't usually resize down to 640 - unless I have to crop the margins too - since my screen has a native resolution of 1440x900. Therefore it sounds like a better idea to keep the movies with a width of 720, which exactly 50% of my screen width.
removegrain(mode=2), although the source is good-quality, e.g. Runaway Jury (2003).
[/list=1]

As for XviD, I typically use the following options:
[list]
AQ: only when there's lots of dark scenes in the movie.
GMC.
Chroma optimizer.
Motion search precision: 6 - Ultra High.
VHQ: 4 - Wide Search.
VHQ for bframes too.
Chroma motion.
Turbo ;-)
Trellis.


I don't use QPel. The rest options are default, except for the quantization matrices, but I've used several. Minimum frame quantizers are default too, i.e. 1.

BTW: the encoded movies are kind of soft in any case, but I've also noticed that they look softer with WMP than with Media Player Classic.

BTW2: yes, there is some blocking in fast-paced scenes.

[Didée]

Okay, lets have a short shot on these points:

First off: through which software are you performing your filtering? Virtualdub, Avisynth, something other? (No, that's *not* trivial...)

BicubicResizeYou know that bicubic is more on the soft side. And most guides on the bicubic parameter refer only to the cubic value, and call a higher value "sharp bicubic", heh. But that's only the "ringing" value ... with <0.5, you'll get a soft image without ringing. With =0.5, you'll get a neutral soft image without ringing. And with >0.6, you'll get a soft image with ringing.
In AviSynth, a "neutral sharp" bicubic would be somewhere around BicubicResize(x,y,-0.25,0.63). In VirtualDub, iirc you can not at all do it like *that*. But, since you're after sharpness, you should use Lanczos resizing anyways.

my screen has a native resolution of 1440x900. Therefore it sounds like a better idea to keep the movies with a width of 720, which exactly 50% of my screen width.You are right: it *sounds* like a good idea to do so. Alas, the sound of it is the only benefit. The resizer working in the background during playback couldn't care less.

removegrain(mode=2), although the source is good-quality, e.g. Runaway Jury (2003).Haven't used RemoveGrain very much yet, but I think this one is okay - it should not produce a soft overall look.

As for XviD:
AQ: only when there's lots of dark scenes in the movie.
GMC.
Chroma optimizer.
Motion search precision: 6 - Ultra High.
VHQ: 4 - Wide Search.
VHQ for bframes too.
Chroma motion.
Turbo ;-)
Trellis.AQ - for me it's the other way round: if at all, I use it for hi-contrast sources, and leave my hands off it on lo-contrast sources.

Chroma optimizer - matter of taste. I've used it for a long time, but stepped back from in the past. However this one is not your 'problem'.

VHQ - everything okay.

Turbo - well, if you wish so. I don't like it at all, and only use it for [custom] 1st-passes, or for quick jobs when quality does not matter ...

I don't use QPel.Quarter pel motion search is one key to sharpness. *Especially* when the source was downsized.

BTW: the encoded movies are kind of soft in any case, but I've also noticed that they look softer with WMP than with Media Player Classic.Then they are using different renderers (HW/SW overlay). This makes a difference too, yes.

[Teegedeck]

I think that it is also arguable what a 'soft' look is. A picture with little real detail can quickly attain a 'soft' look once you remove all filmgrain, so perhaps dumping 'removegrain' would help. (I like this filter but I prefer to use removedirt except when I recklessly want to downsize something and throw in removegrain as well.)

Quarter pixel really is a very good idea in such a case.

BTW, AQ doesn't kick in on low-contrast frames, anyway, to my experience.

Edit: Returning to the original question, I have made only the very best experiences using Didée's matrices with AQ - results sharp as a razorblade!

[merlinhd]

I'm using AviSynth, because I've noticed it's much faster than using VirtualDub filters.

Is QPel activated in the first pass? Because I read somewhere that GMC is, but I don't really know about the rest of the options. I've done the first pass for a few movies and I was about to do the second now, I'm just wondering if it's fine activating this option now.

What about AQ?

Also, since Sharktooth brought matrices for different bitrates I was wondering if this applies to all matrices: that they are better for certain bitrates. I haven't read anything about this referring to 6of9.

[Teegedeck]

In first pass most advanced stuff is deactivated, there was a thread about it. And GMC definitely is deactivated. But you should use AQ in both passes, if you use it, because this one isn't deactivated.

Matrices aren't all about bitrates, but about compressiblity (how much do you have to shrink the filesize) and about input-type and preference. There has also been a thread about that recently.

I'm a simple kind'a guy and use SixOfNine if I can encode my input file at anything up to quant=4 (we're talking p-frame quants here); if the file would get too big at quant=4 (equals: if I get quants above the occasional '5' in 2nd pass), I go for SixOfNine-HVS; if filesize gets too big at SixOfNine at quant=6, I downsize the picture. I myself would never do that with a movie, only with TV series. Who am I kidding - anything less than SixOfNine at quant=4 won't do for me. :)

[merlinhd]

Do you usually set the minimum quantizer to 2? I've started doing this just because I saw this recommendation in so many guides, but I wonder why it isn't better to let XviD decide. Is there any situation in which the minimum quantizer should be set to 1?

And just another question about QPel. I thought this was just a way to measure how blocks move from one frame to another, so that if a frame is "found" in the new frame, it can be reused from the previous one, therefore reducing the information necessary to describe the new frame. Is this wrong? How does QPel affect sharpness?

[Teegedeck]

Well, I think you got the idea there, though I'm not sure. Any motion is described in vectors ('hi there, object; move north by north-west for the next 5 frames, OK?' ;)) as effectively and precisely as possible. If the motion of an object is imprecisely described, the endpoint of that motion will not be quite correct. And XviD will have to make up for this incorrect placement by spending more bits on describing texture.

Qpel doubles the precision with which XviD describes motion. While the standard halfpel precision can only describe a minimum motion by 1 pixel per 2 frames, with Qpel precision XviD can describe motion by 1 pixel in 4 frames. This doesn't save bits, actually, because storing the more precise data will take more space in the bitstream. But because the motion has been described more precisely, you can compress the picture stronger to make up for the additional use of data (i.e. use a higher quantizer) and still the result will look better than what you get when you only use halfpel! More details, more sharpness.

(Koepi or sysKin, please feel free to correct me on that if I am being imprecise myself...)

This thread (http://forum.doom9.org/showthread.php?s=&postid=225242#post225242) explains a lot about Qpel; also see this part (http://ronald.vslcatena.nl/docs/xvidfaq.html#C3e) of Crusty's XviD FAQ.

About minimum quantizer: Quantizer=1 really is just a waste of bits, it doesn't look visibly better. The only reason that there isn't a minimum quant of 2 by default is that people never stopped complaining about undersized XviD encodes. With quant=1 undersizing will never happen... The sensible thing to do is a minimum quant=2, and increase the resolution if your 2-pass result is undersized.