Newbie: constant quantizer

[hitbit]

I'm encoding my clips with mencoder and xvid with costant quantizer=4 and I'm usually very happy with the results, both quality- and size-wise. My xvid encoding options are:

fixed_quant=4:me_quality=6:quant_type=mpeg:max_bframes=2:qpel:gmc:trellis:chroma_opt:hq_ac:vhq=4:autoaspect

but I'm wondering if all the options I've set make sense in a fixed quantizer environment. For example, quant_type or trellis are really useful or I'm only wasting my CPU time? BTW, there are any suggestion for improving quality or size even further (without resorting to 2-pass, of course)?

[Ark]

You can use a different matrix (your "quant_type" parameter).

There are many around, such as MPEG, h.263, hvs-good/better/best, some from users of this forum, but the choice is related to your scenario: movie type, lenght, resolution, destination media type (cd, dvd)...

[Manao]

You can add option for b-frames' vhq ( but I don't know if it's already in mencoder ). Apart from that, you might want to try adaptive quantization, but you may not like the result. I'd definitely use another matrix than Mpeg, or least give them a try ( though you may need to change the quantizer in order to reach similar size / quality )

[celtic_druid]

Yep, option is bvhq=1.

[hitbit]

Thanks to everybody.
Seems that b-vhq is worth trying, so I'll download last version of xvidcore and rebuild mencoder.
Mmmm, custom matrices? I've read some thread and I've found this matter rather confusing, so I think that I'm better sticking to mpeg for now.
As for trellis, I made some little experiments and I found that even with constant quantizer it reduced filesize by 2-3%, so is definitely worth activating.

[akupenguin]

Yes, trellis is good, but:

Do NOT measure compression options by filesize!
Trellis uses rate/distortion optimization, which means that it may decrease the quality of the video if it can save enough bits to be "worth it" (where the definition of worth depends on quantizer). Or it might have increased the quality and you could be underestimating the improvement by looking at only filesize. The tradeoff threshold is supposed to be calibrated such that it doesn't modify quality much, but it's not perfect, and can only be accurately measured in 2pass mode. (The same advice applies to all options, but especially vhq, bvhq, and anything related to B-frames.)

Now, I don't know whether you'll actually come to any false conclusions about trellis, but I have seen this come up in other situations: lavc's mbd=1 produces smaller files at constant quantizer than does mbd=2, but mbd=2 is still the better compression option because it takes into account quality in addition to size.

Also, no matter how good you think quant=4 is, there is room for improvement using 2pass.

[hitbit]

Thanks akupenguin, your replies are always very insightful.

lavc's mbd=1 produces smaller files at constant quantizer than does mbd=2, but mbd=2 is still the better compression option because it takes into account quality in addition to size.

I've never tried lavc mpeg4, do you think it's worth trying?

Also, no matter how good you think quant=4 is, there is room for improvement using 2pass.

I know, but I'm not "in search for perfect encoding". I think that the compromize between quality/size/speed-of-encoding of cq=4 is good enough to satisfy my needs (and, I think, the majority of "mainstream movie watchers"' needs). Of course, if there's room for improvement it's welcome (and I'm really waiting for x264 and others new codecs!). But, without wanting to start a flame, I think that with the diffusion of dvd burners and other future removable mass storage devices the entire matter of 2pass encoding will become less important. Who really cares if a movie is 100 MB longer if you have several GB of space to save it? Think about it: why noone ever bother to make 2pass mp3 (AFAIK)? The middle-of-the-road user wants standard, easy and effective enough tools for movie backups without too much complications. Only my humble opinions, of course.

[Manao]

hitbit : two passes isn't only meant for bitrate accuracy. It also allows a better quality at same filesizes.

Moreover, the fast first pass of XviD is at least four times faster than your q4 @ VHQ 4 encoding ( so doing a first pass would only multiply the total encoding time by 1.25, provided that your prefiltering isn't too heavy ).

[akupenguin]

Originally posted by hitbit
I've never tried lavc mpeg4, do you think it's worth trying? Yes. When fully tuned, it can produce slightly nicer results than XviD (YMMV, as always with codec comparisons), or it can produce quite decent quality very fast. However, it's definitely more complicated to configure than XviD. (More options, and the defaults are all optimized for absolute fastest speed and compatibility, not for good compression.)

but I'm not "in search for perfect encoding". I think that the compromize between quality/size/speed-of-encoding of cq=4 is good enough to satisfy my needs (and, I think, the majority of "mainstream movie watchers"' needs). Of course, if there's room for improvement it's welcome (and I'm really waiting for x264 and others new codecs!). If you're doing a significant amount of filtering, then I can see being put off by the extra time for duplicating the filtering vs the inconvenience of a huffyuv. But then if you're so concened about speed, you probably wouldn't be doing heavy filtering either. So a first pass with turbo (or equivalent fast settings in lavc) adds only a little extra time, and makes much more difference than incremental improvements in the final pass options (like vhq).

Who really cares if a movie is 100 MB longer if you have several GB of space to save it? Think about it: why noone ever bother to make 2pass mp3 (AFAIK)? Agreed (I think). The reason no one makes 2pass mp3 (or other audio) is because you don't care how exactly how big a given mp3 is; any given storage device can hold so many that all that matters is quality vs average compression. And movies will soon become the same.
Ideally, a good 1pass video would be just as good as a 2pass video, the only difference being that you would specify quality instead of size. But 1pass VBR video isn't there yet: Constant quality (like good audio encoders use) is not the same as constant quantizer.

And processing power is getting cheaper, too. If I can improve quality and/or reduce size just by throwing more CPU time at the problem, that's a good tradeoff in my view.

[hitbit]

There's another, subtler, reason for which I don't like 2pass very much, that I'll try to explain.

With constant quantizer I am roughly aware what the final quality of my encodings will be. If a movie results larger than another, well, it simply means that needs it. All my movies will have roughly the same quality.

But with 2pass, I have to choose filesize (or bitrate), and invariably I'll fall in the "CD Syndrome", i.e. aiming for 700MB (or 1400). That means that some movies will fit well, but some will results oversized for their content (=extra quality that I don't need) and some undersized (=lower quality). It's like putting the same trousers on different persons. 2pass simply tries to adapt the person into the trousers (something that I'm sure you don't like if your tailor should propose you ;) ) and not viceversa, as I think would be preferable.

Yes, I could choose another filesize, but which one? You should guess in advance what the movie needs! And how?

I know that 2pass offer the best quality for a given filesize, the problem is: why you need that particular filesize? The answer generally is: to put my movie on CD (and squeezing every possible bit of quality from it). But I don't use CDs anymore.

So, the best solution so far seems to me using constant quantizer.

Akupenguin, your remark about the difference between constant quantizer and constant quality seems interesting. I've always thought that they are two ways to express the same thing, but it seems that I was wrong. Can you explain further please? And there are actually codecs that do REAL constant quality?

[Manao]

hitbit : just wait for the result of the first pass. Then choose half the first pass estimated size, and you should end up with a size roughly the same as a constant quant 4 encoding.

For constant quality / constant quantizer : at fixed quantizer, the PSNR ( ie quality ) varies from frame to frame. For exemple, a blurred frame will get a higher PSNR than a sharp and noisy frame. Hence, for constant quality, you need to make the quantizer vary.

[hitbit]

hitbit : just wait for the result of the first pass. Then choose half the first pass estimated size, and you should end up with a size roughly the same as a constant quant 4 encoding.

Oh! Thanks, that's more like it. Is there some general rule for this? e.g. (estimated size)/4 is roughly equivalent to quant 8?

[Manao]

No, it depends on too many factor. Usually, q4 will give you a size between one third and a half of q2's size. And a fourth of the first pass size may be reached as well at q5 than at q8, depending on the clip.

Moreover, enabling trellis / vhq 4 only on the second pass will tend to increase the difference.

[akupenguin]

Even if "q2 size / 2" isn't a great predictor for q4 size, it may be just as good a measure of constant quality, due to the mentioned difference between quality and quantizer.

And the other reason 2pass is better than 1pass VBR: temporal prediction. Unlike audio, the quality you use for one frame affects the quality of subsequent frames. Spending a few more bits at the beginning of a low-motion sequence improves the whole scene, while spending a few more bits in a high-motion sequence only improves 1 or 2 frames.

[PiXuS]

Originally posted by Manao
For constant quality / constant quantizer : at fixed quantizer, the PSNR ( ie quality ) varies from frame to frame. For exemple, a blurred frame will get a higher PSNR than a sharp and noisy frame. Hence, for constant quality, you need to make the quantizer vary.

Damn it. I never thought so. I too was doing most of my encodes @ fixed quant because I don't really need to make my backups fit to a particular size (i.e., I don't care if one of my backup ends up taking 1500MB instead of 1400MB). But I was always under the impression that fixed quant was always giving the best quality possible because of the non fluctuation of the quants (which some people report was perceptible on playback).

I personally use SixOfNine @ q=4 with B-VOPs @ 2/1.50/1.00 for 99% of my encodes. Single-pass.

The real question is, on a two-pass encode, will XviD really make the quants fluctuate to attain maximum PSNR or the attain maximum correspondance to the target size or target bitrate?

Edit: I clearly remember having done 2-pass encodes with a 1-st pass with q=4 (using 1 zone) and then on the second pass, I had fixed the bitrate @ 8000 kbps. Not one frame got a quant @ 3. Maybe this is anecdotal, but it strengthens my opinion that quants fluctuation on 2-pass isn't for max PSNR but for max correlation with the target size or target bitrate. I am not trying to "win" an argument... I am just trying to figure out if XviD is intelligent enough to know which quant to use to up the PSNR while respecting it's target size(|bitrate).

2nd Edit: See this post (http://forum.doom9.org/showthread.php?s=&postid=499243#post499243) from sysKin on the subject.

Thanks!

p.s.: hitbit, you can find a list of the available custom quant matrices here (http://forum.doom9.org/showthread.php?s=&threadid=84728). A good CQM will give you much better quality (especially if you are targeting q=4!!)

[hitbit]

Great, another q=4 fan!! Thanks pixus, I think that I'll try SixOfNine then. (Sigh, so many options to try and not enough time...)

[akupenguin]

Originally posted by PiXuS
The real question is, on a two-pass encode, will XviD really make the quants fluctuate to attain maximum PSNR or the attain maximum correspondance to the target size or target bitrate?
Normally it's a tradeoff between really optimal quality and size control. But you can tell it how much weight to put on the size using the "overflow control strength" option. While I haven't tried it, I have read the relevent sourcecode: If you set "overflow control strength" to 0, it should just use your specified bitrate to select an average quality when the 2nd pass encoding begins, and thereafter ignore the number of bits actually used.

I clearly remember having done 2-pass encodes with a 1-st pass with q=4 (using 1 zone) and then on the second pass, I had fixed the bitrate @ 8000 kbps. That's because zones are special. If you force a zone to use q=4, XviD assumes that you really want q=4 there, and overrides the normal ratecontrol. I don't know if it's possible to use quants other than 2 on the first pass, without modifying XviD. Even if you do find out how, a first pass with q=3 might be better than q=4, since lower quants do better at predicting complexity. (An exessively low q on the 1st pass will hurt bitrate prediction, but if you disable overflow control, it's not clear whether there will be any negative impact on quality.)

[PiXuS]

Well... after doing more tests myself and reading all that sysKin had to say on the subject (do a search for "constant" "quant" and then "fixed" "quant" posted by sysKin), I still believe constant quant produces better quality than a 2-pass encode.

I didn't take note of who said it but, while reading posts from sysKin, one person brought up the fact that fixed quant != fixed bitrate. Two frames encoded @ the same quant level will probably be given a different amount of bits depending on their content. So, a 1 pass encode @ a fixed quant is not a CBR encode, but a VBR encode.

@akupenguin: BTW, during my tests, I came to the conclusion that you are right about not doing a first pass @ a too high quant. First passes @ q=4 for example will not be helpful on the 2nd pass. I conclude that first passes should be done @ either q=2 or q=3 (q=3 for high bitrate matrices, as reported by others).

[Teegedeck]

Didée once pointed out that fluctuating quants means XviD wastes bits to recover quality when going from the higher to the lower quant. Or someting. So theoretically, a constant-quantizer-encode is what we want.

BUT: On the other hand, a constant-quantizer encode can't accurately hit a filesze, so it might waste bits and quality by not using the capacity of a DVD-R or CD-R to the full extent. Also, at a lower bitrate you might want to use the infamous benefits of scraping a little bitrate off the big frames and distributing the saved bits onto several smaller frames. Which in the end might yield a better quality than a constant-quantizer encode at an altogether higher constant quantizer. Still, it cannot be desirable to have quants fluctuating all the time but only for example quant=3 for these ten p-frames and quant=4 for those ten p-frames (which isn't such a grave difference when using high-bitrate matrices).

For me the ratecontrol doesn't work all that bad, sysKin! :)

[akupenguin]

Originally posted by Teegedeck
Didée once pointed out that fluctuating quants means XviD wastes bits to recover quality when going from the higher to the lower quant. Or someting. That just means the current 2pass code isn't optimal. At the very least, you could let the quantizer be constant within one GOP, but still vary from one to the next. Or to get fancier, allocate quants not based just on the current frame's complexity, but also based on future frames.

[PiXuS]

Originally posted by akupenguin
That just means the current 2pass code isn't optimal. At the very least, you could let the quantizer be constant within one GOP, but still vary from one to the next. Or to get fancier, allocate quants not based just on the current frame's complexity, but also based on future frames.

I think your ideas are very interesting. I believe it would make a lot of sense to have a constant quant for a given GOP. THOUGH.. as of XviD 1.1, if the 2nd pass algo isn't optimal... AND you don't care about fitting an encode to an exact size, 1 pass @ fixed quant seems to be the way to go if you desire to obtain the best quality possible.

It would be nice to have to input of sysKin, Koepi or someone else active in devel on that subject. Or maybe they figure they already have answered too many times this question in a form or another... ;)