XVID preset conversion to CQ preset version?

[moonoo]

Hello All,

My first post to this fine forum so be gentle :)

I don't have much experience of encoding but I'm trying to get a feel for it and my levels of acceptability for the final encode results. I thought I would start with Teegedeck's Xvid presets but I have no final file size to match for my requirements so I would like to turn these presets into CQ versions.

Teegedeck quotes that: "These presets are meant for two-pass. Of course you can also use the presets for constant-quantizer encoding. BUT if you want to do that use a constant quantizer 3 or 4." Does that mean you should really only use a CQ value of 3 or 4 for any of the presets? I did a test of this and for an original file size of 252 MB (including audio) the presets came out thus:

Preset------------------------File size
Original File (inc audio)--------252 MB
90%hq(but CQ set to 4)-------182 MB
58%hq(but CQ set to 4)-------160 MB
45%hq(but CQ set to 4)-------123 MB
30%hq(but CQ set to 4)--------86 MB

I'm quite surprised at the variation of the final file sizes for a CQ value of 4 for all of these encodings. I guess this just demonstrates the effect of the custom quantization matrices used and the other preset settings.

I know that if you are using CQ mode you can't predict the final file size but I would have thought that in general a CQ encode of 2 would have a larger file size than a CQ encode of 5 so that being the case could you "map" this information to Teegedeck's presets like this:

Preset----------Appropriate CQ setting
90%hq------------------2
58%hq------------------3
45%hq------------------4
30%hq------------------5

Do you think this is a suitable way to "convert" these presets for CQ encoding?

Also in addition, are there any other settings I should change in the presets for CQ encoding? e.g. the 30%hq preset has Adaptive Quantization selected but I did read in the forums that this setting is a bit inappropriate for one pass CQ encoding.

Thanks all for your help.

Regards
moonoo

[Ranguvar]

My advice is to not use those presets at all. They really limit the quantizers, resulting in very inefficient encoding.

[henryho_hk]

I think the presets are better chosen based on a compressibility test (that's why I wrote a script to automate the comp. test).

To convert them to 1-pass "constant quality" presets, we can no longer rely on the rate control. We can only calculate the compression (bit)rate/size of a given preset and the chosen quantizer value. Hence, by testing the encoded size of different sources on ">90%-Fast-CQ2", ">90%-HQ-CQ2", ">90%-Fast-CQ3", ">90%-HQ-CQ3", ">58%-Fast-CQ3", we can give them more meaningful names such as "95-100%-Fast-CQ", "90-96%-HQ-CQ", "58-80%-Fast-CQ". i.e., use the (estimated) range of compressibility result as the preset+quantizer name.

[moonoo]

Hello Ranguvar, thanks for your reply. What you say is interesting as I thought I might have been in the "ballpark" for efficient encoding using the presets as a basis for CQ versions.

Are there any threads you could point to I could read for my homework. :)

I'm happy to try other settings that might be a better starting point for CQ encoding.

[moonoo]

Hello henryho_hk, thanks for your answer, mmmm this could be a bit more complicated than I thought. :)

When I read through the information in the sticky preset thread I thought I would follow it and try a compressibility check but I got confused straight away when it said "In detail you need to do this: Start up Enc, select your .avs, choose 'compressibility check' and enter the filesize you're aiming for in kilobytes." What do you do when you are not aiming for a final file size? That's when I got stuck.

Is it true that if you encoded a source video at "transparent" settings i.e. CQ 2 then found the bitrate that the encoder had used then that is a measure of the compressibility of the original file? A high bitrate figure would indicate a "difficult" source to encode/compress, a low bitrate figure would indicate an easy source to encode/compress.

Is it also true to say that if you are encoding to a quality threshold you perhaps don't need to care about the compressibility?

e.g lets say I have managed to create, (with the help of the fine folk here at doom9) a good quality CQ preset that produces results I like the look of and this CQ value is 3 for example. If I then make an encoding and the file turns out to be 95% the same size as the original I can then assume that the source is "difficult" and the compressibility is low/hard to compress. At this point I then have a choice, I can choose to take a drop in quality and push the CQ value up to e.g. 4 or 5 and this will give me a smaller encoded file size or live with the larger file size to preserve the quality I like.

regards, moonoo.

[henryho_hk]

If we do not target at certain filesize (or an upper limit), we will all pick the most detailed q. matrix and use constant Q2 (e.g. the “Editing Only” preset). No more compressibility test, two-pass encoding, etc. are required; we dun need to care about Q3, 4, ….. 31 either. And we don’t need to apply noise reduction filters to the clips. The compression will be so much faster too.

Yet, we always have a target (or at least a maximum) filesize in our mind. The encoding difficulty of a source is always relative to the target filesize. When the target size is “extremely big” (maybe 1TB), almost all sources are easy (well, it still depends on the source length). As you said, in CQ encoding, we intend to encode at _a_ quality threshold, i.e., we need to choose an acceptable quality level. Such quality compromise (_a_ quality threshold) exists because we do have an expectation on the outcome size. The issue is how to establish a reasonable means to estimate and quantify these compromises. For example, under the same other settings, approximately how bigger size would be when we go from CQ4 to CQ3? It’s best if we can make a rough prediction before making another (lengthy) encode.

[ronnylov]

You can think of the CQ mode (which means constant quantizer) as something that result in almost constant quality as long as you use a certain quantizer matrix. It is the combination of the matrix and the CQ-value that in a way tell the encoder "how much quality to throw away".

You have already seen that different matrices (different %hq as in your table) give different file size. A lower CQ will give bigger file size and a higher CQ gives lower file size. The quality pretty much corresponds to the file size (when encoded the same source file) but may give different look when changing the matrix. One matrice at cq=3 may give the same file size as another at cq=5. Too high cq-value will result in compression artifacts and too strong matrice will soften the picture too much (less details). You have to decide what you prefer, less artifacts or softer picture or something in between?

I have settled for using CQ=3 with a "medium compression" type of matrix for standard definition content (like digial tv captures or DVD) and CQ=3 with a "high compression" type of matrix for HDTV captures or HDV captures. CQ=2 gives sometimes bigger files than the original source and CQ>=5 may give too low quality in my opinion. So using CQ=3 or CQ=4 is a normal compromise to give good quality with acceptable file size. Using the same resolution as the source I normally get lower than half the file size with xvid compared to MPEG-2 sources. This is the compression level I aim for and the quality is almost as good as the original.

So do some testing with different matrices and different CQ-values and watch file size and compare quality to find a compromise that you like.

[moonoo]

Thanks for your replies henryho_hk and ronnylov, sorry for not replying sooner.

I was thinking about your answer henryho_hk and I realise now that what you say is true I have a target final encode size expectation in mind. I think though that because I don't have any experience to draw on, the best I could approximate this size to is "much smaller". Part of the fun I guess is seeing what xvid is capable of. How small one can encode the file down to while still maintaining good quality. I know this threshold will be different for everyone.

Reading the information that ronnylov posted was also very useful. He seemed to say that for a standard definition source, using a "medium compression" matrix and CQ=3 you can expect to encode a good looking final file at the original resolution for a bit less than half the original file size.

Just to simplify the process further and establish a good staring "medium compression" point, what about:

Source - Standard definition pal dvb mpeg2 capture
Encoding type - Anamorphic (use source resolution)
Matrix - Didees SixOfNine.cqm (is this a good choice for medium compression)
CQ - 3

Am I correct in saying that with these settings you should be able to reduce the file size by about half and maintain a reasonable quality?

From here you could decide to increase the CQ value to 4 for a smaller file and or change the matrix to eqm_v3hr.xcm. Furthermore you could decide to explore some pre-filtering of the source to reduce noise and obtain a smaller final encode file size.

What do you think, are these good "ballpark" settings?

Thanks again all for your insight and replies.

Regards
moonoo

[henryho_hk]

6of9 seems not to be a "medium compression matrix". V3HR or LR may be (yet I recall one of them looks bad at Q=4).

[Ranguvar]

6of9 is an over-the-top matrix for keeping almost all detail. For very, very high bitrate encodes.

Read this (http://forum.doom9.org/showthread.php?t=83125) for info about the V3 matrices.