XviD Quantizers

[sirt]

Hi,

As you all know XviD is now obsolete compared to x264 but it doesn't prevent me from trying to understand how it works, so I have some questions :

Let's say I encode in two passes. My expectations : the slowest encoding as possible, the less blur as possible, no fluctuations around the bitrate setted : I need the encoder to respect it - at least - trough the second pass or both if possible. This is the parameters I think I have to focuse on :

http://i1240.photobucket.com/albums/gg481/abel413/aq2.jpg

Is that great for a video at 24 img/s ? In particulary, what will imply VHQ metric 1 and what will happen with 0 ? I've setted I-frame interval to 240, is it correct ?

http://i1240.photobucket.com/albums/gg481/abel413/aq3.jpg

Well I've setted all to 1 because I'm not sure how to achieve the goals mentionned below (especially the respec of the bitrate imposed), what are the consequences of setting everything to 1 ? My tests revealed the bitrate is too high compared to my settings, but I may be wrong ? I read somewhere I might use 1,8/1,16/1,24 to be optimal, what would you recommand ?

http://i1240.photobucket.com/albums/gg481/abel413/aq1.jpg

I have no idea how to set that window but I read somewhere I should only change the reduced % to 20, what I should I do to achieve my goals ?

http://i1240.photobucket.com/albums/gg481/abel413/aq4.jpg

I'm configurating those windows like that because I've read using MPEG matrix would be good to keep details (especially if the video doesn't have dark areas), but I'm not sure : what will be the difference with H263 exactly ? What if I would use a custom matrix such one of the matrix's made by *mp4 guy (http://forum.doom9.org/showthread.php?p=887125) ? Should I use Adaptize Quantization and how ? I don't tick anything excepted B-Vops regarding what I want because I've read it may be a bad thing, am I right ? Finally somebody told me raising the Quantizer Ratio (for example to 2) could improve the quality, should I do that ? I usually don't mess with Level and Aspect Ratio windows, is there something to change there ?

[Guest]

Why are you obscuring filenames in your posts?

[LoRd_MuldeR]

Let's say I encode in two passes. My expectations : no fluctuations around the bitrate setted

Why you don't want any bitrate fluctuations?

As a matter of fact, the "complexity" of almost any video varies over time. In order to get a consistent quality, the bitrate has to be adjusted with respect to the content of the video.

That's what the idea of 2-Pass mode is all about: Allow the bitrate to "fluctuate" (in other words: Move the bits to where they are actually needed!) and still be able to hit the target average(!) bitrate.

If you would be encoding with a 100% constant bitrate (i.e. no bitrate fluctuations), this would result in quality fluctuations, which certainly isn't desired ;)

I'm configurating those windows like that because I've read using MPEG matrix would be good to keep details (especially if the video doesn't have dark areas), but I'm not sure : what will be the difference with H263 exactly ? What if I would use a custom matrix such one of the matrix's made by *mp4 guy (http://forum.doom9.org/showthread.php?p=887125) ?

"MPEG" and "H.263" are two different types of quantization, supported by the MPEG-4 ASP format. As you may know, MPEG-4 ASP has evolved from H.263.

The "MPEG" type is known to retain more detail/sharpness, but also has a tendency to produce "blocking" artifacts. At the same time the "H.263" type is known to better avoid blocking and to produce a slightly smoother image.

Also note that only the "MPEG" type can use custom matrices. The "H.263" type can NOT do this. Also there is NO custom matrix for the "MPEG" type that would resemble the "H.263" type.

Should I use Adaptize Quantization and how ?

You probably want to use Adaptive Quantization for the same reason you want to use it with x264 :)
http://forum.doom9.org/showpost.php?p=1567194&postcount=9

Xvid supports two AQ algorithms. The old "Lumi Masiking", which is not very good. And "Variance Masking", the AQ algorithm ported back from x264.

[sirt]

Why are you obscuring filenames in your posts?

My name appears on the folder and I was lazy to rename it.

[sirt]

Why you don't want any bitrate fluctuations?

As a matter of fact, the "complexity" of almost any video varies over time. In order to get a consistent quality, the bitrate has to be adjusted with respect to the content of the video.

That's what the idea of 2-Pass mode is all about: Allow the bitrate to "fluctuate" (in other words: Move the bits to where they are actually needed!) and still be able to hit the target average(!) bitrate.

If you would be encoding with a 100% constant bitrate (i.e. no bitrate fluctuations), this would result in quality fluctuations, which certainly isn't desired ;)



"MPEG" and "H.263" are two different types of quantization, supported by the MPEG-4 ASP format. As you may know, MPEG-4 ASP has evolved from H.263.

The "MPEG" type is known to retain more detail/sharpness, but also has a tendency to produce "blocking" artifacts. At the same time the "H.263" type is known to better avoid blocking and to produce a slightly smoother image.

Also note that only the "MPEG" type can use custom matrices. The "H.263" type can NOT do this. Also there is NO custom matrix for the "MPEG" type that would resemble the "H.263" type.



You probably want to use Adaptive Quantization for the same reason you want to use it with x264 :)
http://forum.doom9.org/showpost.php?p=1567194&postcount=9

Xvid supports two AQ algorithms. The old "Lumi Masiking", which is not very good. And "Variance Masking", the AQ algorithm ported back from x264.

Yes another idea came to my mind...comparing an x264 encode with an xvid one ; I know x264 can win but I want to see how the XviD will proceed such-and-such video. I was still using not long time ago to save some of my favorite movies because it was faster and not killing my computer.

About the fluctuations : indeed I commited a mistake by expressing my goals like that. I just wanted to say I want the encoder not to overtake the bitrate limitation. With 1,1/1,1/1,1 quantization it didn't respect the final size intended whereas it worked almost correctly with 1,4/1,16/1,8 but I gave a try to a custom matrix called "For Home Enternainment" but it didn't reach my final size and the bitrate setted. That's why I was asking how to set up that Quantization window. I also tried 1,4/1,2/1,4 but again the final size intended was not respected. I simply don't know how to set that !

Then I will give a try to the Variance Masking. Futhermore this one seems to new (I just got latest XviD few days ago) because such adaptitve method was not avaible before. D

[LoRd_MuldeR]

Just a side-note: By forcing all the min/max quantizers to 1, you completely cripple all rate-control!

If you now use 1-Pass ABR mode or 2-Pass mode, you will effectively get 1-Pas Constant Quantizer mode with a quantizer of 1, which is not the intend of these modes. Needless to say that the file will also be LARGE ;)

Also: If you want to compare your Xvid encodes to your x264 encodes, you must ensure that the file size (i.e. average bitrate) of the files you'll compare is identical. Otherwise the comparison is worthless!

[sirt]

Just a side-note: By forcing all the min/max quantizers to 1, you completely cripple all rate-control!

Yes but I don't really know what quantizers will help to obtain the final size intended by the bitrate setted. What I'm sure is that letting 1,31/1,31/1,31 will result in the poorest quality possible ; at least, it's what I read on a web guide where they advise to use 1,4/1,16/1,8 but I ignore why.

Also: If you want to compare your Xvid encodes to your x264 encodes, you must ensure that the file size (i.e. average bitrate) of the files you'll compare is identical. Otherwise the comparison is worthless!

I do but only for short clips or movies (I couldn't encode a movie with x264 even if I used medium preset) and honestly x264 always win the toss (in 2 pass, I haven't tried ABR) ; but I still use it for some movies or non HD TV Rips ; what I simply prefer is using LAGARITH and that's all (or MSU) ; I'm not sure if there is a "lossless" mode with xvid, but I guess not.

[Asmodian]

Yes but I don't really know what quantizers will help to obtain the final size intended by the bitrate setted. What I'm sure is that letting 1,31/1,31/1,31 will result in the poorest quality possible ; at least, it's what I read on a web guide where they advise to use 1,4/1,16/1,8 but I ignore why.

The limits there don't really help with anything; they just limit what the rate control can use. Because quant 1 is much larger than quant 2 and not really any better looking I actually like using 2,31 as min,max. This gives Xvid the freedom to use whatever quant it thinks is best but if the file is looking undersized it won't go to the mostly pointless quant 1 just to hit the required bit rate.

IIRC the default was set to 1-31 because less experienced users would complain about undersized files using 2-31. Adding the quant 1 option doesn’t increase the visual quality of the encodes but it does make them larger, preventing angst about undersized files for users.

Remember those are just the min/max. Xvid doesn't have to use any particular quantizer unless it thinks it is the best option to hit a desired file size.

A caveat: I have not used Xvid in several years as x264 has superseded it.

[LoRd_MuldeR]

Yes but I don't really know what quantizers will help to obtain the final size intended by the bitrate setted.

And you, as a user, don't have to know ;)

In the bitrate-based encoding modes, like 1-Pass ABR or 2-Pass, the rate-control algorithm will pick the "right" qunatizers to hit the target average(!) bitrate that you have chosen.

By changing the "minimum/maximum quantizer" options you only restrict the interval of quantizers from which the rate-control algorithm can choose - which can easily lead to missing the desired average bitrate!

If, for example, you set both, the minimum and the maximum quantizer, to a value of 1, then you force the rate-control algorithm to always pick 1, regardless of the chosen bitrate :rolleyes:

You would only limit the minimum (or maximum) quantizer, if you believe that the rate-control algorithm sometimes goes too high (or too low) for no good reason...

[sirt]

But I want to know ! It's possible I am not able to understand without any specifical knownedge ; so, according to you, there is no need to set that in so far as the quantizer will AUTOMATICALLY choose the adequate quantizers and lowering the values could restrict its work, am I right this time ? You will say I am asking too much but let's try again...how can you be sure the "right" quantizer will be chosen ? And what do you mean by "right" ? I am also confused with the I-Frames and B-Frames chosen during an encode...how is that possible the encoder will choose the right one as a reference ? From what I have read it is possible to set manually what will be a I-Frame or B-Frame trough an override text but it is probably dedicated to XviD or x264 gurus. Of course I won't assume to tell you if a good frame was used a reference by the encoder to reconstruct other frames around basing its work on that refrenced one but...what if this reference frame is one of the worse from the video such as a ringed or destroyed one ? Then I think the segment near that frame will be encoded with a bad quality and I couldn't control that.

[LoRd_MuldeR]

But I want to know ! It's possible I am not able to understand without any specifical knownedge ; so, according to you, there is no need to set that in so far as the quantizer will AUTOMATICALLY choose the adequate quantizers and lowering the values could restrict its work, am I right this time ? You will say I am asking too much but let's try again...how can you be sure the "right" quantizer will be chosen ? And what do you mean by "right" ?

The "right" quantizers are that which will hit desired average(!) bitrate you have selected.

If you want to know which quantizers happened to be the "right" ones for a particular source and for a particular target bitrate, just take the final output file and throw it at some analyze tool which can show the qunatizers ;)

I am also confused with the I-Frames and B-Frames chosen during an encode...how is that possible the encoder will choose the right one as a reference ? From what I have read it is possible to set manually what will be a I-Frame or B-Frame trough an override text but it is probably dedicated to XviD or x264 gurus. Of course I won't assume to tell you if a good frame was used a reference by the encoder to reconstruct other frames around basing its work on that refrenced one but...what if this reference frame is one of the worse from the video such as a ringed or destroyed one ? Then I think the segment near that frame will be encoded with a bad quality and I couldn't control that.

As explained earlier, scenecut detection is used to detect scene boundaries. If a scenecut is detected, a key-frame (I-Frame in the case of Xvid, IDR-Frame in the case of x264) will be placed at this point.

Deciding whether a particular frame "in the middle of a scene" should be coded as a P-Frame or as a B-Frame is a complex task and requires pretty complex algorithms/heuristics.

If you want to know how x264's adaptive B-frame decision algorithm works (actually there are two), you will have to study the source code. Many more simple encoders use a fixed number of consecutive B-Frames...

[sirt]

I think it will not be easy to study the C files ; I understand the language used (I study C/C++ on my own) but it would be a payoff I could understand like 5000 lines of code myself.

Then the question is perhaps more complex : what will be the most accurate bitrate to use (with XviD or x264 in 2 Pass mode ? Of course you could reply it would be around 800/900 for a movie and a bitrate calculator will help to find the proper value to reach 700 mb or 1400 mb (and same idea for HD Ready or HD Full Blurays) ; I know setting a high bitrate means the quality will - normally - be better, but in that case why not just setting a bitrate of 20000 for each encode and that's all in spite of the large file I might collect. More precisely, isn't each time a bitrate not to exceed (I only refer to high bitrates) ? I wonder for example if setting the bitrate too high (like 50 000 for example) won't give a bad output even though I've read in many places a high bitrate could never be bad.

[Guest]

why not just setting a bitrate of 20000 for each encode and that's all in spite of the large file I might collect If you don't care about file size, then leave it uncompressed. Your questions are not making much sense to me.

[sirt]

If you don't care about file size, then leave it uncompressed. Your questions are not making much sense to me.

neuron2, well I care about file size but I have not a precise idea of how to set a bitrate if you prefer. Of course there are "commercial" standards such as the one mentionned above (700 MB for a 1CD movie, eventually 1400 MB for a 2CD movie, 4 GB for a HD heavy Blu Ray and 8 GB for a HD Full Bluray) but my question was : is that accurate ? How would you know the "accurate" bitrate ? I could answer myself : it depends on the needs...of course it does but I wonder if setting a high bitrate could have negative effects that could be avoided by a lower one. That' why I ask for a "accurate" bitrate (depending on the source) not to be exceeded.

[Guest]

It's all subjective. There is no "accurate bitrate" to find. You have two conflicting goals: minimizing file size and minimizing quality degradation. Only *you* can decide which point of the tradeoff satisfies you while adhering to the constraints of your application domain. The only negative for setting a high bitrate is that the file size grows, and if you are streaming it might exceed the channel capacity. You also have a "channel" from your hard disk and uncompressed HD (the maximum bitrate!) can exceed the hard disk maximum read rate.

[sirt]

Okay, let's do it simple...if you are encoding in a 2 pass mode how do you (personally) choose your bitrate regarding the input source, your expectations ect ?

[Groucho2004]

Okay, let's do it simple...if you are encoding in a 2 pass mode how do you (personally) choose your bitrate regarding the input source, your expectations ect ?
Is this question directed at neuron2 or anybody?

[sirt]

I was excpeting him to answer but any advise is welcomed of course.

[LoRd_MuldeR]

Okay, let's do it simple...if you are encoding in a 2 pass mode how do you (personally) choose your bitrate regarding the input source, your expectations ect ?

If you use 2-Pass mode, this usually indicates that you need to hit a specific average bitrate or a specific file size. It's both the same, because the file size is a direct consequence of the average bitrate.

So, if you need to hit a specific file size, then you know what that size that is. For example 4,7 GB to make the video fit on a DVD-R. Or maybe 25 GB to make it fit on a BD-R.

Now that you have made clear to yourself which file size you need to hit, calculating the corresponding average bitrate is simple math. However take into account the space for the audio track and the container overhead!

Also note: In case you do not need to hit a specific file size, there is not much reason to use 2-Pass mode. Instead you would simply use CRF mode with the highest possible CRF value that still satisfies you eyes.

(In case of Xvid there is no CRF mode, unfortunately, so you would have to use CQ mode instead - or switch to x264 ^^)

[Groucho2004]

Okay, let's do it simple...if you are encoding in a 2 pass mode how do you (personally) choose your bitrate regarding the input source, your expectations ect ?
If you want to do a 2 pass encode, you obviously want to hit a certain target size. The bitrate depends heavily on the source so the first thing I'd do is running a compressibility test to figure out the minimum bitrate and take it from there.

Edit: LM beat me to it - more or less.:)

[Guest]

Okay, let's do it simple...if you are encoding in a 2 pass mode how do you (personally) choose your bitrate regarding the input source, your expectations ect ? I use one pass at the highest bitrate that my channel will allow.

[henryho_hk]

See the sticky thread for XviD Presets and Compatibility Test.

Compatibility Test gives a figure telling u how your desired bitrate and your source fit with each other.
XviD Presets help you to choose the settings accordingly.

Or you can take the easy way and stick with H.263+VAQ. Though you really consider x264 if the CT result is 45 or below.

[hello_hello]

By changing the "minimum/maximum quantizer" options you only restrict the interval of quantizers from which the rate-control algorithm can choose - which can easily lead to missing the desired average bitrate!

I inquired about this in my own thread quite a while ago but didn't find anyone who could answer it at the time, so as it's relevant to this discussion....

I've checked how AutoGK sets up the Xvid encoder on quite a few occasions just to see what it's doing, and AutoGK seems to set the min and max quantizers fairly close together while changing them according to the result of the compression test.
For example if the compression test result is around 70% it'll use quantizers of 2 and 3, at 60% it'll use 2 and 4, at 40% it uses 2 and 5 etc.... in the case of AutoGK it doesn't seem to effect it's ability to achieve a particular file size.

I assume the reason for doing so would be to keep the quality of individual frames higher than Xvid otherwise would using the default quantizers of 1 and 31 (or 2 and 31). Would that be correct?
If I remember correctly limiting the maximum quantizer stops Xvid from being able to decrease the file size as much as it otherwise would, but the quality of individual frames of motion scenes tends to remain higher due to limiting the max quantizer (well the same applies to static scenes but I think I could only actually see a quality difference in frames with motion).

Anyway, any thoughts on the AutoGK way vs default quantizers?

One other question for which I never found an answer.....
My basic understanding is that Xvid's compression/quality.... say 70% for example.... is literally obtained by running a complete Q2 single pass encode and then using 70% of the final file size for a 2 pass encode (is that correct?). Or it's normally worked out by running a compression test on x% of the video and then doing the math, but given AutoGK reduces the maximum allowed quantizer, you'd have to calculate the quality differently.... I assume?
For instance when I ran a single pass Q2 encode on a small file (where AutoGK used min and max quantizers of 2) the final file size was 31.3MB. When I ran a 2 pass encode using a file size of 22MB (70%) AutoGK reported the quality as being only 65% (while using min and max quantizers of 2 and 4). Any idea how AutoGK does it's quality calculations?

[hello_hello]

Okay, let's do it simple...if you are encoding in a 2 pass mode how do you (personally) choose your bitrate regarding the input source, your expectations ect ?

When it comes to Xvid I generally still use AutoGK, let it run it's compression test, and change the file size if need be to achieve around 70% to 75%. I use a fixed width so AutoGK won't try to adjust the resolution according to the compression test result.
Unfortunately AutoGK can make that a little hard in that you can't run a compression test first and then pick a file size. One of the few current GUIs which will run a compression test first (if memory serves me correctly) is HDConvertToX.

For x264 I pick a CRF value I'm happy with (usually 19) and let the bitrate be whatever it needs to be.

[sirt]

Lord_Mulder, I know how to calculate a bitrate or a final output size by taking in account some crucial parameters such as the AR, the complexity (such as the use of B-Frames or not), duration (i.e. number of frames) ect but what I'm still wondering is that if there is a "theorical" bitrate for a specific video that should not be outstripped ; then my question is not "how could I reach that specific final size or that bitrate" but rather "should I avoid going under either bitrate not to destroy, blur, smooth my video ?" ; By tweaking I would say it is not that hard to answer for a low bitrate : I tried myself a very low one and I experienced huge blocks and blurring areas in a master that is almost perfect, but I don't know if something like that could happen with a too high bitrate. I'm confused because I read using a very high bitrate would never cause anything bad to the video (counter to a low one) but it would ensue in a very large file, is that right ?

[Groucho2004]

By tweaking I would say it is not that hard to answer for a low bitrate : I tried myself a very low one and I experienced huge blocks and blurring areas in a master that is almost perfect, but I don't know if something like that could happen with a too high bitrate.
There is no such thing as a "too high bitrate". If you pick a very high bitrate, the encoder will eventually saturate.

I'm confused because I read using a very high bitrate would never cause anything bad to the video (counter to a low one) but it would ensue in a very large file, is that right ?
I'm confused that you have to ask this question after all the info you got in the last few days. :confused:
Maybe it's a language thing...

[Guest]

I'm confused because I read using a very high bitrate would never cause anything bad to the video (counter to a low one) but it would ensue in a very large file, is that right ? I answered that here:

http://forum.doom9.org/showthread.php?p=1569901#post1569901

Did you read it?

[LoRd_MuldeR]

Lord_Mulder, I know how to calculate a bitrate or a final output size by taking in account some crucial parameters such as the AR, the complexity (such as the use of B-Frames or not), duration (i.e. number of frames) ect but what I'm still wondering is that if there is a "theorical" bitrate for a specific video that should not be outstripped ; then my question is not "how could I reach that specific final size or that bitrate" but rather "should I avoid going under either bitrate not to destroy, blur, smooth my video ?" ; By tweaking I would say it is not that hard to answer for a low bitrate : I tried myself a very low one and I experienced huge blocks and blurring areas in a master that is almost perfect, but I don't know if something like that could happen with a too high bitrate. I'm confused because I read using a very high bitrate would never cause anything bad to the video (counter to a low one) but it would ensue in a very large file, is that right ?

Don't make things more complex than they are ;)

The output file size is defined as follows and that's it (basically):
file_size = average_bitrate × duration

It follows that you can calculate the target bitrate, given the desired file size:
file_size / duration = average_bitrate

Of course some videos may need a higher average bitrate than others to retain the same level of quality. But in case you need to hit a specific file size, you don't need to care about that!
As your target file size is known, the target average bitrate is also known. Now all you can do is getting the best out of the available bitrate. That's exactly what 2-Pass mode does.

And, as said before, if you don't need to hit a specific file size, then 2-Pass Mode (or 1-Pass ABR) is not the right thing to use. Instead CRF mode (or CQ in case of Xvid) is what you'd want.

[Guest]

sirt never told us if he is trying to target a specific file size. All we know is he is "experimenting".

[sirt]

Thanks to all of you for advise and answers. As neuron2 says, I just experiment and "waste my time" in a way, I don't target anything special.

[manono]

I assume the reason for doing so would be to keep the quality of individual frames higher than Xvid otherwise would using the default quantizers of 1 and 31 (or 2 and 31). Would that be correct?

Yes, that's correct. The idea is to make a reasonable attempt at a CQ encode, while at the same time doing it for a defined file size. For that reason the quants are heavily restricted.

[henryho_hk]

You may try a CQ encode with reference to the XviD 45% or 58% comp. check presets:

CQ 3, b-frames: max. 2 consecutive, ratio 1.62, offset 0
MSP=6, VHQ=4 (or 1 if too slow or blurry), VHQ for b-frames, Trellis, chroma ME, chroma opt.,
custom matrix=SixOfNine (or EQM-v3-HR if too big)