Implementing compressibility checking for Xvid X264 and if possible other codecs

[OvejaNegra]

can anybody explain to me the mechanics behind a compressibility check for xvid and X264 (well maybe is applicable to other codecs like theora or mpeg2).

The info i found on the network is a little ambiguos and confusing.
For example some people say that is only create a file with selectrange on avisynht to get the 10 or 5% of the video and make a compression with the desired codec using constant quality settings.The final filesize gives an estimate about how far or near you are from your target file size. If its to much the difference, then change resolution or apply filters to get close to the desired size.

But AutoGk does something different, it create a file called frames.log. What is it?
And using selectrange on avisynth does not make the codec use lots of keyframes for the many scene changes? Is not that a little artificial?

I want to make a tool to make easy the process of comp checking for me thats why i ask.
If the question is not appropiated for this forum section, please, feel free to move it to another thread.

Thanks again and sorry for my english

[Darksoul71]

Heya,

wow, that kind of a tough question. No need to excuse for your English.
It is quite understandable and mine is far from perfect.

I will try to explain what I know from this. Keep in mind that I am no video encoding expert.

I do not remember anymore what AutoGK did with the frames.log....sorry...

A compressibility check basically is a test in order to determine how well a movie compresses. If you do not use bitrate / size-based encoding but a constant quality method, then your encoder will spent as much bitrate needed for the video to meet a certain level of visual quality.

For x264 this is called CRF:
http://mewiki.project357.com/wiki/X264_Settings

The other "non-bitrate-based" encoding method which is often supported, is constant quatizer. Different from a bitrate-based approach where the quantisation of a video is adjusted to meet a certain size, simply every frame of the video is encoded with a constant quantisation factor.

Often a certain value for constant rate factor or constant quantiser will result in a desirable quality. This is an important implication ! By using a CRF value (for h264) or CQ value (DivX / MPEG2) you can be sure that the video looks good. That is quite different from a bitrate-based approach. If you encode a video with a bitrate / sized-based algorithm you will never achieve a good ratio between size and quality for all movies. A movie which compresses quite well and is encoded at a high bitrate will waste space. A movie with a bad compressibility which is encoded at the same bitrate will suffer quality decrease. A multi-pass based bitrate encoding algorithm will make degration in quality less noticable but it still will be there.

There were several tools back then which used a constant quality / constant quatizer approach to meet the desired target size. Small snippets evenly distributed across the whole movie (mostly 1-3%) were used for quick encoding. You usually started at a somewhat higher value (may be the middle of your quantizer / quality scale) and encoded those snippets. Based on the percentage of the movie the final size of the movie was calculated. Then the new quatizer value was calculated by some mathematical approach (e.g. bi-section method, Newton, etc). Most likely you were able to empirical obtain the CQ value required to meet your target size. If the obtained CQ-value to get your desired target size was not close enough to your acceptable CQ value, then you could decide what to do: Adjust your output resolution, add some filters to enhance compressibility, use a higher target size.

If you have a CQ value where you know that things look good, then you can use this as some "quality-o-meter". A value which results is better quality than your "CQ border" is of course more desirable than one producing worse quality. Search for the old D2SRobBa tool which was used for DVD2SVCD.

Personally I have moved away from any bitrate-based encoding. For my BD backups and HDTV I almost always use CRF-based h264 encoding with values around 20-22. Usually I can hardly notice a big difference between the original source and the shrinked video. Since most peoples are less space-limited than back then (either storing video on NAS / HTPC or burning them to BD), I am afraid that a compressibility tool adds little to no value today.

What IMO is really outdated today though are bit-rated only based encoding algorithms as featured in many video conversion tools (DVDFab, Freemake, Pavtube, etc). They all either waste space or quality. This being said I can hardly imagine a use-case for a compressibility tool.

Sorry for the long rant !

Feel free to ask questions as they arise !

HTH,
D$