Compressibility & XviD-Custom-Matrices

[SpikeSpiegel]

Which is the most precise way* to check, a priori (before the encoding process or, at least, before 2nd pass), the compressibility (for XviD) of a DVD?

Usually I use BPF, but in an "adaptive" way, according to motion, definition and "average brightness" of the source: I know that it's all but precise, but...life is too short to do other tests! :devil:

Is compressibility directly inverse-proportional to resolution of the output file? (I mean: is their connection REALLY linear?)
I always believed that, but I'm starting to have some doubts...

Which Quant. Matrix (custom included) would you use at a given compressibility value (of the scale of your favourite comp-test-program)?
4example: min25%-max30% --> H.263; min45% --> MPEG; min80% --> "JohnSmith's perfect picture"

Thanks in advance for your help!

* "eyes" not included ;) Something like GordianKnot Comp.test or ArCalculator's one...

[DarkDudae]

Compressibility is directly inverse-proportional to resolution of the output file, but it is not 100% linear. For example, you do a comp. check with ARcalculator at 640xYRES, and you get a compressibility of 50%. Now, you change your resolution to 672xYRES, and you will see that compressibility is lower, for example: 46%. That 46% is only an estimation of real compressibility with that resolution, but the only way to get max. accuracy, is to repeat the comp.check with new resolution. Anyways, estimations usually works pretty fine, and it is useless to repeat the test. Since XviD, as other codecs, divide image into blocks, for motion vectors and other stuffs, the results with different inputs resolution allways are different. That is the reason that first and second pass of XviD and all codecs need the same input file in each pass.

About Matrix quantizations, I usually look for 70% of compressibility with all quant m. if I use a normal script, or 50-60% if I use AVSOptimizer.

Greetings

[SpikeSpiegel]

Originally posted by DarkDudae
Since XviD, as other codecs, divide image into blocks, for motion vectors and other stuffs, the results with different inputs resolution allways are different.

Thanks for your quick answer!

Well, considering how MPEG4-based-codecs work, this problem should be more noticeable when changing resolution from mod-16 (more compressible) to non-mod-16 (less compressible) and vice-versa.
But I just did a little test about it and seems that 16-divisibility's not so much important:
640*256 --> real 65,74%
640*260 --> estimated 64,73% / real 63,04% Much less than expected!
Anyway, if you mean that, using only standard res., this non-linearity is negligible, that's enough for me! :)

P.S. Could you give me some tips about AVS Optimizer settings?
I always thought that +adaptiveness=+quality (that's why I always use trellis), so this may be really useful!

[Soulhunter]

Originally posted by SpikeSpiegel
Thanks for your quick answer!

Well, considering how MPEG4-based-codecs work, this problem should be more noticeable when changing resolution from mod-16 (more compressible) to non-mod-16 (less compressible) and vice-versa.
But I just did a little test about it and seems that 16-divisibility's not so much important:
640*256 --> real 65,74%
640*260 --> estimated 64,73% / real 63,04% Much less than expected!
Anyway, if you mean that, using only standard res., this non-linearity is negligible, that's enough for me! :)


Hi Spike, and welcome to this forum !!!

The waiting period is finally over, huh... :D


Ok, something general about non-linear compressibility !!!

If you downsize a picture you wont only lose some pixels...

You will lose (or enhance) some details as well -> through interpolation (amount is resizer dependent) !!!


Example:

Bilinear resizing will blur the image a little bit, this way you could gain some extra compressibility...


Ramble perhaps ???

Gah, I assume you already knowing this anyway... ;)


Bye

[DarkDudae]

Originally posted by SpikeSpiegel
P.S. Could you give me some tips about AVS Optimizer settings?
I always thought that +adaptiveness=+quality (that's why I always use trellis), so this may be really useful!

If you have a clean source (DVD i.e), then do not check any cleaning filter and do AVSOptimizer partial pass. You can check lumafilter or undot, witch are not aggresive with image.

If your source is noisy, check a general cleaning filter in AVS Edition before AVSOptimizer.

If you need more info, you can visit ARCalculator Thread. (http://forum.doom9.org/showthread.php?s=&threadid=78704&highlight=ARCalculator)

Greetings

[SpikeSpiegel]

Originally posted by Soulhunter
Hi Spike, and welcome to this forum !!!

The waiting period is finally over, huh... :D


Ok, something general about non-linear compressibility !!!

If you downsize a picture you wont only lose some pixels...

You will lose (or enhance) some details as well -> through interpolation (amount is resizer dependent) !!!


Example:

Bilinear resizing will blur the image a little bit, this way you could gain some extra compressibility...


Ramble perhaps ???

Gah, I assume you already knowing this anyway... ;)


Bye

Hi, Soulhunter!

Finally I can show my badly-lanczos-resized-avatar (true... :D ) even here!

I must admit that this doesn't sound new to my ears, but thanks, anyway! ;)

[SpikeSpiegel]

Originally posted by DarkDudae
If you have a clean source (DVD i.e)

Well, I wonder if a clean-flat-surface ever existed in a DVD! :devil:

Ok, asap I'm gonna give a try with lumafilter and undot (never used); about the other filters, I already know Convolution3D even if I've never used it (too aggressive), while I think fluxsmooth to be one of the best denoisers (good choice!)

Muchas gracias!

[thoralf]

Originally posted by SpikeSpiegel
[...] to non-mod-16 (less compressible) [...]
wasn't that a big no-no?

[Soulhunter]

Originally posted by SpikeSpiegel

Well, I wonder if a clean-flat-surface ever existed in a DVD! :devil:

Ok, asap I'm gonna give a try with lumafilter and undot (never used); about the other filters, I already know Convolution3D even if I've never used it (too aggressive), while I think fluxsmooth to be one of the best denoisers (good choice!)

Muchas gracias!

Have you already tried VagueDenoiser ???

IMO its unbeatable for slight denoising... ;)


Bye

[lordadmira]

Just a few thoughts. The best way to check the compressibility of a video is to just watch the video. A computational approach will only give u meaningful results if u actually run the encode. Ur eye can tell u how much motion there is and how complex the image is. Sure it's a developed skill. ;)


LA

[DarkDudae]

Just a few thoughts. The best way to check the compressibility of a video is to just watch the video. A computational approach will only give u meaningful results if u actually run the encode. Ur eye can tell u how much motion there is and how complex the image is. Sure it's a developed skill.

I am not agree with that. If you examine a video with your own eyes, maybe you get a general idea of its compressibility and the optimal resolution for it.

However, there are hundreds of details that must be counted in the visual test: Video Aspect Ratio, length, colours, luminance, motion, final bitrate... IMHO, too many details. If you do a compressibility check, you are reading the data directly from xvid codec stats, witch can tell you exactly what video needs.

For example:
Terminator 3
Length: 1:44:39
It is not a dark movie, and it has a lot of high motion scenes. A visual test, would suggest a low resolution for the encoding, however, if you do a comp. check, you can see it is a very compressible film, so you can go for higher resolutions and get better results without undersize risk.

Greetings!

[lordadmira]

Originally posted by DarkDudae
If you do a compressibility check, you are reading the data directly from xvid codec stats, witch can tell you exactly what video needs. True, but then u no longer have a "check" but a full first pass of the video.

[DarkDudae]

Originally posted by lordadmira
True, but then u no longer have a "check" but a full first pass of the video.

You don´t need a full first pass. A 5% of the video with a SelectRangeEvery(280,14) in avisynth script is enough to get average compressibility.

Greetings

[lordadmira]

That's a very big if. That will only work if the frames grabbed are a representative sample of all the scenes in the video. Given that a video is not a random distribution phenomenon but a man-made ordered construct, I don't think any selective frame grab will give u accurate results in the general case.

[Teegedeck]

Of course not. But experience tells it gives you a rather accurate guess. Though I prefer to do several full first passes if necessary.

[SpikeSpiegel]

Originally posted by thoralf
wasn't that a big no-no?
Well, it should be avoided because it isn't MPEG4 compliant and because it makes the codec work with fractions of 16*16 blocks, etc..
...but, practically, there are no real negative aspects.

[SpikeSpiegel]

Originally posted by Soulhunter
Have you already tried VagueDenoiser ???

IMO its unbeatable for slight denoising... ;)


Bye
True!
As I already told you, I've done some tests and I can say that there are no many spatial denoise filters that can keep as much details!
Anyway now I'm searching for temporal filters, like fluxsmoothT, that can erase selectively "dynamic noise" (IMHO, the most annoying) without reducing definition (that is almos inevitable with normal denoisers).
This one works pretty well, but it has no motion detection, so it may erase details in hi-motion scenes (but I noticed it only once).

[SpikeSpiegel]

@ lordadmira & DarkDudae
We all know that the best way to decide how to encode a movie is to use both "a computational approach" and "eyes" (for resolution, filters, etc.), but, as DarkDudae's saing, is almost impossible to forsee how those settings (matrix, motion/quant options, filters, etc.) can work with those characteristics of the movie (motion, average brightness, panning, noise, etc.) because there are too many "variables".
Again, a program can't tell what will be the absolute perceived quality of the output file, but it can tell which are the (relatively) better settings to use to improve compressibility, that's another reason why they're so important.

@ DarkDudae
"You don´t need a full first pass. A 5% of the video with a SelectRangeEvery(280,14) in avisynth script is enough to get average compressibility."

Interesting, this way almost all scenes are checked, so, even if the movie structure isn't linear, the results are reliable anyway...really interesting :D

[Soulhunter]

Originally posted by SpikeSpiegel
Well, it should be avoided because it isn't MPEG4 compliant...

IIRC the MPEG4 specs say the minimum is a multiple of X4/Y4 !!!

But yes, efficiency wise X16/Y16 is the way to go...


Bye

[SpikeSpiegel]

Originally posted by Soulhunter
IIRC the MPEG4 specs say the minimum is a multiple of X4/Y4 !!!

But yes, efficiency wise X16/Y16 is the way to go...


Bye

DivX & XviD can handle even X4/Y2 res., but I've read somewere that the smallest blocks that MPEG4 can analyse are 16*16, so, if the res. is not X16/Y16, some blocks are partially "out of the borders"

Strangely, this doesn't affect compressibility nor it creates errors...

[lordadmira]

Yes it works on 16 x 16 macroblocks no matter what. If there are some blocks "hanging off the edge" because the rez is not by 16 it clips the junk portion of the block on playback. I don't know what optimizations if any it takes when filling in the junk portion of the hanging blocks. It would be good if it picked some data that would aid in the DCT compression of the good portion of the block. If I'm off base here somebody will yell at me eventually.

[Soulhunter]

Originally posted by SpikeSpiegel

DivX & XviD can handle even X4/Y2 res., but I've read somewhere that the smallest blocks that MPEG4 can analyze are 16*16, so, if the res. is not X16/Y16, some blocks are partially "out of the borders"

Strangely, this doesn't affect compressibility nor it creates errors...

Think you are right, X4/Y2 it was, sorry... ;)


EDIT:

Or is it a decoder restriction rather than a encoder one ???


Bye

[lordadmira]

I just encoded a 222x166 clip and it worked just fine. That's x2/y2. I thought it only had to be even...

[Soulhunter]

Originally posted by lordadmira
I just encoded a 222x166 clip and it worked just fine. That's x2/y2. I thought it only had to be even...

Wow, does it work with all decoders... :confused:

Maybe X1/Y1 works as well !!!


Bye

[SpikeSpiegel]

Originally posted by SpikeSpiegel
DivX & XviD can handle even X4/Y2 res

A little precisation:

X4/Y2 is a DivX (encoder) restriction, but, as lordadmira noticed, XviD can handle even X2/Y2.

[lordadmira]

Originally posted by Soulhunter
Wow, does it work with all decoders... :confused:

Maybe X1/Y1 works as well !!! It "should" work in any decoder. Odd resolutions won't work because the YV12 color format requires 4 pixel squares. That's where the even resolution requirement comes from.

[Soulhunter]

Wow, learning something new every day... :D


Thanks n' Bye