Thoughts on compressibility check, please comment

[bira]

I'm really missing a compressibility check in Megui!!!

My encodes are always based on a target file size. Usually 1/3 of a DVD is enough for most of my encodes. But there are times when 1/3 or even 1/4 of a dvd is overkill.

Everyone knows that 'talking heads against a black background' needs less bits than 'talking heads against a football match background'. So, there lies the problem.

Right now, I encode targetting 1/4 of dvd and if it turns out bad I just encode again. I don't have to tell you that this is as non-productive as it can be.

So, without the compressibility check, how do you come up with the bitrate or file size needed?

I don't need the compressibility check to give me a exact number, but I need a hint of what bitrate should be enough.

Can't we have a hint anymore?

[berrinam]

My encodes are always based on a target file size.Well, there's your problem. The should be crf (constant ratefactor/constant quality).

I don't need the compressibility check to give me a exact number, but I need a hint of what bitrate should be enough.

Can't we have a hint anymore?
I'll give you a hint: always target 1/3 DVD, and don't worry about the wasted space. Or, I could add a random number generator to MeGUI, and tell you to use that.

My point? The compressibility check is a placebo. People think, since the computer spends several minutes checking, "Wow, it's such a powerful computer and it spent all that time checking -- it must at least be on the right track". This is wrong, because compressibility can simply vary so much within a film, and it so greatly depends on your sampling scheme.

If you are interested in doing constant quality, then use crf with an acceptable value for you (most people look around 18 -- 26). If you aren't, then it is just a guessing game, and I don't really think a compressibility guess would be a true help to you.

[bira]

Since I backup to DVD's, I must have a target file size. So, constant quality is not a option for me.

What have changed from divx and xvid to x264 that I can't have a compressibily 'guess' anymore? Or was it always useless? It seemed to be working for me, though.

My first passes are at a rate of 35fps and second passes at 10fps. I don't understand how 2 passes really work, but couldn't I use the first pass or even a faster pass to come up with a quality estimate?

[Sharktooth]

Compressibility check should be done with the same settings that will be used for the final encode, at least for x264 (that's due to the many encoder options that can radically change the compressibility). That's why it is so hard to implement. It will require a complete rewrite of the app workflow.

[check]

What have changed from divx and xvid to x264 that I can't have a compressibily 'guess' anymore? Or was it always useless? It seemed to be working for me, though.

Nothing has changed, it wasn't very accurate in the first place :P

Since I backup to DVD's, I must have a target file size. So, constant quality is not a option for me.

I'm not sure I understand. You want constant quality, but you wish to achieve it with semi-constant filesize? It's one or the other matey :P
Find a crf setting that you like and encode to it. If you were happy with files that came out around 1/4 DVD before, they will still come out around the same size, just those that could be smaller will be and those that would have been bigger will come out at 1/3 DVD. You'll end up with free space on some of your DVDs but at least you will have constant quality.

[bira]

I'm not sure I understand. You want constant quality, but you wish to achieve it with semi-constant filesize? It's one or the other matey :P
Find a crf setting that you like and encode to it. If you were happy with files that came out around 1/4 DVD before, they will still come out around the same size, just those that could be smaller will be and those that would have been bigger will come out at 1/3 DVD. You'll end up with free space on some of your DVDs but at least you will have constant quality.

... just those that could be smaller will be and those that would have been bigger will come out at 1/3 DVD. You'll end up with free space on some of your DVDs but at least you will have constant quality.

That's what I don't want, because I could end up with too much free space that is too little for another encode.

Just an example: let's say I set a constant quality and end up with 3 movies each 1.01/3 of a DVD. You see? I'm pretty sure I could squeeze the 3 movies in a dvd without perceived quality hit.

That's why a quality estimate would be nice. That way, I would know if I need 1/2, 1/3, 1/4 or 1/5 of a DVD for that encode.

[bira]

Compressibility check should be done with the same settings that will be used for the final encode, at least for x264 (that's due to the many encoder options that can radically change the compressibility). That's why it is so hard to implement. It will require a complete rewrite of the app workflow.

I don't know about other megui users, but I would love to have that!

[elguaxo]

My encodes are always based on a target file size too. I use the first pass as my compressibility check and look at the 'final ratefactor' after the first pass as my hint.

[bira]

...and look at the 'final ratefactor' after the first pass as my hint.


Where can I find that "final ratefactor"?

[foxyshadis]

What do you do if three movies come out to 1/4 DVD, and then the fourth needs 1/3?

If it was really important, you'd ratecontrol all of the movies as one whole, rather than encoding the three or four separately. That way each takes up the exact % dvd that they need. (This is actually how I encode, in the rare situations when I need to fit to a size.) If it's not that important, just accept 1/3 as your standard and buy a couple more discs.

[elguaxo]

Where can I find that "final ratefactor"?

Look at MeGUI's log after the 1st pass. For example:

Log for job job1-1
...
x264 [info]: mb B I16..4: 1.3% 0.0% 0.0% B16..8: 23.9% 0.0% 0.0% direct:20.5% skip:54.4%
x264 [info]: final ratefactor: 20.65
x264 [info]: direct mvs spatial:99.9% temporal:0.1%
...

[bira]

Log for job job1-1
...
x264 [info]: mb B I16..4: 1.3% 0.0% 0.0% B16..8: 23.9% 0.0% 0.0% direct:20.5% skip:54.4%
x264 [info]: final ratefactor: 20.65
x264 [info]: direct mvs spatial:99.9% temporal:0.1%
...

No signs of "final ratefactor". Any special setting needed?

Sample of log file:

Log for job job3-1

avis [info]: 704x352 @ 23.98 fps (195452 frames)
x264 [warning]: VBV maxrate specified, but no bufsize.
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
x264 [info]: slice I:1317 Avg QP:15.79 size: 24970
x264 [info]: slice P:69709 Avg QP:17.93 size: 9179
x264 [info]: slice B:124426 Avg QP:19.73 size: 2943
x264 [info]: mb I I16..4: 25.9% 0.0% 74.1%
x264 [info]: mb P I16..4: 20.5% 0.0% 0.0% P16..4: 70.1% 0.0% 0.0% 0.0% 0.0% skip: 9.4%
x264 [info]: mb B I16..4: 1.5% 0.0% 0.0% B16..8: 30.1% 0.0% 0.0% direct:28.6% skip:39.8%
x264 [info]: direct mvs spatial:99.9% temporal:0.1%
x264 [info]: SSIM Mean Y:0.9804116
x264 [info]: kb/s:1019.5

encoded 195452 frames, 36.39 fps, 1020.62 kb/s

[elguaxo]

I had the same problem, but I can't reproduce it anymore, so I edited my post as you can see. It happened to me once, and then I got the final ratefactor again...

[bira]

What do you do if three movies come out to 1/4 DVD, and then the fourth needs 1/3?

If three come out to 1/4 DVD and the fourth needs 1/3 then there's not much I can do. I just can't squeeze a 1/3 dvd movie to 1/4 of a dvd. That would be too much. I'll have just to wait for a 1/4 encode to burn a dvd.

If it's not that important, just accept 1/3 as your standard and buy a couple more discs.

That's the easy answer.

I don't like wasting dvd, cd or hd space. That's like going back in time when I used to encode single pass DIVX 3.11! I remember I would just set a bitrate and wait. I always ended up with encodes spanned to several CD's. Some taking a few bytes of a whole cd. That was sad.

Then the existance of Gordian Knot came to my knowledge. That was heaven.

[xyloy]

Some taking a few bytes of a whole cd. That was sad.
For this kind of problem, overburning may be a work around sometimes(up to 720 MB). ;)
(but it doesn't seem to work at all with DVDs! <_< )


If I understand correctly, you want both constant quality, and appropriate filesize on limited space.

I have not some comp. check in my pocket, but to achieve this goal I use OpenOffice.org's Calc(equivalent of MS' Excel).
Example:

X | A | B | C
1 |Name | Length(seconds) | Size(KB)
|---------------------------------------
2 |1st movie | 2039 | "=((B1/$B$4)*4484)*1024"
|---------------------------------------
3 |2nd movie | 2500 | "=((B2/$B$4)*4484)*1024"
|---------------------------------------
4 |overall |="SUM(B1:B2)" | "=SUM(C1:C2)"
(sorry, I can't fix the third column! :devil: )
B1 gives 4539 seconds
C1 gives 2062636 KB (2014 MB)
C2 gives 2528979 KB (2469 MB)
C4's result is 4591616 KB (4484 MB)

4484 MB is the size of a single layered and single sided DVD-R with 4 MB less, because the .MKV files are VBR-encoded , so are a litte over/under-sized than wanted. Single sided/layered DVD-R is the usual media I use. It can be modified to your preference, of course.

This way, the several movies have an appropriate filesize on limited space. If the movies have different FPS, I do numbers of seconds * FPS, and use the number of frames instead.

The use of "$" "tells" the program to always use this line, so I can add or delete lines/movies without the formula giving wrong results.

with x264's CLI used in High Profile two-pass VBR mode, and two HE-AAC v2 audio streams at 48 kbps, I can fit five 16:9 movies with a final ratefactor of ~22, if I crop correctly and resize enoughly(not too much resize, so no more than five ~2 hours length movies).
They are all ~2 hours length, but +/- 10 minutes of difference raise/lower the average bitrate quit much on limited space. The difference is bigger when the FPS is different(25 FPS progressive versus 29,976 progressive(*), the last time I did this).
Otherwise I wouldn't mind, and OOo's Calc software would have rotten on my hard disk. ^^

Here are two screenshots(the video has a resolution of 528*288 pixels) so you can judge the quality yourself:
1st screenshot (http://membres.lycos.fr/xcomcmdr/GoodScreenshotx264.png)
2nd screenshot (http://membres.lycos.fr/xcomcmdr/GoodScreenshotx264(2).png)

Of course, the number of movies you want to fit has to be modified to suit your encoding methode(perhaps with AAC 256 kbps 5.1 and the video cropped with a Lanczos resize to the nearest mod16 resolution wich has a little AR error percent ? I don't know your preferences in that matter ;) ) and your personnal "output resolution and quality versus filesize" experience.

The most suited way for constant quality is always --crf though. ;)

Hope this will help you a bit.

(*)It was Terminator 2 extreme edition, a 29,976 NTSC source with progressive frames all the way(thanks THX).

[bira]

Thanks xyloy, but I'm still missing a compressibility check.

[SpAwN_gUy]

Compressibility check should be done with the same settings that will be used for the final encode, at least for x264 (that's due to the many encoder options that can radically change the compressibility). That's why it is so hard to implement. It will require a complete rewrite of the app workflow.
Yeah, i would like to have that too.. since i'm GKnot User..

/OFFTOP: i think i'll ask few other questions some where Around.. about MeGUI and It's workFlow ;) ...

[Brother John]

So far the GKnot compcheck has always been a quite reliable quality indicator. I even tested it a few times when the difference to BPF was huge. And always the result was as in this scenario.

- XviD encoding. Other codecs might be different.
- BPF says 0.35 -> Perfectly high enough.
- Compcheck says 40 % -> Hardly perfect. Better change settings.
- Two encodings: a) Ignore compcheck. b) Reduce audio size or increase final size to get at least 60%. Video settings same for both.

In every case a) came out significantly worse than b), and usually with clearly visible artifacts. So in the end experience tells me: The GK compcheck is far from being placebo. It might have its flaws and limitations and maybe it’s a ridiculous thing from a technical point of view. But it works! So why abandon the concept?

P.S.: This is not a feature request for MeGUI as I mainly use GK and cli encoding nowadays.

P.S. 2: As an alternative I was thinking about a SSIM based approach for some time, but stuffed the idea. For once a SSIM compcheck would take a lot more time than the GK compcheck and on top of that would probably mess up the codec’s rate control completely giving useless results.

[xyloy]

- BPF says 0.35 -> Perfectly high enough.
It starts to be "perfect" for a SD XviD DVD-rip at 0.3 BPF.

But outside SD DVD ripping, only wizards can know for sure what BPF values will give good or bad quality.
It means nothing anymore. A XviD HD trailer can have 0.100 BPF and show no blocks nor ringing at all.

[Sharktooth]

Still BPF is not accurate at all.

[Brother John]

I'd even go a step further: As a means to estimate visual quality BPF is useless. Unfortunately visual quality is exactly what we want to estimate.

To get back to the original question:
without the compressibility check, how do you come up with the bitrate or file size needed?
a) I don't. I need the compcheck to make a judgment.
b) I know the movie well and therefore from lots of experience can estimate well enough how demanding it is in terms of BPF.

[bira]

b) I know the movie well and therefore from lots of experience can estimate well enough how demanding it is in terms of BPF.

Lenght of the movie, amount of high motion scenes, frame dimensions, level of detail, amount of noise, etc.

Those are too many variables for many to know well.

Actually, by the time one aquires enough experience another codec or encoding settings will come up, and then one's experience will be almost useless.

[xyloy]

Still BPF is not accurate at all
I'm glad we agree ;-)