How to do manual compressibility check with x264?

[chros]

I have read the pros/cons for comp check with x264.
I want to comp check trailers (max 3:00 min).

So I must do with the same setting that the encode will run and I must comp check the whole file and I'll need "Constant Quality" encoding. (I'm using MeGui, so I can see the command line ...)

So the question is: which way must I do the comp check to know what crf value must I use?

Thanks

[Sharktooth]

if you're going to use crf for encoding you dont need a comp check.

[chros]

OK. So I have misunderstood something...
So what should I do if I will go to a 2 pass encoding?
I guess first run only the first pass, then I shoud analyze this file. But what program ?

[Sharktooth]

Comp test can be done in different ways but there is a method that do not require any external program.
Simply do a first pass at your desired quality (CRF 18 is equivalent to xvid Q2) and take note of the filesize (you can do it also on a part of the movie with somewhat a bit less accuracy... you shuold scale up the resulting file size depending on how much of the movie you encoded. Example: using 10%, you should multiply the resulting comptest filesize by 10 to obtain 100%).
That is the filesize at your desired quality, lets say 1Gb.
If you want your output (with the same codec settings) to be 700Mb (to fit a CD) your finaly quality would be reduced from 100% (1GB) to 70% (700Mb).

[dalihrp]

Use Staxrip.

I have read the pros/cons for comp check with x264.
I want to comp check trailers (max 3:00 min).

So I must do with the same setting that the encode will run and I must comp check the whole file and I'll need "Constant Quality" encoding. (I'm using MeGui, so I can see the command line ...)

So the question is: which way must I do the comp check to know what crf value must I use?

Thanks

[chros]

@dalihrp: I have tried it before you mentioned, but I can't feed my avisynth script in it.

So I have spent 2 days reading and trying what Sharktooth suggested (thanks). As it turned out I didn't understand the concept. Summerizing:

Q1: Why do you want to do a compcheck?
A1: You want to decide that how much "quality" the final movie should be (depending on the codec capabilities). The quality means that how much bit gets the frames in a second (since we provide the bitrate in kbps).

Q2: Does it give reliable value?
A2: Well, it depends on... @ DivX and XviD codecs it's totally reliable.
But in the case of x264 the opinions vary.
1. If you go for target file size, where the frame size will change, some say it doesn't give reliable result. Because of the complexity of the encoder. But in this case, it can give you a rough result.
2. If you go for target frame size, it's almost correct. (Q: Why just almost? (http://forum.doom9.org/showthread.php?t=115801))

Q3: How is compcheck working?
A3: Basically, you do all the encoding with the same codec settings (except for the bitrate).
- At fisrt, you do a quality encode (@xvid: q2, @x264: q18), then you get a final size. This means that's the best what the codec could (at the certain settings).
- then you can calculate. (see below)

Q4: When and how should I use manual compcheck?
A4:
1. if you go for a target filesize, eg. 1CD or 2CD dvdrip (normally it means 2pass encoding)
In this case we have fixed lenght (duration), and fixed filesize (= fix bitrate), so what can change is the frame size (image size).
1. do a quality encode (@x264: -cqrf 18): in megui select the profile of the final 2pass encode (eg. "HQ Slower" or "HQ Insane"), then modify it: at the main tab select "Const. Quality" for the "Mode"
2. see the duration in seconds (d) and the filesize in Byte (fs), then calculate the bitrate from it: fs/1024*8/d -> qb
3. calculate the pixels from the framesize: eg. 640*272=174080 -> p
4. then see what framerate (fr) has the movie (e. 23.976) and calculate this: qb/(p*fr)*1024 -> ct (this is the "Compressibility Test" value in GK (at the right) and "compressibility" in StaxRip)
5. see the calculator-suggested bitrate (cb) and calculate the "Average Bits/(Pixel*Farme)" (like in GK): cb/(p*fr)*1024 -> ac
6. calculate the "Quality" in percent (beside the Load button in GK): ac/ct -> q
now you see how much percent the current encode can be, eg. 0.32 . There's an agreament, that quality @divx-xvid:0.6 and @x264:0.5 is acceptable, but if it's more it's good. There is useless to encode over 1.0 value.
7. if your q value is too low, then select a smaller framesize, then do this calculation again, till you get the desired value.
2. if you go for target framesize: eg. you want to encode at the original framesize after cropping (like in anamorphic overcrop mode) and you want to decide that how much CD do you need, or for a trailer you don't want to waste diskspace needlessly (here isn't such a thing like CD count) (normally it means 2pass encoding)
In this case we have fixed lenght (duration), and fixed frame size, so what can change is the file size (= bitrate). It's more simpler than the previous case.
1. do a quality encode (@x264: -cqrf 18): in megui select the profile of the final 2pass encode (eg. "HQ Slower" or "HQ Insane"), then modify it: at the main tab select "Const. Quality" for the "Mode"
2. see the obtained filesize (fs) in MB
3. target a quality (q) in percent: eg. 70% (0.7) and see how much MB do you need for this: fs*q -> qs
4. see the duration in seconds (d) and calculate the bitrate from it: qs/1024*8/d -> qb
3. Is there any case where compcheck makes sense ?

Q5: How much percent of the movie should I use? 'cause do the whole movie is taking too long.
A5: The optimal case is when you compcheck the whole movie. The bigger percent of the movie you use the bigger the chance that the result of the compcheck is reliable.
If you don't have so much time for it you can use any percent you want. All you need to do is to insert a line et the end of the avs script, like this:
# in general: SelectRangeEvery(every,length)
# So this example means 5% of the movie: 14 frames per pieces
# Selects the frames 0 to 13, 280 to 293, 560 to 573, etc
SelectRangeEvery(280,14)
Suppose you want compcheck the 30% (0.3) (cp) of the movie.
1. see the total frames of the movie (tf) (eg. 3322)
2. calculate the desired "every" value (ev): tf/(tf*cp/length) -> ev (if cp=0.3 and tf=3322 and if length=14 then ev=46)

@codec gurus: Is the Q2/2. and Q4/1./1. right ?

OK. That's it for now.
Maybe I'll create a spreadsheet for this calculation in Excel ... :)

Any suggestion, correction is welcomed.

[bira]

I should try that since I'm missing a compressibility check in MeGui.

So, when you say:

"1. do a quality encode (@x264: -cqrf 18): in megui select a profile, eg. "HQ Slower" or "HQ Insane" then modify it: at the main tab select "Const. Quality" for the "Mode""

I should use the profile I'll choose for the final encode, right?

[Sharktooth]

@codec gurus: Is the Q2/2. and Q4/1./1. right ?
seems good.
I should use the profile I'll choose for the final encode, right?
yeah, but change the encoding mode to CRF18.

[chros]

seems good.
Thanks
yeah, but change the encoding mode to CRF18.
@bira: good observation, I'll edit the corresponding lines.

Can be anything the value of the length variable in the SelectRangeEvery(every,length) function ?
('cause I saw in GK, that "it must be 14")

[foxyshadis]

The larger it is, the more accurate (since you're processing more of the video), and the longer it will take.

[chros]

The larger it is, the more accurate (since you're processing more of the video), and the longer it will take.
Thanks for this comment, I realized this.
But then why GK has limeted this parameter?

[Sharktooth]

Dunno, ask the GK developers..
Maybe they tested with a lot of sources and determined that value is a good tradeoff.

[jonny]

SelectRangeEvery(every,length)

smaller "length" gives better results
this is because there are less jumps between encoded snips (you take a better distribution of frames from the movie)

14 is derived analyzing accurancy vs time required for the test
using a lower value progressively slow down the decoding of the source video because the decoder, internally, start always from keyframes
(the value is derived assuming mpeg2 as source video)

[venky]

Hi Chros,

I went through your methodology of calculating the compcheck for x264 with interest. The main thing I have missed in MeGui is the lack of the compcheck parameter which was present in GK (which was a sort of a security blanket for me with its 40% level, sniggering from doom9 and other established members notwithstanding).

In my situation I am converting a DVD into x264 using the ABR methodology using an average bit rate of ~700 kbps (I am fixing the file size). I have carried out the conversion thrice, once each with the HQ-Insane, HQ-Slowest and HQ-Slower profiles keeping everything else the same.. All three videos have nearly the same size just differing by some 10s of KBs.

Using your methodology all three encodes give me the same quality (because the similar file size). I have the following questions:

1. Then how can I use compcheck to distinguish between the three profiles?

2. Does this I mean I cannot use compcheck to distinguish between the three profiles?

3. Is compcheck useful only to compare encodes with different framseizes?

Thanks,
Venky

[chros]

I have carried out the conversion thrice, once each with the HQ-Insane, HQ-Slowest and HQ-Slower profiles keeping everything else the same..
...
Using your methodology all three encodes give me the same quality (because the similar file size).

You've misunderstood the whole concept (like I had :) ):

"A3: Basically, you do all the encoding with the same codec settings (except for the bitrate)."

1. Then how can I use compcheck to distinguish between the three profiles?
That's the problem ... :) You should encode with 1 profile !!!
Of course the different profiles results different quality:
HQ Insane > HQ Slowest > HQ Slower > HQ Slow
So HQ Insane is the best, but -as the name indicates- it's unnecessary. I'm using HQ slower ...

2. Does this I mean I cannot use compcheck to distinguish between the three profiles?
Yes, that's correct.

3. Is compcheck useful only to compare encodes with different framseizes?
Yes, or you go for a target frame size (the file-size will vary ...)
See A4 !


So the whole method:
1. First, you have to decide which profile you do use : (I'm using HQ Slower)

2. Then you should decide, which way to go (See A4):
- go for a target frame size (eg. as the original, that's means perfect quality)
- go for a target file size (eg: 700MB, 1CD, but nowdays the size is not really important)

3. suppose you choose the 1st: target franmesize: eg. as the original (after croppping 720*xxx)
then you encode with HQ Slower CRF 18 the whole movie (if you have enough time), so you'll get a filesize

4. you want to get the bitrate for the previous encodes:
- see the x264 log, or
- use a bitrate calculator (eg: in megui)

5. final step: you 2pass-encode the movie with the same profile (!!!, HQ Slower) and with the resulted bitrate


I hope, it clears things up ...

Greets
Chros

[SpAwN_gUy]

the funny thing is.. that x264farm.. hm.. encodes first pass as written (default --crf 18 (but you can override it..)) and reads the bitrate.. and then you can specify percentaje of bitrate of first pass for second pass bitrate :) ...