x264: guess crf for given bitrate

[WasF]

I saw there was an old thread about this, but contents seem irrelevant nowadays.

So, here I'm asking: any ideas about guessing the "right" rate factor to use for a given average bitrate target ?

The best solution I've come up with is to encode 20 ou 30% of the source (with AVS's SelectRangeEvery() function) with --ratetol inf and use the "final rate factor" indication in x264's output.
This is not very precise, but, with some experience, and a graph, I can get it to stay within a +/-100kb/s window of the target, which is not so bad, unless your desired bitrate is very low.

To help choosing the suitable crf, I plot x264 logs like so (with GnuPlot) :

http://desmond.imageshack.us/Himg580/scaled.php?server=580&filename=videocrf281525kbs1k.png&res=landing

In this particular case, you can see the bitrate stayed way above the 1500kb/s target most of the time, so I would use the frf + 1 or 2 notches, a notch being 10 base points (0.1 point).
(I plot the 1000 first frames apart, because the bitrate is usually insanely high at the beginning, which stretches the y-range too much for the rest of the frames - makes reading the graph harder).

[Groucho2004]

This (http://komisar.gin.by/CRF2Size.html) might be of interest to you.

[kypec]

I still wonder why people waste time on ideas like this... If you want to hit specific bitrate then just use 2-pass method. If you want to hit specific quality then just use CRF=value obtained from several tests which suits you best.
Right now I'm encoding my whole DVD box-set of Friends, 10 seasons * 4 discs each = 40 DVDs. I've encoded the very first episode with CRF=20 to see at what average bitrate it will actually end up. The result was something ~1474 kbps.
Knowing that I've switched to 2-pass mode with target bitrate=1500 kbps and currently after having encoded entire season 01 I can see that there are variances in final rate factor (logged after the first pass) no more than ~1.5 CRF which is perfectly acceptable by me.

[WasF]

@Groucho2004: thx, I'll try that ! Any links to explain what that tool does ? I skimmed through the code ans saw SSIM and Newton method for solving equations..

@kypec: this is about getting the best of two worlds. ABR produces lesser quality, and 2-pass takes too much time. So, if I can get quality in one pass (CRF), that'd be sweet. I care much about quality and my settings rarely go above 5fps (-me esa -merange 24 --rc-lookahead 60 --8x8dct --b-adapt 2 --subme 10 --trellis 2 --ref 8, try that on your preferred CPU !). So, missing the target is very expensive to me (think 14h encode with a too big file as result !).
Besides, I'm doing the same thing as you right now (encoding an entire TV season), and believe you me, the frf I get varied between 25 and 31 (for very like-looking episodes) !

[Groucho2004]

Any links to explain what that tool does ?I posted the link mainly for the formula which is used to iterate towards the required CRF value:
NextCRF = (-6*(NeedBitrate-GotBitrate)/GotBitrate)+GotCRF

2-pass takes too much time.
Hardly, if you consider how much time you'll need to calculate the desired CRF value.

The important thing to know is that there is no linear relationship between CRF and bitrate.

[WasF]

NextCRF ?! You mean doing multiple encodes ? That's way too expensive !
In my (definitely non-linear) method, crf is estimated with just 20 to 30% of the source, and the resulting frf is adjusted based on the "looks" of the bitrate curve.

[Groucho2004]

NextCRF ?! You mean doing multiple encodes ? That's way too expensive !
In my (definitely non-linear) method, crf is estimated with just 20 to 30% of the source, and the resulting frf is adjusted based on the "looks" of the bitrate curve.

Multiple encodes - yes, but with only 5-10% of the source. 20-30% is not necessary.

[WasF]

O.
That sounds interesting. (if it converges in 3/4 cycles or less).
I will test that on episode 2 of my run. Got frf=26.23 on the 30% sample (1403kbs), took 3h20. My method says 26.15 will get me close to desired 1500kbs.
I'll see what the formula yields after 3 encodes of a 10% sample, and how much time it takes, and then carry out the full encode to see which method was closer to the "truth".

[Groucho2004]

O.
That sounds interesting. (if it converges in 3/4 cycles or less).
I will test that on episode 2 of my run. Got frf=26.23 on the 30% sample (1403kbs), took 3h20. My method says 26.15 will get me close to desired 1500kbs.
I'll see what the formula yields after 3 encodes of a 10% sample, and how much time it takes, and then carry out the full encode to see which method was closer to the "truth".

Once you've completed all the tests, just do a normal 2 pass encode and compare the times.

[nm]

Two estimation passes of 1-2 % coverage should be enough for most sources.

See this discussion: http://forum.doom9.org/showthread.php?t=116773

Selected posts for the lazy :)
http://forum.doom9.org/showthread.php?p=887712#post887712
http://forum.doom9.org/showthread.php?p=894725#post894725

[WasF]

@Groucho2004: The full 26.15 crf encode is actually 4h away from conclusion now. I'll do the full 2pass encode tonight (giving it 100% CPU), and do the 3 cycles on a 200MHz slower PC (E8400 instead of E8600, same RAM) to get the CFR numbers at least. All 3 cycles should have same duration, given the very small variation of the CRF between them, so just 1 cycle tomorrow on the faster PC should be enough to know how long 3 cycles would take.
That way, we can hopefully have the results tomorrow before noon.

I really think the 2-pass deal won't even be finished by then !

Edit: oops, I will actually start from CRF=23.00 for the formula method, so CRF should vary quiet a bit (if it is to move towards 26). So I'll have to do all encodes on the same PC for duration comparison. Takes more time.

[WasF]

@nm: I read that post in detail (the thread I refer to in my first post). I dunno what theory asserts the 1/2 percentage validity, but my own tests say it's nuts !
I wish ! My encodes are painfully long.

On the other hand, 20% (30% on shorter sources) is almost guaranteed money.

[nm]

@nm: I read that post in detail (the thread I refer to in my first post). I dunno what theory asserts the 1/2 percentage validity, but my own tests say it's nuts !
I wish ! My encodes are painfully long.

On the other hand, 20% (30% on shorter sources) is almost guaranteed money.

That's because you were only encoding one large estimation pass and not using proper estimation for the CRF value. Multiple small passes with mathematically sound picking of CRF values to try is much better.

The DABR thread is still relevant when you apply the technique properly!

[WasF]

nm: I confess I didn't completely grasp the logic of that method, but I remember doing some 1% tests and screaming with horror.
Now that you put it that way, maybe I was not doing it properly.

5% samples are just fine if Komisar's formula is sound. Otherwise I might revisit DABR with more coffee this time.
(Still, if you can depict the DABR method in clear terms for me to follow, I will definitely add it to my current test)

[benwaggoner]

Right now I'm encoding my whole DVD box-set of Friends, 10 seasons * 4 discs each = 40 DVDs. I've encoded the very first episode with CRF=20 to see at what average bitrate it will actually end up. The result was something ~1474 kbps.
Knowing that I've switched to 2-pass mode with target bitrate=1500 kbps and currently after having encoded entire season 01 I can see that there are variances in final rate factor (logged after the first pass) no more than ~1.5 CRF which is perfectly acceptable by me.
Why not keep using CRF, and let episodes vary in size based on difficulty to encode? In this case, I imagine most users are concerned about hitting their target quality and total storage size. If some episodes come out bigger, well, they needed those extra bits to look good, but it'll average out over 100+ episodes.

[kypec]

Why not keep using CRF, and let episodes vary in size based on difficulty to encode? In this case, I imagine most users are concerned about hitting their target quality and total storage size. If some episodes come out bigger, well, they needed those extra bits to look good, but it'll average out over 100+ episodes.
Because I prefer to know the bitrate/filesize precisely for complete series when total running time is given. Variations in quality resulting from fixed bitrate are too small for me to notice so that's why I chose this path instead of absolutely constant CRF with bitrate varying (perhaps) too much for my liking between episodes.

[WasF]

Ok, change of plan. Here is accumulated data so far:

Options: x264-10bit_32bit.exe --rc-lookahead 60 --bframes 4 --b-adapt 2 --subme 10 --trellis 2 --partitions all --8x8dct
--me esa --merange 24 --threads 4 --no-dct-decimate --direct auto --no-interlaced --level 4.2 --ref 8 --profile high422
-------------------------
Source: TV, nature (60% bright, 20% night, 75% normal/slow, 25% action), 1280x720p, 29.97fps, 75835 frames
AVS: SelectRangeEvery(600,60) # 10%. Assuming 25fps: pick 2.4s at the begining of every 24s slice.
Or SelectRangeEvery(1200,60) # 5%. Assuming 25fps: pick 2.4s at the begining of every 48s slice.
Or SelectRangeEvery(600,180) # 30%. Assuming 25fps: pick 7.2s at the begining of avery 24s slice.
-------------------------
NeedBitrate: 1500
Komisar's formula: stop when GotBitrate in [NeedBitrate-100, NeedBitrate+100], or when reach 4 runs
Run 1 2 3 CRF Seed: 23.00
GotBitrate 2727.18 1818.28 1564.87 Bitrate error = GotBitrate - NeedBitrate
GotCRF 23 25.70 26.75 Reverse rel. Bitrate err = - Bitrate_error / GotBitrate
NextCRF 25.70 26.75 27.00 NextCRF = GotCRF - [ 6 * Reverse_relative_Bitrate_error ]
Duration 1h18 1h16 1h12
-------------------------
Results after full encode:
My method 30%(26.15) Komisar 10%(27.00)
Bitrate 1605 eta 9h30 .. :(
Error 105 ?
Cost 3h24 3h46

First of all: my method yielded maximum error here (~100kbs), usually, it's much closer than that !
Second of all: cost is not a discriminating factor, it takes roughly the same time to do the estimation. Besides, if Komisar gets much closer, I'll be more than happy to spend some extra 20mns with him !
Good news is, two methods disagree completely. That increases the chance of having a clear winner. But it's looking bad for my method :sly:
If Komisar gets this right, I'll do another episode (season finale, very different complexity) to confirm this is not just some kind of coincidence. Than a movie (Avatar) to seal the deal. And test the 5% option too to see if I can get away with it..

[WasF]

OK, final results are here:
Results after full encode:
My method 30%(26.15) Komisar 10%(27.00)
Bitrate 1605 1418
Error +105 -82
Cost 3h24 3h46

That's.. underwhelming :(
Back to square 1. Any *better* ideas ?

[poisondeathray]

Just use 2pass , and or faster settings . Seriously.

You're wasting way more time doing this testing .

Even if your test showed a lower margin of error on this sample , do you think that would apply universally across different sources ?

The amount of time wasted on this surely you could have spend working and paid off for more hdd's and faster computer

[WasF]

@poisondeathray: are you really not getting this ?! I'm not doing the test for 1 particular sample.
Yes I'm looking for something that applies universally, with some acceptable error margin. That's what a "method" is.

[poisondeathray]

@poisondeathray: are you really not getting this ?! I'm not doing the test for 1 particular sample.
Yes I'm looking for something that applies universally, with some acceptable error margin. That's what a "method" is.

I know you're looking for a method or a process

Do you not understand that you cannot extrapolate results with any accuracy by testing a few samples?

(just because it works on this sample, doesn't mean it will on other videos)

Good luck :)

[WasF]

Got it, but you're reading this in a mirror ! You could say that if it actually worked. It didn't. Done. Disqualified.
No need to even test a second sample.

[poisondeathray]

Got it, but you're reading this in a mirror ! You could say that if it actually worked. It didn't. Done. Disqualified.
No need to even test a second sample.

Actually no you couldn't

Do you understand that it's the approach of the testing and method that I am commenting on ?

Even if it worked or didn't work (in terms of low error%), that wouldn't prove or disprove anything because your sample size is too small. You cannot extrapolate the results to make broad based generalizations from this single observation . It's a single data point

Had you 100's of different samples and testing, then you would be closer to generating a valid model. But you're complaing about 1 test taking hours...

In other words, you 're wasting your time :)

[WasF]

Even if it worked or didn't work (in terms of low error%), that wouldn't prove or disprove anything

That's logically false. :) All I'm saying.
The fact that it didn't work for 1 sample (just one), definitely, positively proves something (that it doesn't work in general, so toss it).

Just read that again ;)

[poisondeathray]

That's logically false. All I'm saying.

The fact that it didn't work for 1 sample (just one), definitely, positively proves something (that it doesn't work in general, so toss it).


That statement was referring to your development of a method or approach. In itself, the statement it's logically false, but you're taking the statement out of context

Again, you can't conclude anything about "in general" from 1 observation . If you want to talk about logical fallacies, you commiting one right there

Yes, it proves it for that particular sample ONLY - but you said you're trying to develop a method apply universally (or did you forget you said that ) :)

For example, this might be a spurious data point . What if in 10 other tests the error margin is significantly smaller? What if on another video it's 0.1% ? You need to do many tests and different sample and look the results - then you will get a range of error and a better idea


Good luck

[WasF]

Do you agree with this: to prove that a method is not universally valid, it is enough to prove it invalid just once.
You don't seem to agree with this "principle". If that's the case, than you're forever right (within your own unusual logical system !). And I'm forever wrong ! (I like it here outside :))

[poisondeathray]

Do you agree with this: to prove that a method is not universally valid, it is enough to prove it invalid just once.
You don't seem to agree with this "principle". If that's the case, than you're forever right (within your own unusual logical system !). And I'm forever wrong !

Yes, in terms of method or test design.

No, in terms of observations. Because in scientific testing you can get spurious results. These are usually discarded. You have made 1 observation in whether this method is accurate in predicting or not.

Also your logic is flawed, because you're applying a binary "vaild" or "invalid" measurement, when the results are continous (you have a range of errors)

I'll illustrate. "x264 encoder scores lower SSIM score on 1 test than Mainconcept AVC encoder under 1 set of conditions. Therefore , x264 encoder is universally worse than Mainconcept" . What fallacy is this? :D



But I understand that for your purposes, that even 1 result that is too large from expected result is not acceptable. And that's fine. Let's just leave it at that

Cheers

[WasF]

O, but my logic is definitely binary ! I want : "method A will get me within X kb/s from desired bitrate. Every. Single. Time. No exceptions."
If method A can sometimes get me 500 kb/s off, then ditch it.
Here X would be the tradeoff between precision/cost.

[poisondeathray]

Yes, I understand . I made an edit above regarding deviation that you probably missed . Sorry it's my mistake in misunderstanding you. And I understand that's not acceptable for your purposes, even if on average, that a particular prediction method might yield acceptable results .

So lets say some particular prediction method yielded acceptable results to you, a positive "valid" test - how many other tests would you have to do to confirm or accept it ? Another way of saying this: how many tests would you need to confirm there are no "invalid" tests ?

[WasF]

3 or 4 confirmations on very different sources would be enough. That doesn't guarantee anything, but is enough to sign me up.
I would keep using that method until I wake up some day (literally ! Encoding through the night..) with some horrifying figures.

Here I'm betting we live in a "reasonable world". If it works on TV, movie, slow/action, dark, bright, cartoon (1 sample each), well, I'm willing to take a chance and change my workflow.

We were discussing there other way around, which suffers no such uncertainties.

[Sharc]

You may want to try:

Run 0 (seed):
CRF(0) => Produces {Size(0)}

Run 1:
CRF(1) = CRF(0) - 6 * {TargetSize - Size(0)} / {Size(0)}

and then onwards for subsequent Runs:
CRF(N+1) = CRF(N) - {TargetSize - Size(N)} * {CRF(N) - CRF(N-1)} / {Size(N-1) - Size(N)}

'TargetSize' and 'Size' are in MB

Stop when ABS(TargetSize - Size) becomes < limit
I used 2% sample sizes.

[kypec]

Source: TV, nature (60% bright, 20% night, 75% normal/slow, 25% action), 1280x720p, 29.97fps, 75835 frames
Results after full encode:
Cost 3h24 3h46
Are you sure you can't get similar performance-wise results by using straight 2-pass with fast first-pass options enabled of course? I mean, encoding almost 76000 frames at ~6 fps on average can't be that hard to achieve, or is it? Are you using some heavy Avisynth processing so that you want to avoid 2-pass x264 encoding or necessity for intermediate lossless file? If not then I really can't comprehend your point of all this CRF convergence method, sorry. :confused:

[WasF]

@Sharc: thx for the method. This is Newton's method of solving an equation of CRF as a function of size (instead of bitrate as in Komisar's method).
All of these methods assume CRF is a regular enough function of bitrate or size (or whatever parameter you're willing to use) to even have a derivative against that parameter ! Are there any writings about this to your knowledge ?
Since you use it, can you give some info about it's reliability/speed of convergence (how big an N is needed in your experience ?).

@kypec: your question is making more and more sense.. CRF/size seem too wild to guess through any fast-enough converging algorithm.
In the case of light AVS/pre-processing, you're obviously right. Otherwise, it could be useful to have a "good enough" method to just have an option out of a very lengthy/resource-hungry 2-pass encode.

[Didée]

Those convergence methods are generally okay (or even pretty good). However, the pitfall is the small sample size that's typically used. When oly small snipplets - like 2% or 4 or 5% - are used, then in the big number of plenty-of-sources, the final bitrate will be good. But, for single individual sources, there will always be cases where the sniplets fall particularly "lucky" or "unlucky" onto the full file. Hence, when using such small snip sizes, you cannot guarantee that the final bitrate will always be met. Perhaps the sampling got a little too much of action scenes, perhaps it got a little too much of slow scenes - there is no guarantee.

Statistic is for big numbers, but not for individuals. Statistic says that on average, you must play lottery for 2000 years in order to hit the jackpot. However, every week or month someone wins. But nobody ever has played for 2000 years.

[WasF]

Didée: agreed.
Even using "my" method (30% sample), the indication has to be adjusted (I use the "shape" of the bitrate curve to make an "educated" guess).
This proves that 2 and 5 and 10% are actually hopeless. But with the convergence methods, you can't work with 30%: the convergence would be too slow (as in useless slow). So there you have it.
Moreover, in my experience so far, the hit-and-miss rate is pretty bad (last encode was off by 200 kb/s despite an excellent error of 0.2% on the 10% sample !). Given the steep cost to a miss, that makes those rebel "individuals" real nasty. So a method has to be pretty darn reliable.

That doesn't mean a method doesn't exist, just that it won't likely be a convergence one.

1 method exists if feasible: a crf-first-pass: a pass that's used to just collect enough info to make an estimation of crf. If it can be done faster than a normal first pass, then there it is. Maybe some x264 dev will read this !

[Groucho2004]

If it can be done faster than a normal first pass, then there it is. Maybe some x264 dev will read this !
Have you actually tried doing a normal 2-pass encode? Since the default settings for the first pass have been changed in 2009, the first pass can be 10 times faster than the second pass, depending on the settings.

[WasF]

O.
I didn't realize that ! 10 times faster, really ? That means 1-2h in my case, which is even faster that encoding 30%.
That would be great !

What options should I use for first pass ? Just --pass 1 ? Is it harmless to omit all other options ?

[Groucho2004]

O.
I didn't realize that ! 10 times faster, really ?
As I wrote, it depends on your settings (for the second pass).

What options should I use for first pass ? Just --pass 1 ? Is it harmless to omit all other options ?
You would usually just pick a preset and add "pass 1" / "pass 2".

More info on various settings here (http://mewiki.project357.com/wiki/X264_Settings).

[WasF]

O :(
A preset is just a bunch of options. I have mine.
With my options, a first pass is just as slow as a second one.

[Groucho2004]

O :(
A preset is just a bunch of options. I have mine.
With my options, a first pass is just as slow as a second one.
How about posting your settings for the first and second pass?

[WasF]

Done already : --rc-lookahead 60 --bframes 4 --b-adapt 2 --subme 10 --trellis 2 --partitions all --8x8dct --me esa --merange 24 --threads 4 --no-dct-decimate --direct auto --no-interlaced --level 4.2 --ref 8 --profile high10 (--fps if needed)

That's my only "preset": I don't compromise on quality. For me an encode is maximum quality, or isn't (use CUDA !).
With this, even a first pass is slow.

[Groucho2004]

Done already : --rc-lookahead 60 --bframes 4 --b-adapt 2 --subme 10 --trellis 2 --partitions all --8x8dct --me esa --merange 24 --threads 4 --no-dct-decimate --direct auto --no-interlaced --level 4.2 --ref 8 --profile high10 (--fps if needed)

That's my only "preset": I don't compromise on quality. For me an encode is maximum quality, or isn't (use CUDA !).
With this, even a first pass is slow.
Where did you put the "--pass 1"? If you put it at the beginning, your "special" settings will possibly override the fast first pass settings, not sure.

[WasF]

Sorry, here is my script:

@echo off

REM ~ set fps=29.97
set brate=1500

set x264=C:\MySofts\Multimedia\x264\x264-10bit.exe
REM ~ set x264=C:\MySofts\Multimedia\Codecs_Filters\x264\x264-8bit.exe

set options=--stats stats1.stats --bitrate %brate% --rc-lookahead 60 --bframes 4 --b-adapt 2 --subme 10 --trellis 2 --partitions all --8x8dct --me esa --merange 24 --threads 4 --no-dct-decimate --direct auto --no-interlaced --level 4.2 --ref 8 --profile high10
REM ~ --fps %fps%

echo.
%x264% --help | findstr "core"
echo.

time /t

:encode
set fn=video_brate%brate%

logger -append -priority belownormal -log "%fn%_1stPass.log" : "%x264%" --pass 1 %options% --output NUL 2beEncoded.avs
time /t
logger -append -priority belownormal -log "%fn%_2ndPass.log" : "%x264%" --pass 2 %options% --output "%fn%.mp4" 2beEncoded.avs

echo.
time /t

pause

--pass comes before the options.

[Groucho2004]

Sorry, here is my script:

@echo off

REM ~ set fps=29.97
set brate=1500

set x264=C:\MySofts\Multimedia\x264\x264-10bit.exe
REM ~ set x264=C:\MySofts\Multimedia\Codecs_Filters\x264\x264-8bit.exe

set options=--stats stats1.stats --bitrate %brate% --rc-lookahead 60 --bframes 4 --b-adapt 2 --subme 10 --trellis 2 --partitions all --8x8dct --me esa --merange 24 --threads 4 --no-dct-decimate --direct auto --no-interlaced --level 4.2 --ref 8 --profile high10
REM ~ --fps %fps%

echo.
%x264% --help | findstr "core"
echo.

time /t

:encode
set fn=video_crf%brate%

logger -append -priority belownormal -log "%fn%_1stPass.log" : "%x264%" --pass 1 %options% --output NUL 2beEncoded.avs
time /t
logger -append -priority belownormal -log "%fn%_2ndPass.log" : "%x264%" --pass 2 %options% --output "%fn%.mp4" 2beEncoded.avs

echo.
time /t

pause

--pass comes before the options.

Try if the first pass is faster if you remove these:
"--rc-lookahead 60 --bframes 4 --b-adapt 2 --subme 10 --trellis 2 --partitions all --8x8dct --me esa --merange 24 --ref 8"

[WasF]

No need, of course 1st pass will be faster without those extreme settings.
The question is : will the final result (of the second pass) be affected ?
Said differently: is the 1st pass .stats file better (more precise) with the full options, or not ?

[Groucho2004]

will the final result (of the second pass) be affected ?
The difference is marginal, as far as I know. There are several discussions about this on this forum, I suggest you use the search.

[nm]

Those convergence methods are generally okay (or even pretty good). However, the pitfall is the small sample size that's typically used. When oly small snipplets - like 2% or 4 or 5% - are used, then in the big number of plenty-of-sources, the final bitrate will be good. But, for single individual sources, there will always be cases where the sniplets fall particularly "lucky" or "unlucky" onto the full file. Hence, when using such small snip sizes, you cannot guarantee that the final bitrate will always be met. Perhaps the sampling got a little too much of action scenes, perhaps it got a little too much of slow scenes - there is no guarantee.

Well said. Sampling is definitely the problematic part if the final file size needs to be spot-on.

I just encoded 5 very different videos including hard cases with compressible animation and live action spliced together.
I tried this method:

1. ABR encode at the target bitrate and SelectRangeEvery(1000,20) (2% sample)
2. CRF encode using the final ratefactor from the first pass and SelectRangeEvery(2000,300) (15 % sample)
3. Final encode using a CRF value estimated from the previous encode (crf = lastcrf + 6 * (lastbitrate - targetbitrate)/lastbitrate)

Result: In 4 of 5 cases the target bitrate was missed by less than 5 %, but there was one difficult case with an error of 11%.

Thoughts:
There's no need to run more than one sampling CRF pass after the first ABR pass. Additional passes don't improve accuracy at all when using the same sample size.
The higher the sample size in the CRF sampling pass, the higher the accuracy. If 10% error is tolerable in general, you'll probably only need 5% sample size. That's the case where this method makes most sense: targeting an average bitrate, but only approximately.
If you need the bitrate to stay within one or two percents of the target every time, just use 2-pass encoding with a fast first pass.

[Sharc]

@Sharc: thx for the method. This is Newton's method of solving an equation of CRF as a function of size (instead of bitrate as in Komisar's method).
All of these methods assume CRF is a regular enough function of bitrate or size (or whatever parameter you're willing to use) to even have a derivative against that parameter ! Are there any writings about this to your knowledge ?
Since you use it, can you give some info about it's reliability/speed of convergence (how big an N is needed in your experience ?).
Agree with your comments; and no, no writings about this to my knowledge.
Speed of convergence: N = 2....6 for hitting the target size within +/-5% for "normal" movie sources, using a distributed 1% ... 2% snippet.
Still faster than 2-pass even with fast 1st pass, but for reasons mentioned by Didée prediction may occasionally fail e.g. when the majority of the snippets happen to hit dark and quiet scenes - maybe in 1 out of 10 cases.

[WasF]

@Sharc: thanks for the info. That's what I thought (cf. nm's post).
Even with 90% success rate, it's not good enough (for me): a miss is really costly (think 25h of 100% CPU lost). Having to potentially do up to 6 cycles doesn't help..

@Groucho2004: couldn't find a thread about pass1/pass2 option differences (a search with "x264" and "pass" is basically doomed, all posts about x264 contain those two words !). Google found a hint in MeGUI wiki (--b-adapt useless in second pass). And discussions on videohelp that look more like guesses. That's all.

[nm]

@Groucho2004: couldn't find a thread about pass1/pass2 option differences (a search with "x264" and "pass" is basically doomed, all posts about x264 contain those two words !).

You should search for x264 and "fast first pass" or something along those lines.

Of all x264 presets, only "placebo" enables --slow-firstpass (or in other words, disables fast first pass). That alone suggests that using a slow first pass is basically useless in any real world encoding scenario.

[WasF]

OK, "fast first pass" helped ! But, still, no general info about how 1st/2nd pass options relate/should be balanced.

"Fast first pass" is --pass 1, precisely defined in x264's help as: force no-8x8dct, partitions none, me dia, ref 1, subme 1:user,else:2, trellis 0, fast-pskip
Unless you set slow-firstpass/perest placebo.

In my case, that leaves (besides level/profile) : rc-lookahead 60, b-adapt 2, merange 24, no-dct-decimate, direct auto
So, these are the potential offenders. That narrows it down, but I'm still not able to tell if disabling those in first pass would affect final result much.
I guess I'm only left with trying it out and comparing the stats files numbers with and without those options. Could be inconclusive, though, if difference is very small..

[Sharc]

This reminds me of another proposal (made by Dark Shikari?) how to speed up a 2-pass encode. I am not sure though if it has ever been realized as it requires some deeper knowledge:
Run the first pass on a reduced resolution of the video, e.g. 1280x720 instead of the original 1920x1080. Then modify the stats file somehow (?) before doing pass 2 with the original resolution.

I try to find the thread / post and report back when I find it.....

[sneaker_ger]

Has been realized already. Too lazy to search for the thread.

[nm]

In my case, that leaves (besides level/profile) : rc-lookahead 60, b-adapt 2, merange 24, no-dct-decimate, direct auto
So, these are the potential offenders. That narrows it down, but I'm still not able to tell if disabling those in first pass would affect final result much.

Don't change other settings between passes than what's listed there in x264's help under --slow-firstpass. If you'd like to speed up more, use faster settings in both passes. Placebo-level settings such as --me esa are pretty useless for their cost in CPU time.

[WasF]

Maybe. Without doing a detailed comparison :
--pass 1 + Full options:
[info]: frame I:895 Avg QP:35.52 size: 35034
[info]: frame P:20819 Avg QP:38.94 size: 6666
[info]: frame B:54135 Avg QP:43.47 size: 859
[info]: consecutive B-frames: 3.0% 4.9% 10.2% 68.0% 13.9%

Just --pass 1:
[info]: frame I:909 Avg QP:35.22 size: 35797
[info]: frame P:21879 Avg QP:38.53 size: 6170
[info]: frame B:53061 Avg QP:42.95 size: 927
[info]: consecutive B-frames: 3.3% 5.8% 13.7% 77.2%

Selection of frame types is affected (even though the QP numbers don't change much). I see stats there jumping from 68 to 77%. I'd say the full options do have a bit of an effect (and don't think doing a more detailed analysis would produce poorer decisions). So, yes, final quality would be affected.
How much ? I don't know. It seems like a small effect (the full options produced 1.54% less I frames, which would indicate a slightly wiser bit allocation to come in second pass - and more computations). But the raw numbers are hard to interpret for an algorithm as complex as H.264.

Use your eyes, I guess.

[Groucho2004]

So, yes, final quality would be affected.
How much ? I don't know. It seems like a small effect (the full options produced 1.54% less I frames, which would indicate a slightly wiser bit allocation to come in second pass - and more computations).
You compared two extreme sets of settings, you could just pick something in between for pass 1. For example:
--slow-firstpass --ref 4 --me hex --subme 3 --trellis 0

[Dark Shikari]

the full options produced 1.54% less I frames, which would indicate a slightly wiser bit allocation to come in second pass - and more computationsThis conclusion doesn't follow at all from the premise. Slight differences in frame type choices have literally nothing to do with ratecontrol or the "amount of computations".

There is also no particular reason -- beyond pure chance -- that slower settings would produce more or fewer I-frames.

This thread doesn't make much sense. If you're using such needlessly slow encoding settings to begin with, you're better off sacrificing 0.2% compression and doing 2-pass, and using slightly faster settings.

[WasF]

You compared two extreme sets of settings

I didn't choose "extreme" settings on purpose, those are my actual settings. And "extreme" is actually good here, to make differences (if any) pop up.

you could just pick something in between for pass 1.
Until a suitable crf estimation method is found (will it ever ?), that looks like the only reliable option.

Slight differences in frame type choices have literally nothing to do with ratecontrol or the "amount of computations".
I don't know about ratecontrol, but I thought for a fixed set of frames, picking less I frames means more P & B frames (which need ME), which means more ME calculations (the bulk of the algorithm)..

This thread doesn't make much sense. If you're using such needlessly slow encoding settings to begin with, you're better off sacrificing 0.2% compression and doing 2-pass, and using slightly faster settings.
I would sacrifice 0.2% size in a heartbeat. My concern is quality (which doesn't mean I would not care about +20% size for its sake, though). What "reasonable" settings would you use for "very good" encoded result at 1500 kb/s from a 720p source ? (assuming an average source: some action, some dark scenes, mostly slow).

[detmek]

OK, "fast first pass" helped ! But, still, no general info about how 1st/2nd pass options relate/should be balanced.

"Fast first pass" is --pass 1, precisely defined in x264's help as: force no-8x8dct, partitions none, me dia, ref 1, subme 1:user,else:2, trellis 0, fast-pskip
Unless you set slow-firstpass/perest placebo.

In my case, that leaves (besides level/profile) : rc-lookahead 60, b-adapt 2, merange 24, no-dct-decimate, direct auto
So, these are the potential offenders. That narrows it down, but I'm still not able to tell if disabling those in first pass would affect final result much.
I guess I'm only left with trying it out and comparing the stats files numbers with and without those options. Could be inconclusive, though, if difference is very small..

How can comparing stats files give you information about video quality?:confused:
When it comes to lossy video encoding (x264 is lossy encoder except for qp 0) quality is measured/compared by watching video, not by reading statistics.
You said that chainging first pass options changed number of I frames by 1,54%. What that means? Is the quality 1,54% better or 1,54% worse? Or is it 20% better or 20% worse? Maybe it is the same? Is this differece even visible when you watch the video?

[WasF]

@detmek: have you read just that one post ? You're missing my point completely.

[detmek]

Ok, change of plan. Here is accumulated data so far:

Options: x264-10bit_32bit.exe --rc-lookahead 60 --bframes 4 --b-adapt 2 --subme 10 --trellis 2 --partitions all --8x8dct
--me esa --merange 24 --threads 4 --no-dct-decimate --direct auto --no-interlaced --level 4.2 --ref 8 --profile high422
-------------------------
Source: TV, nature (60% bright, 20% night, 75% normal/slow, 25% action), 1280x720p, 29.97fps, 75835 frames
AVS: SelectRangeEvery(600,60) # 10%. Assuming 25fps: pick 2.4s at the begining of every 24s slice.
Or SelectRangeEvery(1200,60) # 5%. Assuming 25fps: pick 2.4s at the begining of every 48s slice.
Or SelectRangeEvery(600,180) # 30%. Assuming 25fps: pick 7.2s at the begining of avery 24s slice.
-------------------------
NeedBitrate: 1500
Komisar's formula: stop when GotBitrate in [NeedBitrate-100, NeedBitrate+100], or when reach 4 runs
Run 1 2 3 CRF Seed: 23.00
GotBitrate 2727.18 1818.28 1564.87 Bitrate error = GotBitrate - NeedBitrate
GotCRF 23 25.70 26.75 Reverse rel. Bitrate err = - Bitrate_error / GotBitrate
NextCRF 25.70 26.75 27.00 NextCRF = GotCRF - [ 6 * Reverse_relative_Bitrate_error ]
Duration 1h18 1h16 1h12
-------------------------
Results after full encode:
My method 30%(26.15) Komisar 10%(27.00)
Bitrate 1605 eta 9h30 .. :(
Error 105 ?
Cost 3h24 3h46



Done already : --rc-lookahead 60 --bframes 4 --b-adapt 2 --subme 10 --trellis 2 --partitions all --8x8dct --me esa --merange 24 --threads 4 --no-dct-decimate --direct auto --no-interlaced --level 4.2 --ref 8 --profile high10 (--fps if needed)

That's my only "preset": I don't compromise on quality. For me an encode is maximum quality, or isn't (use CUDA !).
With this, even a first pass is slow.
When you have average CRF of 26 quality is seriously compromised and there is no option or parametar that can do much to save it.
Maybe. Without doing a detailed comparison :
--pass 1 + Full options:
[info]: frame I:895 Avg QP:35.52 size: 35034
[info]: frame P:20819 Avg QP:38.94 size: 6666
[info]: frame B:54135 Avg QP:43.47 size: 859
[info]: consecutive B-frames: 3.0% 4.9% 10.2% 68.0% 13.9%

Just --pass 1:
[info]: frame I:909 Avg QP:35.22 size: 35797
[info]: frame P:21879 Avg QP:38.53 size: 6170
[info]: frame B:53061 Avg QP:42.95 size: 927
[info]: consecutive B-frames: 3.3% 5.8% 13.7% 77.2%


QP 38,94?! I don't want to know how that looks like.
Main problem is:"
...So, yes, final quality would be affected.
How much ? I don't know.
This is probably the answer:

This thread doesn't make much sense. If you're using such needlessly slow encoding settings to begin with, you're better off sacrificing 0.2% compression and doing 2-pass, and using slightly faster settings.
OK, maybe little more, say, 1-2%. It depends what you change.
Maybe you could try with:

--preset veryslow --b-frames 4 --ref 8

or even

--preset slower --b-frames 4 --subme 10

And if you want to quantify difference betweet this and your settings, add:

--tune ssim --ssim

to your and this command line. Then compare SSIM values.
It is not perfect measure but it helps. Also do a visual comparison, too.

[Dark Shikari]

Until a suitable crf estimation method is found (will it ever ?), that looks like the only reliable option.There is no such thing as a "suitable crf estimation method": this is what 2-pass is for.I don't know about ratecontrol, but I thought for a fixed set of frames, picking less I frames means more P & B frames (which need ME), which means more ME calculations (the bulk of the algorithm)..This is not true, please stop guessing wildly about how x264 works. Your statement is so wrong it's fractally wrong -- not merely incorrect, but even the premise doesn't make any sense.

I-frames are not faster to encode at typical settings (they are usually slower). The number of I-frames is a miniscule, irrelevant variable in terms of the total encoding time. Practically everything else matters more.
I would sacrifice 0.2% size in a heartbeat. My concern is quality (which doesn't mean I would not care about +20% size for its sake, though). What "reasonable" settings would you use for "very good" encoded result at 1500 kb/s from a 720p source ? (assuming an average source: some action, some dark scenes, mostly slow).The same as for any video ever: the slowest preset you have the patience to use. I don't understand why this is so incredibly difficult!

I really don't know why I bother posting here when people simply ignore the answers from someone who actually wrote the code they're using.

[WasF]

@detmek: OK, you don't have to quote all previous posts to prove you read them :p

I'm re-encoding 720p material with the goal of using the smallest possible bitrate without affecting quality. 26 is actually too low. Often times, I use values closer to 27, and I can't see a difference on my 22" screen. I would have to get within 4 inches from the screen to see the slight loss of detail (and this is a serial encoder speaking, highly sensitive to codec limits). Maybe 4 Mb/s bitrates are useful on a 55" screen, but for my needs, I noticed 1.5 Mb/s is generally enough, maybe 1.8/1.9 Mb/s when the source has lots of dark scenes (rarely more). Doesn't hurt the disk space usage.
I guess if I used a fast preset, I wouldn't be as satisfied with 1.5 Mb/s, and the "low" bitrate would affect quality. So I'm paying for "low" bitrate with "stronger" settings (very long encoding time). That's how I don't compromise on quality (and I'm paying the price).

The price is hefty, and that's why I'm here looking for cheat-sheets.

You're still missing the point of my last post. The QP you emphasized comes from a test I ran to check if going from full options in pass 1 (slow) to just --pass 1 (sweet) would have an effect on the final quality, in general, and how much?
To that end, I was comparing the output of the two (and used a 480p source to run the test faster). The small difference I noticed lead me the exact same conclusion you reached (use balanced settings), confirmed by Dark Shikari's assertive contribution.

[WasF]

@Dark Shikari: thanks for the clarifications. Will do as you said.

[detmek]

True. Visual quality is subjective. One advice if you are keeping your settings - don't use --me esa. It is very very slow, especialy with --merange 24 and you gain nothing. Also, you can improve compressibility by doing light denoiseing. MVTools is very good for that as it can remove grain but keep details.

[sirt]

Dark Shikari, I personally don't think anybody ignores what you say or what a developper advises. I grant some people may have inflexible beliefs about such and such parameter or options Or they are simply confused (including myself) because they don't understand how it really works.

But look, you can't say "stick to the presets and be happy" albeit it is probably the safest thing to do unless you want to experiment or understand. But alow me to raise a critical issue : what do you think people will do with http://mewiki.project357.com/wiki/X264_Settings ? Probably nothing constructive as reading the similar AVIdemux wiki related page will not illumine them either if they don't understand the principles. I think you need a deep understanding of video compression to adjust parameters but you can't blame somebody for thinking wrong or melting with some options that should not be adjusted when it is not really clearly explained how they work.

I've seen an old tutorial by you where you *recommand* some settings for *very high quality* encodes (http://community.eveonline.com/ingameboard.asp?a=topic&threadID=503062) but you never explain why this parameters are great or not. It's like you suggest using them and that's all. Of course you could retort me or anybody else that wants to understand should just read about h264/h265 norms and related books. It is surely a good idea but it is perhaps conceited. In my opinion, you should write a complete guide to explain exactly what does either parameter and where it is implemented in the source code. It's not too much to ask to help novices to understand what they do ! Saying use this and that and don't worry about won't help to get anything done.

[Dark Shikari]

I've seen an old tutorial by you where you *recommand* some settings for *very high quality* encodes but you never explain why this parameters are great or not.This was years ago before I was a developer, and before I made the presets. I was also rather clueless back then; please don't use posts of mine from 5 years ago as guides to anything at all, except maybe "what not to do".
In my opinion, you should write a complete guide to explain exactly what does either parameter and where it is implemented in the source code. It's not too much to ask to help novices to understand what they do ! Saying use this and that and don't worry about won't help to get anything done. I'm always happy to explain things for people who want to develop x264, but the obscure parameters aren't intended for ordinary users. All of this this leads to people recommending extremely complicated and convoluted methods of using x264, posting terrible and inaccurate information from others, and scaring people away from using x264 because it's "too hard".

Have you seen the number of awful guides around the internet recommending insanely complicated commandlines? Practically every day I have to deal with users who have been confused by these, from ordinary home users to major corporations using x264. Trying to convince everyone to mess with these obscure options is a great way to get people to stop using x264.

Also, a large part of x264's internals can't be explained without literally dozens of pages of background information about how codecs work. It's not that easy to just "write a guide".

[WasF]

Visual quality is subjective. One advice if you are keeping your settings - don't use --me esa. It is very very slow, especialy with --merange 24 and you gain nothing. Also, you can improve compressibility by doing light denoiseing. MVTools is very good for that as it can remove grain but keep details.

Indeed. With end-of-GOP images taken from esa/umh encodes of the same source, and even after 4x zoom, I couldn't decide which one I liked more. Maybe it can make a difference at higher bitrates, but at 1.5 Mb/s, there was no benefit to spending several hours more on --me esa.

Regarding the pre-denoising, I wouldn't think of using it on a clean source, even if improves compressibility (how ?), if it hurts visual quality. Does-it ?
I guess it's less risky with high-bitrate sources (8 or 10 Mb/s and more for 720p, I'm working with 4-6 Mb/s ones).

[nm]

Indeed. With end-of-GOP images taken from esa/umh encodes of the same source, and even after 4x zoom, I couldn't decide which one I liked more. Maybe it can make a difference at higher bitrates
It makes even less difference at higher bitrates.

Regarding the pre-denoising, I wouldn't think of using it on a clean source, even if improves compressibility (how ?)
Even videos that appear "clean" usually have some amount of temporal noise.

if it hurts visual quality. Does-it ?

Noise can hide artifacts such as slight banding, but if you are encoding at a low bitrate, chances are that denoising and adding noise back on playback gives better quality than trying to keep the noise. 10-bit encoding is also a good idea to avoid banding, if you only target PCs for playback.

I guess it's less risky with high-bitrate sources (8 or 10 Mb/s and more for 720p, I'm working with 4-6 Mb/s ones).

No, it's the other way around. The lower the bitrate, the more denoising makes sense.

[detmek]

As nm said.
But, before you start messing arount with MVTools you can try x264 internal denoise filter. Just add --nr xxx to command line where xxx is in 100-500 range to see results. MVTools is far better then x264 denoiser but is is good for experiment.

[WasF]

@nm: thanks for your kind answer. I do use 10-bit. I had no concept of what "temporal noise" is, but I found and example (http://www.youtube.com/watch?v=WHV6SP0lKuU) on Youtube. I would describe it as "a frenzy of semi-transparent tiny bugs crawling all over the screen" !
My source didn't look anything like that, so I wasn't exactly sure that noise reduction would make things any better (if not the opposite !).
But detmek's suggestion was easy to try out..

And the results are quite surprising :

http://desmond.imageshack.us/Himg254/scaled.php?server=254&filename=67492139.jpg&res=landing

No revolution there, but noise reduction does have a slight beneficial effect !

My understanding is that MVTools does a better job than x264's internal filter, so I can't wait to test it out ! Any references ?

[detmek]

Yes, it does. It preserves fine details better and does not blur image much.
Use MVTools that comes in dither package (http://forum.doom9.org/showthread.php?p=1386559#post1386559) and try with light denoising. Something like:
tr = 1 # Temporal radius
mt = true # Internal multithreading
lsb = false # 16-bit
thSAD = 200 # denoising strength
blksize = 16 # block size
overlap = 4 # block overlap
super = MSuper (mt=true)
multi_vec = MAnalyse (super, mt=true, multi=true, blksize=blksize, overlap=overlap, delta=tr)
MDegrainN (super, multi_vec, tr, mt=true, lsb=lsb, thSAD=thSAD, thSAD2=150)