x264 1st pass "Final ratefactor" = CRF value??

[samtroy]

Hi all,

the x264 programmer Dark Shikari once said that the only reason for the first pass in 2-pass encodings is to find out what CRF to use in the second pass. Now, in the log file of the first pass there is a line that says:

"Final ratefactor = ... "

My question is: Is this the exact CRF value that is beeing used for the second pass encoding? In other words: If I would encode the same movie with that given "Final Ratefactor" in CRF, the quality and filesize would be exactly the same as the 2-pass version?

Any ideas?

[sneaker_ger]

No, they don't result in exactly same bitrate. (maybe the same magnitude)

[Groucho2004]

Have a look here (http://forum.doom9.org/showthread.php?p=1033368#post1033368).

[F J Walter]

No, not 100% exactly, but very similar.

That's correct.

To expand on this:

The first pass of encoding operates on an "average bit rate" type ratecontrol. That this is a poorer rate control method doesn't matter too much, because the purpose of the first pass is not to nail the final product just yet, it's just to choose frame types and get relative frame costs, relative to each other (actually, these days it gets per-macroblock cost too: the first pass becomes the "mbtree" lookahead too). This information is immensely helpful in enabling the second pass to get rate control "almost as good as CRF" that also "targets a specific file size" - something that CRF alone cannot do. (And when I say "almost as good as CRF" - the difference will be infinitesimal enough not to have any practical differences).

The "final rate factor" reported by the first pass will be influenced by the fact that the first pass does not use the same rate control as the final pass, and thus its estimates of frame cost may be slightly off due to some frames using higher QPs and some using lower QPs than the final encode will use. However, it averages it out over the entire movie, so it will be pretty accurate, with the frame types and relative frame cost of all frames being the same as they will be in the final pass.

[Sharc]

That's correct.

To expand on this:

The first pass of encoding operates on an "average bit rate" type ratecontrol. That this is a poorer rate control method doesn't matter too much, because the purpose of the first pass is not to nail the final product just yet, it's just to choose frame types and get relative frame costs, relative to each other (actually, these days it gets per-macroblock cost too: the first pass becomes the "mbtree" lookahead too). This information is immensely helpful in enabling the second pass to get rate control "almost as good as CRF" that also "targets a specific file size" - something that CRF alone cannot do. (And when I say "almost as good as CRF" - the difference will be infinitesimal enough not to have any practical differences).

The "final rate factor" reported by the first pass will be influenced by the fact that the first pass does not use the same rate control as the final pass, and thus its estimates of frame cost may be slightly off due to some frames using higher QPs and some using lower QPs than the final encode will use. However, it averages it out over the entire movie, so it will be pretty accurate, with the frame types and relative frame cost of all frames being the same as they will be in the final pass.
Is there some practical limit about how much the bitrate for --pass 2 may deviate from the bitrate of --pass 1?
The question came in the context of "quality-based" encoding, e.g. --pass 1 at a given --crf and given x.264 settings with unknown bitrate, and --pass 2 using the --stats file from pass 1 but with the target bitrate and stronger (or weaker) x.264 quality settings.
Does this make sense at all? Or is the analysis of --pass 1 invalidated if the bitrate is changed? Any experience?

[MasterNobody]

Or is the analysis of --pass 1 invalidated if the bitrate is changed? Any experience?
No, it is ok to change bitrate of second pass (may be there would be a quality hit if you change it too much, for example from 100Mbit/s stats to 1Mbit/s, but not significant if your change bitrate moderately). Even more, you can change resolution of second pass (but this would probably result in a little bit more quality hit).

[Sharc]

No, it is ok to change bitrate of second pass (may be there would be a quality hit if you change it too much, for example from 100Mbit/s stats to 1Mbit/s, but not significant if your change bitrate moderately). Even more, you can change resolution of second pass (but this would probably result in a little bit more quality hit).
Means for example, I could run --pass 1 at --crf 18 with --preset medium. If the resulting bitrate is 12Mbit/s (=> filesize too large) I can run --pass 2 at 9 Mbit/s and --preset slow?

[MasterNobody]

You can't change presets between passes (some of the options like --bframes and others should be kept the same). Also don't forget to use --slow-firstpass in this case (because you probably want to keep result of first pass without second pass if bitrate was ok).

[Sharc]

You can't change presets between passes (some of the options like --bframes and others should be kept the same). Also don't forget to use --slow-firstpass in this case (because you probably want to keep result of first pass without second pass if bitrate was ok).
Good points. Thanks!
In my tests it seems that --bluray-compat has ironed out some of the critical settings like --bframes which should not be changed between the 2 passes.
Otherwise I am quite happy to see this method working for filesize (bitrate) adjustments - if needed - in case of over/undersized 1st-pass crf encodes.

[samtroy]

OK guys, I hate to say it but something must be wrong here! I did some testing with a 10 minute MKV clip of a 120min 1080p movie:

TestClip-10min.mkv / 2.7GB / Bitrate: 32MBit/sec

First I encoded the file in 2-pass mode with a target bitrate of 12MBit/sec:

1st pass:

"E:\SimpleX264Launcher\toolset\x64\x264_8bit_x64.exe" --bitrate 12000 --pass 1 --stats H:\\TestClip-10min.stats --preset slow --tune film --profile high --level 4.1 --ref 4 --vbv-maxrate 34000 --vbv-bufsize 30000 --output H:\\TestClip-10min.mkv --index C:\...\Temp\~7fb30c5ea892b4218be3.ffindex G:\TestClip-10min.mkv

x264 [info]: frame I:105 Avg QP:11.58 size:197585
x264 [info]: frame P:4635 Avg QP:15.11 size: 83148
x264 [info]: frame B:9644 Avg QP:17.16 size: 51403
x264 [info]: final ratefactor: 17.21
encoded 14384 frames, 47.59 fps, 12026.30 kb/s

and then Pass 2:

E:\SimpleX264Launcher\toolset\x64\x264_8bit_x64.exe" --bitrate 12000 --pass 2 --stats H:\\TestClip-10min.stats --preset slow --tune film --profile high --level 4.1 --ref 4 --vbv-maxrate 34000 --vbv-bufsize 30000 --output H:\\TestClip-10min.mkv --index C:\...\Temp\~7fb30c5ea892b4218be3.ffindex G:\TestClip-10min.mkv

x264 [info]: frame I:105 Avg QP:13.56 size:180904
x264 [info]: frame P:4635 Avg QP:17.60 size: 84195
x264 [info]: frame B:9644 Avg QP:19.06 size: 50840
encoded 14384 frames, 21.80 fps, 11995.21 kb/s
Final file size is 858.0 MB bytes.

The resulting file was around 900MB and the bitrate was exactly 12MBit/sec. Everthing was fine. And according to the info given here I thought that the file had been encoded with a CRF of around 17 in the second pass, because the 1st pass logfile showed a "final ratefactor" of 17.21.

But that's apperantly not the case! Because now I encoded the same 10min original file in a single pass with the CRF value of 17. All other settings stayed the same. The filesize and the applied compression should be around the same, right?? WRONG! See for yourself:

"E:\SimpleX264Launcher\toolset\x64\x264_8bit_x64.exe" --crf 17.0 --preset slow --tune film --profile high --level 4.1 --ref 4 --vbv-maxrate 34000 --vbv-bufsize 30000 --output "H:\\TestClip-10min.mkv" --index C:\...\Temp\~16340ce2e3b7b9f16148.ffindex "G:\TestClip-10min.mkv"

x264 [info]: frame I:111 Avg QP:11.12 size:218429
x264 [info]: frame P:4421 Avg QP:15.35 size:127603
x264 [info]: frame B:9852 Avg QP:16.62 size: 88176
encoded 14384 frames, 17.50 fps, 19430.01 kb/s
Final file size is 1.36 GB bytes.

The CRF 17 encoded file came out with a bitrate of 19.5MBit/sec and was also of course much, much larger (1.36 GB compared to 858 MB)! So there's *no* way that the 2-pass mode did encode the file with a CRF of 17.21 in the second pass. It must have been much, much higher.

Bottom line/conclusion: Maybe the "final ratefactor" shown in the logfile of the first pass has absolutely nothing to do with the "normal" CRF value afterall? :(

And if that is the case, how can I find out what "CRF-like quality" a 2-pass encoding actually has?

I am totally confused now. Any ideas?

[Sharc]

Repeat the --crf encode but set the value to 17.21 rather than 17.0 and see what you get. It can make quite a difference depending on the source.
You may also do the same with added --slow-firstpass for the 1st pass and see how it impacts the final ratefactor.

[samtroy]

@Sharc:

Did it again with CRF 17.21: Still a 18.4MB/sec bitrate! Should be 12 MBit/sec.. :(

[Groucho2004]

And according to the info given here I thought that the file had been encoded with a CRF of around 17 in the second pass, because the 1st pass logfile showed a "final ratefactor" of 17.21.
That means that you either didn't read akupenguin's comments I linked to in post #3 or you didn't understand them.

[the_weirdo]

Bottom line/conclusion: Maybe the "final ratefactor" shown in the logfile of the first pass has absolutely nothing to do with the "normal" CRF value afterall? :(


Like sneaker_ger said, they're very similar but not the same. Have a look at this thread (http://doom10.org/index.php?topic=1393.0).

[sneaker_ger]

No, ignore my first post (edited). I just did a test on a sample and "very similar" was a stretch. "Same magnitude" fits better. But to even reach that you might need --slow-firstpass.

[foxyshadis]

CRF and final ratefactor aren't the same thing. They're vaguely related, but if you use the exact same settings with the exact crf you get from the final... you'll get a different final ratefactor. Rate control is pretty complicated, and doesn't entirely make sense. If you need a bitrate, you have to use 2-pass; if you don't mind if it floats but have your limits, use VBV to constrain it, once you have a rough idea of what CRF and preset is your favorite size/quality/time tradeoff. Don't use unconstrained CRF if you do care, or you will be unpleasantly surprised occasionally.

If you're willing to do extra filtering or reduce resolution if you can't hit your size/quality, instead of just starving bits from hard scenes, then run one of the many compressibility checks first instead of wasting time on a whole encode. Those can flag when a movie will be wildly outsized, like 300.

[Groucho2004]

There have been several threads about this over the years and I could never understand the obsession with figuring out the accurate CRF value in order to hit a certain target file size/bitrate.

The only explanation I can think of is that there seems to be a misconception that CRF yields better quality than a 2-pass/multi pass encode. The differences are minuscule as pointed out on various occasions by the developers.

So, to iterate for the 1000th time - use CRF if you want to maintain a constant quality, or 2 pass, bitrate based if you want to hit a certain target size.

[Sharc]

There have been several threads about this over the years and I could never understand the obsession with figuring out the accurate CRF value in order to hit a certain target file size/bitrate.

- Striving for "the impossible"
- Approximate a target size in 1-pass but better quality than 1-pass VBR ("Save the Planet..." :D)

[samtroy]

Well, the thing is:

Dark Shikari once said (in another forum I think):

"CRF and 2-pass use identical bit allocation algorithms. All 2-pass does is pick the CRF value that gives the bitrate you want. It's still using the CRF algorithm."

And I just wanted to know *which* CRF value is being used on the second pass of my encodes, that's all. I mean, that information should be somewhere, right?

[Groucho2004]

And I just wanted to know *which* CRF value is being used on the second pass of my encodes, that's all. I mean, that information should be somewhere, right?
It's not a static value, it's an iterative algorithm in the second pass that uses info from the first pass. The "Final Ratefactor" is purely informational, and clearly being continuously mis-interpreted. I don't even know what it is good for since the QP values determined and displayed after the first pass have much better informational value.

[raffriff42]

Multi-pass encoding generally selects the compression parameters on a scene-by-scene basis, doesn't it? So the final "CRF" is an estimate of some kind - an average of the quantizer settings in all scenes. You would have to dig into the x264 source code to see how they arrived at that number. Agree w/ Groucho2004, it's meaningless. I don't think there is any single CRF applied throughout the entire video; it changes.

[F J Walter]

The ratecontrol in the 2nd (final) pass comes as close as possible to a "constant" rate factor but doesn't exactly hit it - the rate factor will vary by teeny, tiny, negligible on a frame by frame basis. The extent to which it doesn't hit a constant rate factor is unknown prior to encoding the second pass.

x264 *could* theoretically compute the average rate factor for the second pass and show that at the end of the encode for information passes, and if it did that, you would be able to get an almost identical (but never exactly) ratecontrol by choosing CRF and using that ratefactor on the same video.

Note that the more options you change in between the first and second pass, the less closely the final ratecontrol will resemble a true CRF ratecontrol, however for most cases the difference is negligible (which is why --slow-first-pass is usually considered overkill). But even if all options are identical in both passes (ie, you use --slow-first-pass), it still only comes very close to, but doesn't exactly hit, a constant rate factor. This is because the first pass stats are an extrapolation of the frame costs from the imperfect ABR style ratecontrol used on the first pass.

The difference between a CRF encode and a 2-pass encode of the same file size is far too negligible to even worry about any of this. The deviation in rate control between the two will be orders of magnitude less than enough to actually see the difference.

If you are that obsessed about reaching a constant rate factor, then you could do 3-pass which will come even closer. I, however, would put 3-pass into a "placebo" category.

Re the quote from Dark Shikari:

CRF and 2-pass use identical bit allocation algorithms. All 2-pass does is pick the CRF value that gives the bitrate you want. It's still using the CRF algorithm

Yes, they use an identical formula to calculate the bit allocation based on the rate factor. However, as I mentioned above in CRF mode the rate factor is constant, in the final pass of 2-pass it is very, very nearly constant, close enough that the difference can be ignored.