Estimating/Predicting CRF bitrate or file size

[Makaveli84]

As the title says, is it possible to estimate beforehand the expected bitrate or file size of a given encode at a given CRF factor??
The reason I ask is because I would like to estimate the output size of an encode at CRF 20 for example, then actually encode the video in a size based 2-pass mode. I am not trying to keep the encodes below a certain file size, but rather, trying to maximize quality at the best file size possible. So encoding in 2-pass without a quality reference would be flying blindly. That's why I rely on a constant quality CRF pass to get the needed bitrate for a 2-pass encode to achieve a certain level of quality.
But as you can imagine, this is a very slow and time/resources consuming approach. This is why I'm looking for some kind of estimation method or quick virtual CRF pass that estimates the bitrate quickly without actually encoding the video.

Thanks.




P.S: Please, let the CRF vs 2-pass debate for another thread. To me, based on the testing that I've done, 2-pass produces slightly better (but pretty noticeable) quality than CRF at a given bitrate.

[Groucho2004]

Originally Posted by Makaveli84

As the title says, is it possible to estimate beforehand the expected bitrate or file size of a given encode at a given CRF factor??

No. The bitrate depends on the source (complexity, etc.).

Originally Posted by Makaveli84

The reason I ask is because I would like to estimate the output size of an encode at CRF 20 for example, then actually encode the video in a size based 2-pass mode.

That's what I do mostly. I run a compressibility pass with a certain CRF value on about 5%-10% of the source which gives me a pretty accurate number for the required bitrate.

[Makaveli84]

Originally Posted by Groucho2004

No. The bitrate depends on the source (complexity, etc.).

Yes, I know. I don't mean estimating in general. I am asking about estimating the bitrate of the output of a CRF n encode of a specific source, and not the bitrate of CRF n in general.

Originally Posted by Groucho2004

That's what I do mostly. I run a compressibility pass with a certain CRF value on about 5%-10% of the source which gives me a pretty accurate number for the required bitrate.

Hmm.. Does this really work? I mean the source has to be really monotone for this to work, which is hardly the case in real-life videos.

[Groucho2004]

Originally Posted by Makaveli84

Yes, I know. I don't mean estimating in general. I am asking about estimating the bitrate of the output of a CRF n encode of a specific source, and not the bitrate of CRF n in general.

I don't know how else to put it. You have to analyse each individual source in order to obtain the required bitrate for a given CRF value. There is no way to calculate it.

Originally Posted by Makaveli84

Hmm.. Does this really work? I mean the source has to be really monotone for this to work, which is hardly the case in real-life videos.

Obviously you have to use something like SelectRangeEvery(1000, 100) in Avisynth (select 100 out of every 1000 frames). It's quite accurate even with just 5%.

[detmek]

x264.exe --crf 20 --pass 1 --slow-firstpass --other options
x264.exe --bitrate (obtaind from the first pass) --pass 2 --other options
But the quality will be the same as the first pass.

[Makaveli84]

Originally Posted by Groucho2004

Obviously you have to use something like SelectRangeEvery(1000, 100) in Avisynth (select 100 out of every 1000 frames). It's quite accurate even with just 5%.

Can u elaborate further? I'm not very savvy in Avisynth. Can you suggest a sample script. Also, what do you feed the script into? I for example encode in Avidemux, so what I would need to do is feed the mentioned script into Avidemux instead of the actual video, right?

[Groucho2004]

Originally Posted by Makaveli84

Can u elaborate further? I'm not very savvy in Avisynth. Can you suggest a sample script. Also, what do you feed the script into? I for example encode in Avidemux, so what I would need to do is feed the mentioned script into Avidemux instead of the actual video, right?

If you are not familiar with Avisynth, a sample won't help you much. You simply put the SelectRangeEvery() line at the end of the script and feed it to x264.

Can't help you with GUIs, never used any, sorry.

[Makaveli84]

Originally Posted by Groucho2004

If you are not familiar with Avisynth, a sample won't help you much. You simply put the SelectRangeEvery() line at the end of the script and feed it to x264.

Can't help you with Gui's, never used any, sorry.

I am familiar with Avisynth, so a sample script would be of great help. For example, would it look something like this?

DirectShowSource("input.mkv", audio=false, fps=23.976)
SelectRangeEvery(1000, 100)

Do I need to add something else, assuming no filtering is to be applied on the source?

[Makaveli84]

Originally Posted by detmek

x264.exe --crf 20 --pass 1 --slow-firstpass --other options
x264.exe --bitrate (obtaind from the first pass) --pass 2 --other options
But the quality will be the same as the first pass.

When you say --bitrate (obtaind from the first pass), is there anyway to get this automatically? If not, then this method beats the purpose of an overnight 2-pass encode. If I have to wait for the result of a full single CRF pass anyway, then I might as well use the bitrate in a full additional 2-pass overnight encode.

[Groucho2004]

Originally Posted by Makaveli84

For example, would it look something like this?

DirectShowSource("input.mkv", audio=false, fps=23.976)
SelectRangeEvery(1000, 100)

Yes, as simple as that.

[J_Darnley]

Originally Posted by Makaveli84

When you say --bitrate (obtaind from the first pass), is there anyway to get this automatically?

Yes. Parse the text printed by x264 for the bitrate value, use that value as the argument for bitrate in the second pass.

I don't know why you would run a slow first pass with crf and then (on average) get the same result again on the second pass.

[detmek]

Originally Posted by J_Darnley

I don't know why you would run a slow first pass with crf and then (on average) get the same result again on the second pass.

He said:

Originally Posted by Makaveli84

P.S: Please, let the CRF vs 2-pass debate for another thread. To me, based on the testing that I've done, 2-pass produces slightly better (but pretty noticeable) quality than CRF at a given bitrate.

Originally Posted by Makaveli84

To me, based on the testing that I've done, 2-pass produces slightly better (but pretty noticeable) quality than CRF at a given bitrate.

Which isn't possibly true if you really did encode with CRF and 2-pass with the same settings and both were at the same bitrate. 2-pass uses the CRF algorithm to encode it just selects the appropriate CRF value for the bitrate you specified. You've basically tried to claim that CRF is inferior to itself which is a nonsensical statement.

And you don't have to take my word for it. From Dark Shikari's mouth:

CRF and 2-pass use identical bit allocation algorithms. All 2-pass does is pick the CRF value that gives the filesize you want. It's still using the CRF algorithm.

There is no way your CRF encode looked worse than a 2-pass unless you didn't use equivalent settings or a CRF value that matched the bitrate you chose for 2-pass. If it did produce a noticeably different result despite everything being exactly the same between the two encodes you have uncovered a bug.

[Groucho2004]

Originally Posted by SassBot

There is no way your CRF encode looked worse than a 2-pass unless you didn't use equivalent settings or a CRF value that matched the bitrate you chose for 2-pass.

He's using a "GUI" so that might explain different settings. I can imagine that these programs tend to hide certain settings from the user because the programmer thinks that an inexperienced user should not touch them. I don't know avidemux so this is just a hypothesis.

Originally Posted by J_Darnley

I don't know why you would run a slow first pass with crf and then (on average) get the same result again on the second pass.

Maybe they needs to keep their house warm? This is exactly correct. You are basically rerunning it to get what will (barring any options you select that result in non-deterministic behavior) get exactly the same result except you've just wasted extra hours doing so. It's amazing that no matter how many times this gets corrected people are still under the misguided notion that CRF and 2-pass are somehow fundamentally different.

Originally Posted by Groucho2004

He's using a "GUI" so that might explain different settings. I can imagine that these programs tend to hide certain settings from the user because the programmer thinks that an inexperienced user should not touch them. I don't know avidemux so this is just a hypothesis.

I guess that's possible. Either that or he's just convinced himself that he sees a difference where there isn't one which is not uncommon.

[Makaveli84]

Originally Posted by SassBot

Which isn't possibly true if you really did encode with CRF and 2-pass with the same settings and both were at the same bitrate. 2-pass uses the CRF algorithm to encode it just selects the appropriate CRF value for the bitrate you specified. You've basically tried to claim that CRF is inferior to itself which is a nonsensical statement.

Originally Posted by SassBot

And you don't have to take my word for it. From Dark Shikari's mouth:

There is no way your CRF encode looked worse than a 2-pass unless you didn't use equivalent settings or a CRF value that matched the bitrate you chose for 2-pass. If it did produce a noticeably different result despite everything being exactly the same between the two encodes you have uncovered a bug.

So much for my plead to leave the CRF x 2-pass discussion out of this thread .
Well, anyways Mr. Sass, I am very much aware of the point you made, having read extensively about this matter. Actually, I had even spent some good time reading the thread you quoted Dark Shikari's post from. Nevertheless, it seems a lot of people are still unconvinced with your claims. I for one am neither convinced nor unconvinced. I am skeptical, and I am running trials to find out myself.
Which brings me to my second point: both my CRF and 2-pass encodes were done with the same settings. I am using a GUI, but I am not aware of any settings it's hiding. So as far as I can tell, both encodes have the exact same settings.

Originally Posted by SassBot

Either that or he's just convinced himself that he sees a difference where there isn't one which is not uncommon.

LOL. Typical arrogance form the "know-it-all"s!!
Anyways, I've ran an ABX test (15 trials), and I scored right 15/15, so trust me, I see a difference.


Anyways, I am willing to accept your claims if you care to explain them further. If 2-pass and CRF are the same, does that mean that the first pass of a 2-pass encode is simply running to determine the "n" needed for the given bitrate or size, and the 2nd pass is nothing more than CRF n ??
Also, what's the precision of "n"?? If it's an integer, then your whole claim is nonsensical. If it's a single float, then it might hold some truth to it, though that would still be inaccurate to hit a given target bitrate. So what is it? CRF n or n.x or n.xy or what?
In the GUI I use, I can only specify integer as CRF's argument. And before you even go there, this has nothing to do with my trial. I encoded in CRF first, and then used the output's bitrate in 2-pass, and not the other way around.

[Makaveli84]

Also, Sass, this from the same thread you quoted:

Q: Is --direct auto a little more efficient in 2 pass mode than crf? If there's a difference, my guess would be that it would be fairly minor, am I correct?
A: Yes, 2-pass results in a better --direct auto, but that's a tiny quality-difference.

And this from this current thread:

I don't know why you would run a slow first pass with crf and then (on average) get the same result again on the second pass.

Which seem to suggest that there is a slight advantage, which no matter how slight it is, contradicts the scientific principle you're proposing which suggests the exact same output because it's the same CRF n pass done twice. I'm willing to be enlightened on this matter too. Please, oh please, do convince me, because I would rather use CRF than 2-pass.

Originally Posted by Makaveli84

So much for my plead to leave the CRF x 2-pass discussion out of this thread .
Well, anyways Mr. Sass, I am very much aware of the point you made, having read extensively about this matter. Actually, I had even spent some good time reading the thread you quoted Dark Shikari's post from. Nevertheless, it seems a lot of people are still unconvinced with your claims. I for one am neither convinced nor unconvinced. I am skeptical, and I am running trials to find out myself

People are also convinced that $100 Monster Cable HDMI cables provide a difference and yet they don't. People routinely convince themselves of truths that don't actually pan out.

Originally Posted by Makaveli84

Which brings me to my second point: both my CRF and 2-pass encodes were done with the same settings. I am using a GUI, but I am not aware of any settings it's hiding. So as far as I can tell, both encodes have the exact same settings.

Then if you saw differences you encountered a bug since they USE THE EXACT SAME ALGORITHM. 2-pass is just a way for x264 to guess the CRF value for you based on the bitrate you specify.

Originally Posted by Makaveli84

LOL. Typical arrogance form the "know-it-all"s!!
Anyways, I've ran an ABX test (15 trials), and I scored right 15/15, so trust me, I see a difference.

Okay, then back this up with your settings, your test clips and the encoded clips.

Originally Posted by Makaveli84

Anyways, I am willing to accept your claims if you care to explain them further. If 2-pass and CRF are the same, does that mean that the first pass of a 2-pass encode is simply running to determine the "n" needed for the given bitrate or size, and the 2nd pass is nothing more than CRF n ??

Yes, that is exactly what the analysis pass does. It calculates the CRF value to use for the second pass based on running a first pass with your specified bitrate. How was that not clear from the quote I gave you from Dark Shikari?

Originally Posted by Makaveli84

Also, what's the precision of "n"?? If it's an integer, then your whole claim is nonsensical. If it's a single float, then it might hold some truth to it, though that would still be inaccurate to hit a given target bitrate. So what is it? CRF n or n.x or n.xy or what?

No it's not an integer since you can provide decimal value CRF values. Check your x264 logs and see the value it gives of the ratefactor after the first pass.

Originally Posted by Makaveli84

In the GUI I use, I can only specify integer as CRF's argument. And before you even go there, this has nothing to do with my trial. I encoded in CRF first, and then used the output's bitrate in 2-pass, and not the other way around.

Then your GUI of choice is doing it wrong since CRF is not limited to integer values.

Originally Posted by Makaveli84

Also, Sass, this from the same thread you quoted:

It's highly doubtful that the "small difference" is even remotely visually perceptible.

Originally Posted by Makaveli84

And this from this current thread:


Which seem to suggest that there is a slight advantage, which no matter how slight it is, contradicts the scientific principle you're proposing which suggests the exact same output because it's the same CRF n pass done twice. I'm willing to be enlightened on this matter too. Please, oh please, do convince me, because I would rather use CRF than 2-pass.

No, there really isn't. You are not gaining any actual visual quality and are doing nothing but wasting your time, but it's not my electricity bill.

[Makaveli84]

Originally Posted by SassBot

Then if you saw differences you encountered a bug since they USE THE EXACT SAME ALGORITHM. 2-pass is just a way for x264 to guess the CRF value for you based on the bitrate you specify.

Regardless of what I saw, since they use the EXACT SAME ALGORITHM, there shouldn't be a difference even in still frames (paused video) for example, because we're talking about the same algorithm, and not same "perceptible" quality and/or psychological optimizations, right?
This is not "CRF gives same quality as CQ", this is "CRF = 2-pass because 2nd pass is CRF", and thus, one shouldn't see a difference even in a still frame, right?

Originally Posted by SassBot

Yes, that is exactly what the analysis pass does. It calculates the CRF value to use for the second pass based on running a first pass with your specified bitrate. How was that not clear from the quote I gave you from Dark Shikari?

Clarity is not the issue, I am not asking per say, but rather double checking or reconfirming.. But here's what I still don't get: if all the 1st pass is doing is determining the CRF argument to be used in the 2nd pass, why does it create such large (relatively speaking) .stats and .mbtreestats files, and why does it use them in the 2nd pass??
Or does it not use them in the 2nd pass, in which case, why are they left over then?

Originally Posted by SassBot

No it's not an integer since you can provide decimal value CRF values. Check your x264 logs and see the value it gives of the ratefactor after the first pass.
[..]
Then your GUI of choice is doing it wrong since CRF is not limited to integer values.

What's the precision of CRF's argument (n.xxxx)?? And in CL mode, can the user also provide such a precise value?
Also, how can I check the CRF value of an already encoded file? MediaInfo seems to only provide a rounded integer value.

Originally Posted by SassBot

It's highly doubtful that the "small difference" is even remotely visually perceptible.
[...]
You are not gaining any actual visual quality and are doing nothing but wasting your time

This admittance is what confuses me. I mean I do want to believe, and you make a good scientific argument, but then, you acknowledge the presence of differences that are "small" and that are "visually imperceptible". How can there be differences if it's CRF x CRF? Doesn't CRF = 2-pass mean that the 2 encodes should be identical even on the PSNR level?