Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 4th September 2012, 17:04   #1  |  Link
Makaveli84
Registered User
 
Join Date: Jun 2012
Posts: 71
Estimating/Predicting CRF bitrate or file size

As the title says, is it possible to estimate beforehand the expected bitrate or file size of a given encode at a given CRF factor??
The reason I ask is because I would like to estimate the output size of an encode at CRF 20 for example, then actually encode the video in a size based 2-pass mode. I am not trying to keep the encodes below a certain file size, but rather, trying to maximize quality at the best file size possible. So encoding in 2-pass without a quality reference would be flying blindly. That's why I rely on a constant quality CRF pass to get the needed bitrate for a 2-pass encode to achieve a certain level of quality.
But as you can imagine, this is a very slow and time/resources consuming approach. This is why I'm looking for some kind of estimation method or quick virtual CRF pass that estimates the bitrate quickly without actually encoding the video.

Thanks.




P.S: Please, let the CRF vs 2-pass debate for another thread. To me, based on the testing that I've done, 2-pass produces slightly better (but pretty noticeable) quality than CRF at a given bitrate.
Makaveli84 is offline   Reply With Quote
Old 4th September 2012, 17:10   #2  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by Makaveli84 View Post
As the title says, is it possible to estimate beforehand the expected bitrate or file size of a given encode at a given CRF factor??
No. The bitrate depends on the source (complexity, etc.).

Quote:
Originally Posted by Makaveli84 View Post
The reason I ask is because I would like to estimate the output size of an encode at CRF 20 for example, then actually encode the video in a size based 2-pass mode.
That's what I do mostly. I run a compressibility pass with a certain CRF value on about 5%-10% of the source which gives me a pretty accurate number for the required bitrate.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 4th September 2012, 17:29   #3  |  Link
Makaveli84
Registered User
 
Join Date: Jun 2012
Posts: 71
Quote:
Originally Posted by Groucho2004 View Post
No. The bitrate depends on the source (complexity, etc.).
Yes, I know. I don't mean estimating in general. I am asking about estimating the bitrate of the output of a CRF n encode of a specific source, and not the bitrate of CRF n in general.


Quote:
Originally Posted by Groucho2004 View Post
That's what I do mostly. I run a compressibility pass with a certain CRF value on about 5%-10% of the source which gives me a pretty accurate number for the required bitrate.
Hmm.. Does this really work? I mean the source has to be really monotone for this to work, which is hardly the case in real-life videos.
Makaveli84 is offline   Reply With Quote
Old 4th September 2012, 17:41   #4  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by Makaveli84 View Post
Yes, I know. I don't mean estimating in general. I am asking about estimating the bitrate of the output of a CRF n encode of a specific source, and not the bitrate of CRF n in general.
I don't know how else to put it. You have to analyse each individual source in order to obtain the required bitrate for a given CRF value. There is no way to calculate it.

Quote:
Originally Posted by Makaveli84 View Post
Hmm.. Does this really work? I mean the source has to be really monotone for this to work, which is hardly the case in real-life videos.
Obviously you have to use something like SelectRangeEvery(1000, 100) in Avisynth (select 100 out of every 1000 frames). It's quite accurate even with just 5%.
__________________
Groucho's Avisynth Stuff

Last edited by Groucho2004; 4th September 2012 at 17:43.
Groucho2004 is offline   Reply With Quote
Old 4th September 2012, 17:49   #5  |  Link
detmek
Registered User
 
Join Date: Aug 2009
Posts: 463
x264.exe --crf 20 --pass 1 --slow-firstpass --other options
x264.exe --bitrate (obtaind from the first pass) --pass 2 --other options
But the quality will be the same as the first pass.
detmek is offline   Reply With Quote
Old 4th September 2012, 17:49   #6  |  Link
Makaveli84
Registered User
 
Join Date: Jun 2012
Posts: 71
Quote:
Originally Posted by Groucho2004 View Post
Obviously you have to use something like SelectRangeEvery(1000, 100) in Avisynth (select 100 out of every 1000 frames). It's quite accurate even with just 5%.
Can u elaborate further? I'm not very savvy in Avisynth. Can you suggest a sample script. Also, what do you feed the script into? I for example encode in Avidemux, so what I would need to do is feed the mentioned script into Avidemux instead of the actual video, right?
Makaveli84 is offline   Reply With Quote
Old 4th September 2012, 18:01   #7  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by Makaveli84 View Post
Can u elaborate further? I'm not very savvy in Avisynth. Can you suggest a sample script. Also, what do you feed the script into? I for example encode in Avidemux, so what I would need to do is feed the mentioned script into Avidemux instead of the actual video, right?
If you are not familiar with Avisynth, a sample won't help you much. You simply put the SelectRangeEvery() line at the end of the script and feed it to x264.

Can't help you with GUIs, never used any, sorry.
__________________
Groucho's Avisynth Stuff

Last edited by Groucho2004; 4th September 2012 at 21:48. Reason: typo
Groucho2004 is offline   Reply With Quote
Old 4th September 2012, 18:18   #8  |  Link
Makaveli84
Registered User
 
Join Date: Jun 2012
Posts: 71
Quote:
Originally Posted by Groucho2004 View Post
If you are not familiar with Avisynth, a sample won't help you much. You simply put the SelectRangeEvery() line at the end of the script and feed it to x264.

Can't help you with Gui's, never used any, sorry.
I am familiar with Avisynth, so a sample script would be of great help. For example, would it look something like this?
Code:
DirectShowSource("input.mkv", audio=false, fps=23.976)
SelectRangeEvery(1000, 100)
Do I need to add something else, assuming no filtering is to be applied on the source?
Makaveli84 is offline   Reply With Quote
Old 4th September 2012, 18:27   #9  |  Link
Makaveli84
Registered User
 
Join Date: Jun 2012
Posts: 71
Quote:
Originally Posted by detmek View Post
x264.exe --crf 20 --pass 1 --slow-firstpass --other options
x264.exe --bitrate (obtaind from the first pass) --pass 2 --other options
But the quality will be the same as the first pass.
When you say --bitrate (obtaind from the first pass), is there anyway to get this automatically? If not, then this method beats the purpose of an overnight 2-pass encode. If I have to wait for the result of a full single CRF pass anyway, then I might as well use the bitrate in a full additional 2-pass overnight encode.
Makaveli84 is offline   Reply With Quote
Old 4th September 2012, 18:38   #10  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by Makaveli84 View Post
For example, would it look something like this?
Code:
DirectShowSource("input.mkv", audio=false, fps=23.976)
SelectRangeEvery(1000, 100)
Yes, as simple as that.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 4th September 2012, 21:22   #11  |  Link
J_Darnley
Registered User
 
J_Darnley's Avatar
 
Join Date: May 2006
Posts: 957
Quote:
Originally Posted by Makaveli84 View Post
When you say --bitrate (obtaind from the first pass), is there anyway to get this automatically?
Yes. Parse the text printed by x264 for the bitrate value, use that value as the argument for bitrate in the second pass.

I don't know why you would run a slow first pass with crf and then (on average) get the same result again on the second pass.
__________________
x264 log explained || x264 deblocking how-to
preset -> tune -> user set options -> fast first pass -> profile -> level
Doom10 - Of course it's better, it's one more.
J_Darnley is offline   Reply With Quote
Old 4th September 2012, 21:38   #12  |  Link
detmek
Registered User
 
Join Date: Aug 2009
Posts: 463
Quote:
Originally Posted by J_Darnley View Post
I don't know why you would run a slow first pass with crf and then (on average) get the same result again on the second pass.
He said:
Quote:
Originally Posted by Makaveli84 View Post

P.S: Please, let the CRF vs 2-pass debate for another thread. To me, based on the testing that I've done, 2-pass produces slightly better (but pretty noticeable) quality than CRF at a given bitrate.
detmek is offline   Reply With Quote
Old 4th September 2012, 21:53   #13  |  Link
SassBot
Guest
 
Posts: n/a
Quote:
Originally Posted by Makaveli84 View Post
To me, based on the testing that I've done, 2-pass produces slightly better (but pretty noticeable) quality than CRF at a given bitrate.
Which isn't possibly true if you really did encode with CRF and 2-pass with the same settings and both were at the same bitrate. 2-pass uses the CRF algorithm to encode it just selects the appropriate CRF value for the bitrate you specified. You've basically tried to claim that CRF is inferior to itself which is a nonsensical statement.

Last edited by SassBot; 4th September 2012 at 22:03.
  Reply With Quote
Old 4th September 2012, 22:17   #14  |  Link
SassBot
Guest
 
Posts: n/a
And you don't have to take my word for it. From Dark Shikari's mouth:

Quote:
CRF and 2-pass use identical bit allocation algorithms. All 2-pass does is pick the CRF value that gives the filesize you want. It's still using the CRF algorithm.
There is no way your CRF encode looked worse than a 2-pass unless you didn't use equivalent settings or a CRF value that matched the bitrate you chose for 2-pass. If it did produce a noticeably different result despite everything being exactly the same between the two encodes you have uncovered a bug.
  Reply With Quote
Old 4th September 2012, 22:28   #15  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by SassBot View Post
There is no way your CRF encode looked worse than a 2-pass unless you didn't use equivalent settings or a CRF value that matched the bitrate you chose for 2-pass.
He's using a "GUI" so that might explain different settings. I can imagine that these programs tend to hide certain settings from the user because the programmer thinks that an inexperienced user should not touch them. I don't know avidemux so this is just a hypothesis.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 4th September 2012, 22:32   #16  |  Link
SassBot
Guest
 
Posts: n/a
Quote:
Originally Posted by J_Darnley View Post
I don't know why you would run a slow first pass with crf and then (on average) get the same result again on the second pass.
Maybe they needs to keep their house warm? This is exactly correct. You are basically rerunning it to get what will (barring any options you select that result in non-deterministic behavior) get exactly the same result except you've just wasted extra hours doing so. It's amazing that no matter how many times this gets corrected people are still under the misguided notion that CRF and 2-pass are somehow fundamentally different.

Quote:
Originally Posted by Groucho2004 View Post
He's using a "GUI" so that might explain different settings. I can imagine that these programs tend to hide certain settings from the user because the programmer thinks that an inexperienced user should not touch them. I don't know avidemux so this is just a hypothesis.
I guess that's possible. Either that or he's just convinced himself that he sees a difference where there isn't one which is not uncommon.

Last edited by SassBot; 4th September 2012 at 22:39.
  Reply With Quote
Old 5th September 2012, 00:43   #17  |  Link
Makaveli84
Registered User
 
Join Date: Jun 2012
Posts: 71
Quote:
Originally Posted by SassBot View Post
Which isn't possibly true if you really did encode with CRF and 2-pass with the same settings and both were at the same bitrate. 2-pass uses the CRF algorithm to encode it just selects the appropriate CRF value for the bitrate you specified. You've basically tried to claim that CRF is inferior to itself which is a nonsensical statement.
Quote:
Originally Posted by SassBot View Post
And you don't have to take my word for it. From Dark Shikari's mouth:

There is no way your CRF encode looked worse than a 2-pass unless you didn't use equivalent settings or a CRF value that matched the bitrate you chose for 2-pass. If it did produce a noticeably different result despite everything being exactly the same between the two encodes you have uncovered a bug.
So much for my plead to leave the CRF x 2-pass discussion out of this thread .
Well, anyways Mr. Sass, I am very much aware of the point you made, having read extensively about this matter. Actually, I had even spent some good time reading the thread you quoted Dark Shikari's post from. Nevertheless, it seems a lot of people are still unconvinced with your claims. I for one am neither convinced nor unconvinced. I am skeptical, and I am running trials to find out myself.
Which brings me to my second point: both my CRF and 2-pass encodes were done with the same settings. I am using a GUI, but I am not aware of any settings it's hiding. So as far as I can tell, both encodes have the exact same settings.



Quote:
Originally Posted by SassBot View Post
Either that or he's just convinced himself that he sees a difference where there isn't one which is not uncommon.
LOL. Typical arrogance form the "know-it-all"s!!
Anyways, I've ran an ABX test (15 trials), and I scored right 15/15, so trust me, I see a difference.


Anyways, I am willing to accept your claims if you care to explain them further. If 2-pass and CRF are the same, does that mean that the first pass of a 2-pass encode is simply running to determine the "n" needed for the given bitrate or size, and the 2nd pass is nothing more than CRF n ??
Also, what's the precision of "n"?? If it's an integer, then your whole claim is nonsensical. If it's a single float, then it might hold some truth to it, though that would still be inaccurate to hit a given target bitrate. So what is it? CRF n or n.x or n.xy or what?
In the GUI I use, I can only specify integer as CRF's argument. And before you even go there, this has nothing to do with my trial. I encoded in CRF first, and then used the output's bitrate in 2-pass, and not the other way around.
Makaveli84 is offline   Reply With Quote
Old 5th September 2012, 01:06   #18  |  Link
Makaveli84
Registered User
 
Join Date: Jun 2012
Posts: 71
Also, Sass, this from the same thread you quoted:
Quote:
Q: Is --direct auto a little more efficient in 2 pass mode than crf? If there's a difference, my guess would be that it would be fairly minor, am I correct?
A: Yes, 2-pass results in a better --direct auto, but that's a tiny quality-difference.
And this from this current thread:
Quote:
I don't know why you would run a slow first pass with crf and then (on average) get the same result again on the second pass.
Which seem to suggest that there is a slight advantage, which no matter how slight it is, contradicts the scientific principle you're proposing which suggests the exact same output because it's the same CRF n pass done twice. I'm willing to be enlightened on this matter too. Please, oh please, do convince me, because I would rather use CRF than 2-pass.
Makaveli84 is offline   Reply With Quote
Old 5th September 2012, 04:18   #19  |  Link
SassBot
Guest
 
Posts: n/a
Quote:
Originally Posted by Makaveli84 View Post
So much for my plead to leave the CRF x 2-pass discussion out of this thread .
Well, anyways Mr. Sass, I am very much aware of the point you made, having read extensively about this matter. Actually, I had even spent some good time reading the thread you quoted Dark Shikari's post from. Nevertheless, it seems a lot of people are still unconvinced with your claims. I for one am neither convinced nor unconvinced. I am skeptical, and I am running trials to find out myself
People are also convinced that $100 Monster Cable HDMI cables provide a difference and yet they don't. People routinely convince themselves of truths that don't actually pan out.

Quote:
Originally Posted by Makaveli84 View Post
Which brings me to my second point: both my CRF and 2-pass encodes were done with the same settings. I am using a GUI, but I am not aware of any settings it's hiding. So as far as I can tell, both encodes have the exact same settings.
Then if you saw differences you encountered a bug since they USE THE EXACT SAME ALGORITHM. 2-pass is just a way for x264 to guess the CRF value for you based on the bitrate you specify.

Quote:
Originally Posted by Makaveli84 View Post
LOL. Typical arrogance form the "know-it-all"s!!
Anyways, I've ran an ABX test (15 trials), and I scored right 15/15, so trust me, I see a difference.
Okay, then back this up with your settings, your test clips and the encoded clips.

Quote:
Originally Posted by Makaveli84 View Post
Anyways, I am willing to accept your claims if you care to explain them further. If 2-pass and CRF are the same, does that mean that the first pass of a 2-pass encode is simply running to determine the "n" needed for the given bitrate or size, and the 2nd pass is nothing more than CRF n ??
Yes, that is exactly what the analysis pass does. It calculates the CRF value to use for the second pass based on running a first pass with your specified bitrate. How was that not clear from the quote I gave you from Dark Shikari?

Quote:
Originally Posted by Makaveli84 View Post
Also, what's the precision of "n"?? If it's an integer, then your whole claim is nonsensical. If it's a single float, then it might hold some truth to it, though that would still be inaccurate to hit a given target bitrate. So what is it? CRF n or n.x or n.xy or what?
No it's not an integer since you can provide decimal value CRF values. Check your x264 logs and see the value it gives of the ratefactor after the first pass.

Quote:
Originally Posted by Makaveli84 View Post
In the GUI I use, I can only specify integer as CRF's argument. And before you even go there, this has nothing to do with my trial. I encoded in CRF first, and then used the output's bitrate in 2-pass, and not the other way around.
Then your GUI of choice is doing it wrong since CRF is not limited to integer values.

Quote:
Originally Posted by Makaveli84 View Post
Also, Sass, this from the same thread you quoted:
It's highly doubtful that the "small difference" is even remotely visually perceptible.

Quote:
Originally Posted by Makaveli84 View Post
And this from this current thread:


Which seem to suggest that there is a slight advantage, which no matter how slight it is, contradicts the scientific principle you're proposing which suggests the exact same output because it's the same CRF n pass done twice. I'm willing to be enlightened on this matter too. Please, oh please, do convince me, because I would rather use CRF than 2-pass.
No, there really isn't. You are not gaining any actual visual quality and are doing nothing but wasting your time, but it's not my electricity bill.

Last edited by SassBot; 5th September 2012 at 04:28.
  Reply With Quote
Old 5th September 2012, 11:15   #20  |  Link
Makaveli84
Registered User
 
Join Date: Jun 2012
Posts: 71
Quote:
Originally Posted by SassBot View Post
Then if you saw differences you encountered a bug since they USE THE EXACT SAME ALGORITHM. 2-pass is just a way for x264 to guess the CRF value for you based on the bitrate you specify.
Regardless of what I saw, since they use the EXACT SAME ALGORITHM, there shouldn't be a difference even in still frames (paused video) for example, because we're talking about the same algorithm, and not same "perceptible" quality and/or psychological optimizations, right?
This is not "CRF gives same quality as CQ", this is "CRF = 2-pass because 2nd pass is CRF", and thus, one shouldn't see a difference even in a still frame, right?



Quote:
Originally Posted by SassBot View Post
Yes, that is exactly what the analysis pass does. It calculates the CRF value to use for the second pass based on running a first pass with your specified bitrate. How was that not clear from the quote I gave you from Dark Shikari?
Clarity is not the issue, I am not asking per say, but rather double checking or reconfirming.. But here's what I still don't get: if all the 1st pass is doing is determining the CRF argument to be used in the 2nd pass, why does it create such large (relatively speaking) .stats and .mbtreestats files, and why does it use them in the 2nd pass??
Or does it not use them in the 2nd pass, in which case, why are they left over then?



Quote:
Originally Posted by SassBot View Post
No it's not an integer since you can provide decimal value CRF values. Check your x264 logs and see the value it gives of the ratefactor after the first pass.
[..]
Then your GUI of choice is doing it wrong since CRF is not limited to integer values.
What's the precision of CRF's argument (n.xxxx)?? And in CL mode, can the user also provide such a precise value?
Also, how can I check the CRF value of an already encoded file? MediaInfo seems to only provide a rounded integer value.



Quote:
Originally Posted by SassBot View Post
It's highly doubtful that the "small difference" is even remotely visually perceptible.
[...]
You are not gaining any actual visual quality and are doing nothing but wasting your time
This admittance is what confuses me. I mean I do want to believe, and you make a good scientific argument, but then, you acknowledge the presence of differences that are "small" and that are "visually imperceptible". How can there be differences if it's CRF x CRF? Doesn't CRF = 2-pass mean that the 2 encodes should be identical even on the PSNR level?

Last edited by Makaveli84; 5th September 2012 at 11:19.
Makaveli84 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 23:43.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.