Estimating/Predicting CRF bitrate or file size [Archive]

Makaveli84

4th September 2012, 17:04

As the title says, is it possible to estimate beforehand the expected bitrate or file size of a given encode at a given CRF factor??
The reason I ask is because I would like to estimate the output size of an encode at CRF 20 for example, then actually encode the video in a size based 2-pass mode. I am not trying to keep the encodes below a certain file size, but rather, trying to maximize quality at the best file size possible. So encoding in 2-pass without a quality reference would be flying blindly. That's why I rely on a constant quality CRF pass to get the needed bitrate for a 2-pass encode to achieve a certain level of quality.
But as you can imagine, this is a very slow and time/resources consuming approach. This is why I'm looking for some kind of estimation method or quick virtual CRF pass that estimates the bitrate quickly without actually encoding the video.

Thanks.

P.S: Please, let the CRF vs 2-pass debate for another thread. To me, based on the testing that I've done, 2-pass produces slightly better (but pretty noticeable) quality than CRF at a given bitrate.

Groucho2004

4th September 2012, 17:10

As the title says, is it possible to estimate beforehand the expected bitrate or file size of a given encode at a given CRF factor??
No. The bitrate depends on the source (complexity, etc.).

The reason I ask is because I would like to estimate the output size of an encode at CRF 20 for example, then actually encode the video in a size based 2-pass mode.
That's what I do mostly. I run a compressibility pass with a certain CRF value on about 5%-10% of the source which gives me a pretty accurate number for the required bitrate.

Makaveli84

4th September 2012, 17:29

No. The bitrate depends on the source (complexity, etc.).
Yes, I know. I don't mean estimating in general. I am asking about estimating the bitrate of the output of a CRF n encode of a specific source, and not the bitrate of CRF n in general.

That's what I do mostly. I run a compressibility pass with a certain CRF value on about 5%-10% of the source which gives me a pretty accurate number for the required bitrate.
Hmm.. Does this really work? I mean the source has to be really monotone for this to work, which is hardly the case in real-life videos.

Groucho2004

4th September 2012, 17:41

Yes, I know. I don't mean estimating in general. I am asking about estimating the bitrate of the output of a CRF n encode of a specific source, and not the bitrate of CRF n in general.
I don't know how else to put it. You have to analyse each individual source in order to obtain the required bitrate for a given CRF value. There is no way to calculate it.

Hmm.. Does this really work? I mean the source has to be really monotone for this to work, which is hardly the case in real-life videos.
Obviously you have to use something like SelectRangeEvery(1000, 100) in Avisynth (select 100 out of every 1000 frames). It's quite accurate even with just 5%.

detmek

4th September 2012, 17:49

x264.exe --crf 20 --pass 1 --slow-firstpass --other options
x264.exe --bitrate (obtaind from the first pass) --pass 2 --other options
But the quality will be the same as the first pass.

Makaveli84

4th September 2012, 17:49

Obviously you have to use something like SelectRangeEvery(1000, 100) in Avisynth (select 100 out of every 1000 frames). It's quite accurate even with just 5%.

Can u elaborate further? I'm not very savvy in Avisynth. Can you suggest a sample script. Also, what do you feed the script into? I for example encode in Avidemux, so what I would need to do is feed the mentioned script into Avidemux instead of the actual video, right?

Groucho2004

4th September 2012, 18:01

Can u elaborate further? I'm not very savvy in Avisynth. Can you suggest a sample script. Also, what do you feed the script into? I for example encode in Avidemux, so what I would need to do is feed the mentioned script into Avidemux instead of the actual video, right?

If you are not familiar with Avisynth, a sample won't help you much. You simply put the SelectRangeEvery() line at the end of the script and feed it to x264.

Can't help you with GUIs, never used any, sorry.

Makaveli84

4th September 2012, 18:18

If you are not familiar with Avisynth, a sample won't help you much. You simply put the SelectRangeEvery() line at the end of the script and feed it to x264.

Can't help you with Gui's, never used any, sorry.

I am familiar with Avisynth, so a sample script would be of great help. For example, would it look something like this?
DirectShowSource("input.mkv", audio=false, fps=23.976)
SelectRangeEvery(1000, 100)
Do I need to add something else, assuming no filtering is to be applied on the source?

Makaveli84

4th September 2012, 18:27

x264.exe --crf 20 --pass 1 --slow-firstpass --other options
x264.exe --bitrate (obtaind from the first pass) --pass 2 --other options
But the quality will be the same as the first pass.

When you say --bitrate (obtaind from the first pass), is there anyway to get this automatically? If not, then this method beats the purpose of an overnight 2-pass encode. If I have to wait for the result of a full single CRF pass anyway, then I might as well use the bitrate in a full additional 2-pass overnight encode.

Groucho2004

4th September 2012, 18:38

For example, would it look something like this?
DirectShowSource("input.mkv", audio=false, fps=23.976)
SelectRangeEvery(1000, 100)
Yes, as simple as that.

J_Darnley

4th September 2012, 21:22

When you say --bitrate (obtaind from the first pass), is there anyway to get this automatically?

Yes. Parse the text printed by x264 for the bitrate value, use that value as the argument for bitrate in the second pass.

I don't know why you would run a slow first pass with crf and then (on average) get the same result again on the second pass.

detmek

4th September 2012, 21:38

I don't know why you would run a slow first pass with crf and then (on average) get the same result again on the second pass.
He said:

P.S: Please, let the CRF vs 2-pass debate for another thread. To me, based on the testing that I've done, 2-pass produces slightly better (but pretty noticeable) quality than CRF at a given bitrate.

SassBot

4th September 2012, 21:53

To me, based on the testing that I've done, 2-pass produces slightly better (but pretty noticeable) quality than CRF at a given bitrate.

Which isn't possibly true if you really did encode with CRF and 2-pass with the same settings and both were at the same bitrate. 2-pass uses the CRF algorithm to encode it just selects the appropriate CRF value for the bitrate you specified. You've basically tried to claim that CRF is inferior to itself which is a nonsensical statement.

SassBot

4th September 2012, 22:17

And you don't have to take my word for it. From Dark Shikari's mouth (http://doom10.org/index.php?topic=267.msg2071#msg2071):

CRF and 2-pass use identical bit allocation algorithms. All 2-pass does is pick the CRF value that gives the filesize you want. It's still using the CRF algorithm.

There is no way your CRF encode looked worse than a 2-pass unless you didn't use equivalent settings or a CRF value that matched the bitrate you chose for 2-pass. If it did produce a noticeably different result despite everything being exactly the same between the two encodes you have uncovered a bug.

Groucho2004

4th September 2012, 22:28

There is no way your CRF encode looked worse than a 2-pass unless you didn't use equivalent settings or a CRF value that matched the bitrate you chose for 2-pass.

He's using a "GUI" so that might explain different settings. I can imagine that these programs tend to hide certain settings from the user because the programmer thinks that an inexperienced user should not touch them. I don't know avidemux so this is just a hypothesis.

SassBot

4th September 2012, 22:32

I don't know why you would run a slow first pass with crf and then (on average) get the same result again on the second pass.

Maybe they needs to keep their house warm? This is exactly correct. You are basically rerunning it to get what will (barring any options you select that result in non-deterministic behavior) get exactly the same result except you've just wasted extra hours doing so. It's amazing that no matter how many times this gets corrected people are still under the misguided notion that CRF and 2-pass are somehow fundamentally different.

He's using a "GUI" so that might explain different settings. I can imagine that these programs tend to hide certain settings from the user because the programmer thinks that an inexperienced user should not touch them. I don't know avidemux so this is just a hypothesis.

I guess that's possible. Either that or he's just convinced himself that he sees a difference where there isn't one which is not uncommon.

Makaveli84

5th September 2012, 00:43

Which isn't possibly true if you really did encode with CRF and 2-pass with the same settings and both were at the same bitrate. 2-pass uses the CRF algorithm to encode it just selects the appropriate CRF value for the bitrate you specified. You've basically tried to claim that CRF is inferior to itself which is a nonsensical statement.

And you don't have to take my word for it. From Dark Shikari's mouth (http://doom10.org/index.php?topic=267.msg2071#msg2071):

There is no way your CRF encode looked worse than a 2-pass unless you didn't use equivalent settings or a CRF value that matched the bitrate you chose for 2-pass. If it did produce a noticeably different result despite everything being exactly the same between the two encodes you have uncovered a bug.

So much for my plead to leave the CRF x 2-pass discussion out of this thread :rolleyes:.
Well, anyways Mr. Sass, I am very much aware of the point you made, having read extensively about this matter. Actually, I had even spent some good time reading the thread you quoted Dark Shikari's post from. Nevertheless, it seems a lot of people are still unconvinced with your claims. I for one am neither convinced nor unconvinced. I am skeptical, and I am running trials to find out myself.
Which brings me to my second point: both my CRF and 2-pass encodes were done with the same settings. I am using a GUI, but I am not aware of any settings it's hiding. So as far as I can tell, both encodes have the exact same settings.

Either that or he's just convinced himself that he sees a difference where there isn't one which is not uncommon.
LOL. Typical arrogance form the "know-it-all"s!!
Anyways, I've ran an ABX test (15 trials), and I scored right 15/15, so trust me, I see a difference.

Anyways, I am willing to accept your claims if you care to explain them further. If 2-pass and CRF are the same, does that mean that the first pass of a 2-pass encode is simply running to determine the "n" needed for the given bitrate or size, and the 2nd pass is nothing more than CRF n ??
Also, what's the precision of "n"?? If it's an integer, then your whole claim is nonsensical. If it's a single float, then it might hold some truth to it, though that would still be inaccurate to hit a given target bitrate. So what is it? CRF n or n.x or n.xy or what?
In the GUI I use, I can only specify integer as CRF's argument. And before you even go there, this has nothing to do with my trial. I encoded in CRF first, and then used the output's bitrate in 2-pass, and not the other way around.

Makaveli84

5th September 2012, 01:06

Also, Sass, this from the same thread you quoted:
Q: Is --direct auto a little more efficient in 2 pass mode than crf? If there's a difference, my guess would be that it would be fairly minor, am I correct?
A: Yes, 2-pass results in a better --direct auto, but that's a tiny quality-difference.

And this from this current thread:
I don't know why you would run a slow first pass with crf and then (on average) get the same result again on the second pass.

Which seem to suggest that there is a slight advantage, which no matter how slight it is, contradicts the scientific principle you're proposing which suggests the exact same output because it's the same CRF n pass done twice. I'm willing to be enlightened on this matter too. Please, oh please, do convince me, because I would rather use CRF than 2-pass.

SassBot

5th September 2012, 04:18

So much for my plead to leave the CRF x 2-pass discussion out of this thread :rolleyes:.
Well, anyways Mr. Sass, I am very much aware of the point you made, having read extensively about this matter. Actually, I had even spent some good time reading the thread you quoted Dark Shikari's post from. Nevertheless, it seems a lot of people are still unconvinced with your claims. I for one am neither convinced nor unconvinced. I am skeptical, and I am running trials to find out myself

People are also convinced that $100 Monster Cable HDMI cables provide a difference and yet they don't. People routinely convince themselves of truths that don't actually pan out.

Which brings me to my second point: both my CRF and 2-pass encodes were done with the same settings. I am using a GUI, but I am not aware of any settings it's hiding. So as far as I can tell, both encodes have the exact same settings.

Then if you saw differences you encountered a bug since they USE THE EXACT SAME ALGORITHM. 2-pass is just a way for x264 to guess the CRF value for you based on the bitrate you specify.

LOL. Typical arrogance form the "know-it-all"s!!
Anyways, I've ran an ABX test (15 trials), and I scored right 15/15, so trust me, I see a difference.

Okay, then back this up with your settings, your test clips and the encoded clips.

Anyways, I am willing to accept your claims if you care to explain them further. If 2-pass and CRF are the same, does that mean that the first pass of a 2-pass encode is simply running to determine the "n" needed for the given bitrate or size, and the 2nd pass is nothing more than CRF n ??

Yes, that is exactly what the analysis pass does. It calculates the CRF value to use for the second pass based on running a first pass with your specified bitrate. How was that not clear from the quote I gave you from Dark Shikari?

Also, what's the precision of "n"?? If it's an integer, then your whole claim is nonsensical. If it's a single float, then it might hold some truth to it, though that would still be inaccurate to hit a given target bitrate. So what is it? CRF n or n.x or n.xy or what?

No it's not an integer since you can provide decimal value CRF values. Check your x264 logs and see the value it gives of the ratefactor after the first pass.

In the GUI I use, I can only specify integer as CRF's argument. And before you even go there, this has nothing to do with my trial. I encoded in CRF first, and then used the output's bitrate in 2-pass, and not the other way around.

Then your GUI of choice is doing it wrong since CRF is not limited to integer values.

Also, Sass, this from the same thread you quoted:

It's highly doubtful that the "small difference" is even remotely visually perceptible.

And this from this current thread:

Which seem to suggest that there is a slight advantage, which no matter how slight it is, contradicts the scientific principle you're proposing which suggests the exact same output because it's the same CRF n pass done twice. I'm willing to be enlightened on this matter too. Please, oh please, do convince me, because I would rather use CRF than 2-pass.

No, there really isn't. You are not gaining any actual visual quality and are doing nothing but wasting your time, but it's not my electricity bill.

Makaveli84

5th September 2012, 11:15

Then if you saw differences you encountered a bug since they USE THE EXACT SAME ALGORITHM. 2-pass is just a way for x264 to guess the CRF value for you based on the bitrate you specify.
Regardless of what I saw, since they use the EXACT SAME ALGORITHM, there shouldn't be a difference even in still frames (paused video) for example, because we're talking about the same algorithm, and not same "perceptible" quality and/or psychological optimizations, right?
This is not "CRF gives same quality as CQ", this is "CRF = 2-pass because 2nd pass is CRF", and thus, one shouldn't see a difference even in a still frame, right?

Yes, that is exactly what the analysis pass does. It calculates the CRF value to use for the second pass based on running a first pass with your specified bitrate. How was that not clear from the quote I gave you from Dark Shikari?
Clarity is not the issue, I am not asking per say, but rather double checking or reconfirming.. But here's what I still don't get: if all the 1st pass is doing is determining the CRF argument to be used in the 2nd pass, why does it create such large (relatively speaking) .stats and .mbtreestats files, and why does it use them in the 2nd pass??
Or does it not use them in the 2nd pass, in which case, why are they left over then?

No it's not an integer since you can provide decimal value CRF values. Check your x264 logs and see the value it gives of the ratefactor after the first pass.
[..]
Then your GUI of choice is doing it wrong since CRF is not limited to integer values.
What's the precision of CRF's argument (n.xxxx)?? And in CL mode, can the user also provide such a precise value?
Also, how can I check the CRF value of an already encoded file? MediaInfo seems to only provide a rounded integer value.

It's highly doubtful that the "small difference" is even remotely visually perceptible.
[...]
You are not gaining any actual visual quality and are doing nothing but wasting your time
This admittance is what confuses me. I mean I do want to believe, and you make a good scientific argument, but then, you acknowledge the presence of differences that are "small" and that are "visually imperceptible". How can there be differences if it's CRF x CRF? Doesn't CRF = 2-pass mean that the 2 encodes should be identical even on the PSNR level?

Groucho2004

5th September 2012, 11:24

Makaveli84

5th September 2012, 11:41

By the way, here (http://akuvian.org/src/x264/ratecontrol.txt) is a paper from Loren Merrit about x264 rate control. I don't know how old it is but I suppose it still applies to the current implementation.

Edit:
Just checked, apparently it's from 2006.

Well, that document shows that the 2nd pass of a 2-pass encode is not in fact a simple CRF single pass.. :confused:

detmek

5th September 2012, 14:30

CRF and 2pass mod do not produce bit-identical result. But the visual quality is the same when you watch video on your TV or monitor. If you do frame-by-frame comparisson you will notice the difference. One of the reasons for the difference is that, even in 2pass mod, x264 will not hit the exact bitrate. So, if your CRF 20 produces file with 3423kbs average bitrate and you use that bitrate in 2pass mod, you will not get video with average bitrate of 3423kbs but deviation will be less then 1%.
Direct auto is another option that makes a difference as it works a little bit different i 2pass mod.
But overall, difference in quality between CRF and coresponding 2pass mod will be less then 2%. And in my two tests 2pass gives 0-2% worse quality compared to CRF. I used SSIM to measure differences.

SassBot

5th September 2012, 14:56

Once again I will go back to Dark Shikari this time from another thread (http://forum.doom9.org/showthread.php?t=143904):

If I want the highest quality encodes without time limit, is CRF better or 2 pass? Suppose I encode a video using CRF x, and it output a video that is 800MB, now I use 2 pass, and specify the size to 800, which way of encoding would provide higher quality?

Neither.
Oops, let me change that to which way of encoding between the 2 would create a better encode?
Neither.
So they're the same? 2 pass is only to hit a specific file size? I thought the first pass is to analyze the video file to see where to distribute the bits or something like that.
CRF, 1pass, and 2pass all use the same bit distribution algorithm. 2-pass tries to approximate CRF by using the information from the first pass to decide on a constant quality factor. 1-pass tries to approximate CRF by guessing a quality factor over time and varying it to reach the target bitrate.

So if you don't want to believe Dark Shikari and waste your time doing 2-pass encodes with needless slow first passes for pretty much no gain in quality go ahead, but don't continue to spread misleading statements that then constantly have to be corrected. 2-pass is purely for wanting to hit specific sizes for your encodes as 1-pass CRF is for when you do not care about final size. 2-pass does not give you magical better quality. As detmek says the streams will not be bit-identical, which I didn't say they would be, but you will not see any perceptible quality difference between them assuming you have the exact same settings and they are equivalent size. If you are seeing noticeable differences even under those constraints then you need to report it as it is most likely a bug.

SassBot

5th September 2012, 15:15

To drive this home more:

CRF and 2-pass use the exact same bit distribution algorithm. If you do a CRF encode and a 2-pass encode at the same bitrate, the results will be nearly identical.

Hypothetically, there could be a bug in the 2-pass code that didn't show up in CRF, but outside of a hypothetical bug workaround, this is probably not a good suggestion.

Edit to add: Oops, that first post was about Xvid. Removed it.

detmek

5th September 2012, 16:06

CRF and 2-pass use the exact same bit distribution algorithm. If you do a CRF encode and a 2-pass encode at the same bitrate, the results will be nearly identical.

Hypothetically, there could be a bug in the 2-pass code that didn't show up in CRF, but outside of a hypothetical bug workaround, this is probably not a good suggestion.
Nearly identical. Main difference comes from the fact that you can not hit exact CRF with 2-pass mod.
So, in theory CRF and 2-pass are the same. In practice, there is a small difference in obtaind video. It it visible? If you compare frame-by-frame both files, yes.
Just try this:
Encode 1:
x264.exe --crf xx --preset xxxxx --tune ssim --ssim
Encode 2:
x264.exe --bitrate (from CRF encode) --pass 1 --tune ssim --ssim
x264.exe --bitrate (form CRF encode) --pass 2 --tune ssim --ssim
Compare bitrates, SSIM and images.

SassBot

5th September 2012, 18:02

Yes, "nearly identical" meaning there are differences that are so minor that no one will notice them. You are attempting to imply that Dark Shikari is stating more than he really did. Especially since I even posted him saying that neither 2-pass or CRF give better quality than the other repeatedly.

It it visible?

To most people who don't obsessively frame-by-frame and zoom in the video to spot microscopic differences? No. If I use Avisynth to subtract the frames, then enhance and zoom to try to spot differences? Maybe.

If you compare frame-by-frame both files, yes.

Which is a ridiculous standard to use and I'd even question that. Last night did a number of CRF vs 2-pass encodes to test and even single-framing there were almost no visible differences and they only really stood out when I zoomed in and over-enhanced the video to make any artifacts stand out. This also ignores the fact that Makaveli84 was stating it was visible during playback not when single framing which is highly unlikely outside of bugs.

akupenguin

5th September 2012, 18:24

if all the 1st pass is doing is determining the CRF argument to be used in the 2nd pass, why does it create such large (relatively speaking) .stats and .mbtreestats files, and why does it use them in the 2nd pass??
So that it doesn't have to recompute the frame type and mbtree decisions. 2nd pass would make exactly the same decisions, so there's no point in wasting cpu-time doing them all over again when you can just record the outcomes from the 1st pass. Also, you need data about the complexity of each frame in order to do VBV if that's enabled.

And the prediction of what CRF value would result in what filesize is imperfect, so 2pass adjusts the CRF slightly after observing any such misprediction. This again requires knowledge of what the bitrate distribution should have been on a per-frame level, not just the complexity of the movie as a whole. Incidentally, any such adjustment is a (very slight) detriment to the quality-per-bitrate of 2pass, but it's necessary in order to meet the target filesize.

What's the precision of CRF's argument (n.xxxx)?
Floating-point, so you'll get non-identical results out to 7 significant digits. That said, some of the other computations that involve QP are performed in fixed-point, so you might consider the meaningful precision to be .001

Makaveli84

5th September 2012, 18:50

Sass, what about applying sharpness filters, slowing down effects, etc?? Does that not enhance the artifacts? What if I'd be making a sports video edit that requires slowing down a certain sequence and/or freezing a certain frame for effect or whatever.. I mean sometimes, one would care about even the smallest artifacts that are not visible at regular playback.
For example, the initial difference I was talking about was initially spotted with a moderate sharpness filter applied in MPC-HC playback on both encodes. With filtering off, I was able to spot the difference in freeze-frame. From then on, I was able to tell the difference at regular playback in that sequence with sole focus on the "artifacts area". That's because my eye was by then trained to catch that difference. But this is the whole point of blind testing. You can train as much as you want.
Again, this is very minor, and I'm sorry if my initial post implied that the difference was major. I only wanted to make the point that it was noticeable, but maybe I should've explained in more details.
Anyways, CRF is more than suitable most of the time, but in some cases, when editing and applying effects on videos, I'd like to achieve the best possible quality, even for single frames.
But even for that, I am unconvinced that 2-pass always yields better quality. I mean all that was posted here points to the fact that since 2-pass supposedly uses CRF for its 2nd pass, then there should not be even single-frame differences. However, CRF proponents always say that there is a slight difference that is not worth the extra time. Well, that should be subjective and up to the individual if it's worth it or not. But more importantly, detmek and Sass seem to be at odds regarding whether the 1% minimal difference affects single frame by frame comparisons or not. detmek says yes, Sass says no. Which is it? If 2nd pass is merely a single CRF pass that is not bit identical to CRF because of the impossible task of hitting an exact CRF or bitrate, then one would not expect this very minimal difference to be visible in frame by frame comparisons.
Finally, as an additional test, would encoding 2-pass first, then reading the obtained CRF form x264 logs and then encoding in CRF with the obtained value, would such a test produce bit-identical streams or at least a frame-by-frame match?

Groucho2004

5th September 2012, 19:00

Anyways, CRF is more than suitable most of the time, but in some cases, when editing and applying effects on videos, I'd like to achieve the best possible quality, even for single frames.
But even for that, I am unconvinced that 2-pass always yields better quality. I mean all that was posted here points to the fact that since 2-pass supposedly uses CRF for its 2nd pass, then there should not be even single-frame differences. However, CRF proponents always say that there is a slight difference that is not worth the extra time. Well, that should be subjective and up to the individual if it's worth it or not. But more importantly, detmek and Sass seem to be at odds regarding whether the 1% minimal difference affects single frame by frame comparisons or not. detmek says yes, Sass says no. Which is it? If 2nd pass is merely a single CRF pass that is not bit identical to CRF because of the impossible task of hitting an exact CRF or bitrate, then one would not expect this very minimal difference to be visible in frame by frame comparisons.
Finally, as an additional test, would encoding 2-pass first, then reading the obtained CRF form x264 logs and then encoding in CRF with the obtained value, would such a test produce bit-identical streams or at least a frame-by-frame match?
You should just read akupenguin's post. I think it makes the difference between CRF and 2-pass very clear.

Makaveli84

5th September 2012, 19:04

So that it doesn't have to recompute the frame type and mbtree decisions. 2nd pass would make exactly the same decisions, so there's no point in wasting cpu-time doing them all over again when you can just record the outcomes from the 1st pass. Also, you need data about the complexity of each frame in order to do VBV if that's enabled.

And the prediction of what CRF value would result in what filesize is imperfect, so 2pass adjusts the CRF slightly after observing any such misprediction. This again requires knowledge of what the bitrate distribution should have been on a per-frame level, not just the complexity of the movie as a whole. Incidentally, any such adjustment is a (very slight) detriment to the quality-per-bitrate of 2pass, but it's necessary in order to meet the target filesize.

Floating-point, so you'll get non-identical results out to 7 significant digits. That said, some of the other computations that involve QP are performed in fixed-point, so you might consider the meaningful precision to be .001

First, thanks for your insights. I mean explaining in a way that an average user can understand instead of simply saying "take my word for it" always goes a long way in convincing such user.
Interestingly enough, your post not only implies that there is a very slight difference between 2-pass and CRF, but also that this difference is in CRF's favor. Am I correct?

Makaveli84

5th September 2012, 19:08

You should just read akupenguin's post. I think it makes the difference between CRF and 2-pass very clear.
I just did.. But you personally said initially that you run a compressibility test before encoding in 2-pass mode. I'm a bit puzzled since you seem to be convinced that there is no need for that, yet you do it anyways?

Groucho2004

5th September 2012, 19:52

I just did.. But you personally said initially that you run a compressibility test before encoding in 2-pass mode. I'm a bit puzzled since you seem to be convinced that there is no need for that, yet you do it anyways?

I like to backup 2-3 movies to one BD-R. I run the compressibility test in order to get a rough idea of the required bitrate for each movie. Once I know that, I can combine the movies so they fit nicely on the medium.

sneaker_ger

5th September 2012, 20:10

Sass, what about applying sharpness filters, slowing down effects, etc?? Does that not enhance the artifacts? What if I'd be making a sports video edit that requires slowing down a certain sequence and/or freezing a certain frame for effect or whatever.. I mean sometimes, one would care about even the smallest artifacts that are not visible at regular playback.
For example, the initial difference I was talking about was initially spotted with a moderate sharpness filter applied in MPC-HC playback on both encodes. With filtering off, I was able to spot the difference in freeze-frame. From then on, I was able to tell the difference at regular playback in that sequence with sole focus on the "artifacts area". That's because my eye was by then trained to catch that difference. But this is the whole point of blind testing. You can train as much as you want.
Again, this is very minor, and I'm sorry if my initial post implied that the difference was major. I only wanted to make the point that it was noticeable, but maybe I should've explained in more details.
Anyways, CRF is more than suitable most of the time, but in some cases, when editing and applying effects on videos, I'd like to achieve the best possible quality, even for single frames.
But even for that, I am unconvinced that 2-pass always yields better quality. I mean all that was posted here points to the fact that since 2-pass supposedly uses CRF for its 2nd pass, then there should not be even single-frame differences. However, CRF proponents always say that there is a slight difference that is not worth the extra time. Well, that should be subjective and up to the individual if it's worth it or not.

Not so fast. No one said that 2pass looks better than CRF, only that they are not bit-identical. The devs seem to suggest that CRF actually has a slight advantage, so 2pass "not being worth the extra time" is kind of an understatement.
Also: the time that goes into a second pass could be spend in better encoding or filtering, further increasing CRF's advantage.

Makaveli84

5th September 2012, 22:22

Not so fast. No one said that 2pass looks better than CRF, only that they are not bit-identical. The devs seem to suggest that CRF actually has a slight advantage, so 2pass "not being worth the extra time" is kind of an understatement.
Also: the time that goes into a second pass could be spend in better encoding or filtering, further increasing CRF's advantage.
Not so fast. I know. That's why I said:
"But even for that, I am unconvinced that 2-pass always yields better quality. I mean all that was posted here points to the fact that since 2-pass supposedly uses CRF for its 2nd pass, then there should not be even single-frame differences."

Which is clearly stating that, although I still have some unanswered questions, I am doubting my initial hypothesis. I misspoke and said "always", when I meant to say "constantly", as in during a single encode. I still stand by the difference I saw, but maybe that was the exception in that video. Maybe that sequence or bunch of frames were the exception, and if I frame-by-frame compare both encodes, I'd find that the CRF encode is as good or better than the 2-pass one. But that would be almost impossible for a 2 hours long movie, so I might just accept and believe what seems to be the majority of posters that CRF is indeed enough.
As for the unsatisfactory CRF encoding I occasionally run into (that I thought can be bettered by 2-pass encoding at same bitrate), I think I have 2 options:

a) Increasing CRF rate, which was always a pain in the ass for me since Avidemux only allows integer increments, and that usually meant a 100 MB/movie hike on average. But now that I know I can specify more precise arguments for CRF, switching to another GUI or even CL will probably fix that.

b) Slower and better CRF encoding as sneaker_ger suggested, and I would welcome any suggestions on what settings to tweak. I am already using:

cabac=1 / ref=5 / me=umh / subme=9 / me_range=32 / psy_rd=1:0 / trellis=2 / bframes=3 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / open_gop=0 / scenecut=40 / intra_refresh=0 / rc_lookahead=60 / mbtree=1

Which are settings that already tend to favor quality over encoding time and file size. So any additional suggestions??

Audionut

6th September 2012, 00:09

so I might just accept and believe what seems to be the majority of posters that CRF is indeed enough.

I tend not to believe anyone until I see results myself. But when the devs say "this is how it works", well, it's time to listen.
It's been said a million times, but I'll say it again, use CRF unless you need to hit a specific size.

Which are settings that already tend to favor quality over encoding time and file size. So any additional suggestions??

--preset veryslow --merange 32

SassBot

7th September 2012, 15:43

I just did.. But you personally said initially that you run a compressibility test before encoding in 2-pass mode. I'm a bit puzzled since you seem to be convinced that there is no need for that, yet you do it anyways?

As has been said before and I repeated, 2-pass is for hitting a deserved file size and nothing more. It will not give better quality. If you actually see noticeably worse quality between a 2-pass or CRF encode you are seeing a bug that should be reported.

I tend not to believe anyone until I see results myself. But when the devs say "this is how it works", well, it's time to listen.
It's been said a million times, but I'll say it again, use CRF unless you need to hit a specific size.

The main issue is the fact that it's been stated over and over and over many times that 2-pass and CRF do not either give you more quality. This is why people such as myself get frustrated since this questioned has repeatedly been answered. Also, even though he doesn't mean to intentionally, when he makes claims that 2-pass gives better quality than CRF when this is not possible outside of bugs it does nothing but further this misunderstanding. People will google 2-pass vs CRF come into the thread and see that statement and if it is not corrected they will then become misinformed.

Makaveli, I will apologize if I came across as rude and flippant in my posts. Misinformation like this continues to spread and confuse newbies so many people tend to get frustrated when people continue to state misleading facts even though I know you weren't meaning to mislead anyone by your statement. It's like when people come here asking about how to make their command string better and it's some weird string of redundant and sometimes contradictory commands that they've cobbled together by trying to turn ever knob or because they saw someone else using something similar. People misunderstand x264 enough as it is without long since debunked myths continuing to live on.

hello_hello

15th September 2012, 23:57

My 2 cents worth......

I was involved in a similar 2 pass vs CRF discussion in another forum recently. I ran a couple CRF encodes and used the resulting bitrate for 2 passes. The first time I managed to get it wrong and specify the wrong bitrate but they were still only 30kbps different over a 2 and a half hour movie and 2 pass achieved the exact bitrate I requested. The second time I didn't mess up (started with a different CRF value the second time) and the bitrates were identical.

I ran the encodes side by side and under normal "playing" conditions, and even sitting close to the TV, CRF and 2 pass are identical. I also tried, at the suggestion of another poster, using ffdshow's on screen display to show the input bitrates as the videos progressed. Someone mentioned earlier in this thread that 2 pass makes slight adjustments as it progresses in order to achieve the requested bitrate and if the input bitrate being displayed by ffdshow is anything to go by, that's exactly what happens. I watched a large section of the video and for the first few minutes the bitrate being displayed by the 2 pass encode sat very constantly at around 30Kbps more than CRF. Then it got to a more complex section of the video where the bitrates climbed and fairly quickly the difference lessened until the bitrates were the same, then CRF took the lead by a small margin for a while.
I did notice that right at the end of the video (last minute or so of credits) the bitrate for the 2 pass encode climbed quite high compared to CRF. I guess it was playing catch-up trying to hit the requested bitrate before the video ended.

Comparing CRF and 2 pass frame by frame and looking for differences, there was no doubt in my mind no two frames were ever compressed the same way. Most of the time the differences were very small and if I added the source to the mix and compared all three it seemed to me any differences between 2 pass and CRF were just that... differences. There's no way I would say 2 pass or CRF compressed the frame "better" or even more accurately. Just different. I could at times say "well for this section of the frame 2 pass was more faithful to the source, over there CRF wins" etc, but that's frame by frame comparing sitting close to the TV and in terms of quality they're still the same.

Well there were a couple of exceptions.... in a scene or two with slight camera motion and lots of detail throughout the entire frame. For example on one occasion I stopped on a quite detailed scene and noticed a distinctive "halo" around an object in the CRF encoded version which just wasn't there in the 2 pass version, but moving both encodes on by a single frame the effect was reversed, and playing the two at normal speed they looked the same.

When it came to comparing 2 pass and CRF at lower qualities the difference in the way individual frames are compressed becomes more obvious but I still didn't see any quality differences as such. As the bitrate/quality drops they both achieve an equal flavour of "average", or "bad" and I'd guess as the differences become more apparent it might be easier to compare frames and say 2 pass was way better here, CRF was way better there, but combine frames at normal playing speed and even at low quality 2 pass and CRF don't seem to have actual "quality" differences to me.

My tentative conclusion, given 2 pass and CRF don't seem to always be using the same bitrate at any given time (according to ffdshow), it's the reason for minor differences in compression frame by frame, but given 2 pass is (for want of a better way to put it) constantly playing catchup compared to CRF then if anything, in theory it'd be the method with the less "consistent" quality.

I guess I should qualify all the above my saying one comparison was done using a DVD source and anamorphic encoding without resizing (just cropping the black bars etc). The second using a HD source resized to a SD lossless AVI which I then re-encoded. I ran the encodes full screen on a 51" Plasma to look for differences frame by frame (MPC-HC using the same resizing method each time). One day I might do the same again while encoding at a higher definition, but I can't imagine anything changing.

hello_hello

15th September 2012, 23:58

One question as I've not found the answer to it anywhere....

I've been told several times x264 doesn't report the CRF value used for a 2 pass encode so there's no way to know, but MeGUI's log file reports a "final ratefactor" after a 2 pass encode which seems suspiciously like a CRF value to me. For example when I ran the CRF 18 encode and used the resulting bitrate for 2 passes (it's the one where I got the bitrate slightly wrong) the final ratefactor reported after 2 passes was 17.68, and when the bitrates matched exactly after comparing 2 pass to CRF 20 the final ratefactor reported was 19.79. Anyone know what "final ratefactor" is exactly and if it can be taking as a pretty good guide to the CRF value used for 2 passes as it appears?

Groucho2004

16th September 2012, 00:15