large difference in bitrate destribution of crf VS 2pass. Why?

[junglemike]

Hi. I recently started to play with constant quality 1 pass mode in x264. Overall I'm verysatisfied. But it's bitrate distribution is very different from 2-pass.
Here's an example. Will be glad if somebody can explain me why.
I've shot a test 1minute clip, 1st half i shoot moving trees, grass, etc, second half i shoot almost static picture (put camera on the table)

First I encoded video with Unrestricted 1 pass constant quality HQ, and chose crf 22
Second, I encoded same video with Unrestricted 2-pass HQ, and put exact size of the first encode to be desired file size.
Here is difference (quite dramatic) in bitrate distribution.
2 pass
http://img178.imageshack.us/img178/3991/2pass.png

crf
http://img259.imageshack.us/img259/1920/crf2.png

[Sharktooth]

you still dont get it...
1pass CRF and 2pass are different...

[junglemike]

you still dont get it...
1pass CRF and 2pass are different...

That wasn't very helpful. Of course i know they are different. I'm trying to understand how they are different in my application. Specifically why crf gives more rate for fast moving frames, and less rate for more static frames.

[Sharktooth]

coz CRF is one pass and the lookahead can "see" some frames ahead only. while 2 pass RC stats are collected during 1st pass so the 2nd pass has a whole stats file...

[Sagekilla]

It also has to do with the fact that during static scenes, you don't need a lot of bitrate for high quality. CRF = constant rate factor. It's not quite constant quality, but it's very close. To achieve that, you'd only need a few bits in low motion scenes and more in high motion.

[roozhou]

Try 1st pass CRF and 2nd pass bitrate, with b-adapt disabled in 2nd pass

[LoRd_MuldeR]

Try 1st pass CRF and 2nd pass bitrate, with b-adapt disabled in 2nd pass

"--b-adapt" doesn't have any effect in the second pass anyway, because frame-type decision happens in the first pass. Right?

[Quark.Fusion]

coz CRF is one pass and the lookahead can "see" some frames ahead only. while 2 pass RC stats are collected during 1st pass so the 2nd pass has a whole stats file...
I thought CRF isn't bitrate-based.

[Blue_MiSfit]

It's not, but you can use a CRF-based first pass as the basis for a traditional two pass encode. It actually works pretty well :)

~MiSfit

[Dark Shikari]

Dramatically different? Looks extremely similar to me; the small differences are likely accounted for by the fact that CRF measures complexity using AQ-weighted SATD while 2-pass uses bits.

Neither measure is strictly better.

Also, note two small differences in 2-pass mode, both resulting from the fact that 2-pass has a target bitrate and exacerbated by the fact that you used such a short clip:

1. The first pass likely did not use the same settings as the second (particularly psy-RD), so the bitrate distribution varies slightly in 2-pass from the ideal. This is a result of 2-pass thinking the bitrate is (higher|lower) than it should be and adapting accordingly. This is not a problem in long clips where ratecontrol can easily adapt (it might have a small effect in the first minute perhaps); it's moreso an issue in shorter cilps.

2. 2-pass has a bitrate spike at the end as it tries to use up its last few extra bits on something.

[vmrsss]

DS is spot on, as usual. If you look at the outline curve, they are remarkably similar, good evidence for 1-pass CRF.

Overall, I find it interesting that CRF gives more bits to the first part and fewer to the second, which from your description is how it should be.

[junglemike]

Dramatically different? Looks extremely similar to me;
Well, maybe not too dramatically, but still. 2pass uses roughly 1250 kbps for static frames, while crf uses below 1000kbps. On hi-action scenes 2-pass uses well below 2500 mark, while crf somewhat above 2500 mark.
Also, note two small differences in 2-pass mode, both resulting from the fact that 2-pass has a target bitrate and exacerbated by the fact that you used such a short clip:
Point taken, I will try to test with longer clips.
1. The first pass likely did not use the same settings as the second (particularly psy-RD),
in CRF i used: program --crf 18 --ref 5 --mixed-refs --bframes 3 --b-adapt 2 --b-pyramid --weightb --deblock -1:-1 --trellis 2 --partitions all --8x8dct --me umh --threads auto --thread-input --progress --no-psnr --no-ssim --output "output" "input"
in 2-pass I used: program --pass 2 --bitrate XXXX --stats ".stats" --ref 5 --mixed-refs --bframes 3 --b-adapt 2 --b-pyramid --weightb --direct auto --deblock -1:-1 --subme 7 --trellis 2 --partitions all --8x8dct --me umh --threads auto --thread-input --progress --no-psnr --no-ssim --output "output" "input"
Those are HQ profiles form Megui for 2-pass and Crf
2. 2-pass has a bitrate spike at the end as it tries to use up its last few extra bits on something.
This is me approaching to shut down the camera :)
Well, I have read someones remark about this issue on another forum, and started to "worry" a little bit. He said that crf tends to overallocate to action scenes, and underallocate to static/low quality scenes & fades.

[Dark Shikari]

Those are HQ profiles form Megui for 2-pass and CrfYou do know that MeGUI automatically lowers settings on the first pass, right?

... right?This is me approaching to shut down the camera :)Look again; there was a significant difference between the two at the end.
Well, I have read someones remark about this issue on another forum, and started to "worry" a little bit. He said that crf tends to overallocate to action scenes, and underallocate to static/low quality scenes & fades.Which is really silly, because the entire purpose of x264's ratecontrol is to underallocate to high motion scenes and overallocate to static scenes--and people have been complaining about how it goes too far in this regard.

[Sagekilla]

Dark, did you mean overallocate on high motion or underallocate for both?

[Dark Shikari]

Dark, did you mean overallocate on high motion or underallocate for both?Typo, fixed.

[nurbs]

Don't know what effect it has, but you didn't use the same --direct and --subme settings in the two encodes.

[Sagekilla]

Indeed, crf is running with --subme 6. It's not a massive change (like subme 1 -> 9) but subme does have a noticeable impact on bitrate required.

[vmrsss]

Which is really silly, because the entire purpose of x264's ratecontrol is to underallocate to high motion scenes and overallocate to static scenes--and people have been complaining about how it goes too far in this regard.

but that's not what we observe in these graphs... can the difference in --subme (6 vs 7) account for that?

[Dark Shikari]

but that's not what we observe in these graphs...Those graphs do not give enough information for you to say that, as there is no "constant quantizer" bitrate graph.