Final Ratefactor in 2-pass ABR mode (x264) AKA fast 1st pass

[HeadBangeR77]

Just one question and a few explanations:

I used to run 2-passes with exactly the same settings for both of them. Since my old Athlon XP-M, even nicely oc'ed, runs only a few fps with the latest x264 revision, and the AQ which I use slows down the encoding further, I decided to make the first pass faster (manually). For instance:
ME 5, Hexagonal search precision, 2 reference frames - for the 1st
ME 6(RDO), UMHex (range 16) precision, 3 reference frames - for the 2nd.

I'm just testing some samples with exactly the same settings, apart from the first pass, to see if there are any significant differences in quality (metric-wise, probably invisible while watching or even analysing still frames).

(???) I would only like to know, if the final ratefactor established during the first pass (and higher for the fast first pass, which is pretty obvious), gets somehow corrected in the second run? My initial observations are, that there's only a very insignificant difference in the final metrics and average quantizer, while the speed of the first pass is much better (it's a trade-off for me, concerning my current, "ancient" hardware).

I would be grateful for any answers and suggestions.

cheers
HDBR77

PS. I will give the results, when the samples are ready.

[Terranigma]

I'm just testing some samples with exactly the same settings, apart from the first pass, to see if there are any significant differences in quality (metric-wise, probably invisible while watching or even analysing still frames).

There'll be a difference, sure, because it's not completely accurate anymore, but there should'nt be a significant change in the final outcome.

If you want, you can even set partitions/analyse to none, trellis to 0, disable AQ, and subme even at level 1 for the first pass for an even speedier pass, but note that the more you alter, the less accurate your final outcome will become.

I take it that you use avs scripts to slow your encoding? If so, why not disable those slow filters in your .avs file for the first pass using the same settings for both passes, then enable the slow filters for the second?

[microchip8]

Like Terranigma said, lowering settings for the first pass will affect the final outcome but unless you're an alien, you won't be able to spot it. Even the turbo=2 option decreases the PSNR with 0.05 dB (too low for me to tell the difference)

[HeadBangeR77]

Thanks, guys, for some quick replies. :)

Ooops, surprise this time! :confused:

1) No problems with hitting the target bitrate (5080kbps, grainy and high motion sample @ 1024x432)
ME 6(RDO), Hexagonal search precision, 3 reference frames - for the 1st
ME 6(RDO), UMHex (range 16) precision, 3 reference frames - for the 2nd.

2) In this case no problems either
ME 5, Hexagonal search precision, 3 reference frames - for the 1st
ME 6(RDO), UMHex (range 16) precision, 3 reference frames - for the 2nd.

3)The first time in my life when x264 has missed the target bitrate
ME 5, Hexagonal search precision, 2 reference frames - for the 1st
ME 6(RDO), UMHex (range 16) precision, 3 reference frames - for the 2nd.

Extra (0) ;) Settings as in No 1, first pass at CRF 18.

Could it be due to the difference in reference frames? I see no other explanation, yet my experience with x264 is limited.

Ad.3)
final ratefactor: 19.06
SSIM Mean Y:0.9860396
PSNR Mean Y:46.297 U:47.910 V:48.880 Avg:46.864 Global:46.537 kb/s:5114.67
encoded 6986 frames, 5.17 fps, 5115.31 kb/s
SSIM Mean Y:0.9853807
PSNR Mean Y:46.351 U:48.073 V:49.020 Avg:46.947 Global:46.678 kb/s:5111.31
encoded 6986 frames, 3.12 fps, 5111.95 kb/s

Ad.2)
final ratefactor: 19.03
SSIM Mean Y:0.9860381
PSNR Mean Y:46.292 U:47.913 V:48.882 Avg:46.862 Global:46.537 kb/s:5115.01
encoded 6986 frames, 4.73 fps, 5115.65 kb/s
SSIM Mean Y:0.9854263
PSNR Mean Y:46.318 U:48.055 V:49.003 Avg:46.917 Global:46.734 kb/s:5082.49
encoded 6986 frames, 3.47 fps, 5083.13 kb/s

Ad.1)
final ratefactor: 18.16
SSIM Mean Y:0.9853462
PSNR Mean Y:46.349 U:48.058 V:49.012 Avg:46.942 Global:46.667 kb/s:5111.42
encoded 6986 frames, 3.79 fps, 5112.06 kb/s
SSIM Mean Y:0.9854316
PSNR Mean Y:46.316 U:48.052 V:48.999 Avg:46.915 Global:46.736 kb/s:5081.38
encoded 6986 frames, 3.47 fps, 5082.02 kb/s

Ad.0)
1st pass @ CRF 18
SSIM Mean Y:0.9861723
PSNR Mean Y:46.581 U:48.198 V:49.139 Avg:47.152 Global:46.931 kb/s:5404.22
encoded 6986 frames, 3.68 fps, 5404.86 kb/s
SSIM Mean Y:0.9854337
PSNR Mean Y:46.317 U:48.051 V:48.999 Avg:46.916 Global:46.737 kb/s:5081.13
encoded 6986 frames, 3.45 fps, 5081.77 kb/s

PS. There are no slow scripts used while making samples (the final one for the whole movie is indeed slow): just mpegsource, trimming, cropping and resizing with Lanczos3.

PS. Comments underneath ;)

[Terranigma]

It has to be the 1 ref frames difference. IMO, you should never use less than 5, and going much higher, such as 16, would yield only a minor difference (Just check the ssim/psnr between 5 & 16 reference frames), but a huge difference in the cpu usage, and the encode time will drop significantly. Besides Motion Estimation, the number of reference frames dramatically affects the encoding speed, but with what gain, only a small difference in quality?

Stick with using the same reference frames for alternating passes, the rest is probably fair game (well, except maybe cabac/cavlc since the compression difference between the 2 is huge).

[HeadBangeR77]

It has to be the 1 ref frames difference. IMO, you should never use less than 5, and going much higher, such as 16, would yield only a minor difference (Just check the ssim/psnr between 5 & 16 reference frames), but a huge difference in the cpu usage, and the encode time will drop significantly. Besides Motion Estimation, the number of reference frames dramatically affects the encoding speed, but with what gain, only a small difference in quality?
Yes, that's what I though either. I know reference frames make a huge impact on encoding speed, and help compression, though I think 3 references for high quality high bitrate encode should be enough (?). Looking at the percentages of references in my logs I can't see any large gain from using more than 5, maybe even 4. On the other hand the scene is specific, so there could be a higher compression gain from 5 vs. 3 references throughout the whole film, I assume.

Stick with using the same reference frames for alternating passes, the rest is probably fair game (well, except maybe cabac/cavlc since the compression difference between the 2 is huge).
Thanks for the tips - I have to struggle with my hardware to get a good balance between speed and quality, though the latter is much more important to me. ;)

PS. I you looked at my updated data, you would notice:
- slightly better metrics, probably impossible to spot with my eyes,
- that however at lower bitrate, almost exactly the desired one,
- faster 2nd pass, since rate control didn't have to sweat that much.

[Terranigma]

Yes, that's what I though either. I know reference frames make a huge impact on encoding speed, and help compression, though I think 3 references for high quality high bitrate encode should be enough (?). Looking at the percentages of references in my logs I can't see any large gain from using more than 5, maybe even 4. On the other hand the scene is specific, so there could be a higher compression gain from 5 vs. 3 references throughout the whole film, I assume.

Higher compression gain, sure, but still, probably by only a small margin. Since you're ok with using just 3 reference frames, compare it with 5 reference frames and check the final ratefactor between the 2.
(something with some relevance)
The other day I did a test using 16 reference frames with trellis set to 0, and then did another test, this time using 5 reference frames and trellis set to 1, and the trellis-1 sample had a higher ssim/psnr/final ratefactor value than it's 16-ref frames/trellis-0 cousin. I did'nt wanna shame it even more by using trellis-2 :p


Thanks for the tips - I have to struggle with my hardware to get a good balance between speed and quality, though the latter is much more important to me. ;)

Np, I kinda figured this was the issue, but did'nt wanna assume anything. :)

[HeadBangeR77]

You've been very helpful, m8 - thanks.

Gonna update my results from this post (http://forum.doom9.org/showthread.php?p=1033271#post1033271) when I've finished (probably tomorrow).

cheers,
HDBR77

Comments as to my results (some loose thoughts actually):
- lowering the number of reference frames for the 1st pass can improve speed, but the rate control can have some problems in the 2nd run to hit the desired bitrate,
- between the rest of my settings there's no real difference in quality, the metrics differ very slightly, imho outside the boarders of human perception,
- 2 passes with CRF in the 1st pass deliver the most accurate results and the best metrics, yet it's a small difference, and it's the slowest of all configurations,
- my favourite configuration is No 2 then, since it doesn't degrade the quality and the first pass is 1fps faster, which makes a difference in my case.

Any other suggestions on how to speed up the 1st pass without a significant drop in metrics? Maybe Chroma ME off, and Fast P-Skip on? Plz remember, I'm a quality freak as a rule, yet my hardware has its limitations. ;)

Thanks everyone for all the help!

[akupenguin]

2pass doesn't actually use the ratefactor printed by the 1st pass. 2pass uses the 1st pass's qps and bits. The ratefactor is printed just for your information, like all the mb stats and stuff.
Yes, it can somewhat compensate for any differences in compression settings. But it only starts compensating after is has encoded several frames and observed the difference in compression-per-qp. I have not compiled any heuristic to guess the compression difference between specific settings.

[HeadBangeR77]

I've just updated my results at the beginning of the thread, and commented them two posts above. If anyone is interested or has any suggestions /more experience than I have, you're welcome to drop by and leave a comment. ;)

2pass doesn't actually use the ratefactor printed by the 1st pass. 2pass uses the 1st pass's qps and bits. The ratefactor is printed just for your information, like all the mb stats and stuff.
So all the first pass generally does is to decide on the frame type, and initially on it's size (quantizer)? That would be very similar to the first pass in XviD, with the exception it's done at constant quantizer (fast first pass, hybrid first pass I've been using for a long time)? x264's 1st pass is much more accurate then (bitrate shaping depending on certain scenes, not only constant Q).

Yes, it can somewhat compensate for any differences in compression settings. But it only starts compensating after is has encoded several frames and observed the difference in compression-per-qp. I have not compiled any heuristic to guess the compression difference between specific settings.
Thanks for clearing the things from the insider POV. :)
I remember your explanations concerning the trade-off between speed/cpu usage and compression. So no heuristic is needed, I must decide on per film basis, which settings to choose (I don't encode much, so I like to play with the source a little ;)).

cheers
HDBR77