Code:
sps->vui.i_num_reorder_frames = param->b_bframe_pyramid ? 2 : param->i_bframe ? 1 : 0;
/* extra slot with pyramid so that we don't have to override the
* order of forgetting old pictures */
sps->vui.i_max_dec_frame_buffering =
sps->i_num_ref_frames = X264_MIN(16, X264_MAX(param->i_frame_reference, 1 + sps->vui.i_num_reorder_frames));
So it's your formula.