x264 questions about rd_cost and trellis quantization.

[iamthird36]

I have two separate sets of questions.

First, in the x264 trellis quantizer, what exactly is the trellis optimizing over? Does each node correspond to a possible CABAC context?

Second, in x264_rd_cost_mb, does i_bits include the rate required to code all the parameters (such as dqp, motion vectors, etc...) or is this just the rate associated with the quantized coefficients? Also, is this an estimated rate or is it the true rate? Is this true for both CABAC or CAVLC?

Thanks in advance for any feedback.

[akupenguin]

1) Trellis optimizes over the values of the quantized coefficients. The limited state that it keeps track of (allowing it to be a trellis rather than an exponential time search) is the CABAC context. I might say "it trellisses over the CABAC context", though I don't know if that helps anyone or if I'm just inventing terminology on the fly.

http://akuvian.org/images/cabac_trellis_fig.png
The label under each node ("2_0" etc) is its CABAC context.
The horizontal position ("C15" etc) is which coefficient it's optimizing next.
The lines show which CABAC context can turn into which other CABAC context via the addition of one coefficient.
x264's nodes_cur[] contains one column of this chart at any given time, and nodes_prev[] contains the previous (i.e. left neighbor) column.

2) RD's bits include all bits associated with the macroblock, including dqp and mv. Yes, it's the true rate. (In the case of CABAC it's not exact, but it's within some fraction of a bit of the true rate, limited only by arithmetic precision.)

[iamthird36]

Thanks so much, akupenguin.