Help me understand rate distortion (subme 6/7)

[F J Walter]

Hi

For curiosity I'm interested in learning more about how rate distortion fits into the motion estimation process in x264, and specifically in subme 6/7.

I understand that rate distortion factors in bitrate efficiency (quality vs bitrate) into the usual metrics used to make decisions, and it does so by calculating both a quality metric as well as an estimated bit cost.

Now I also know that in subme 6/7, rate distortion optimisation is only done at some late stage of motion estimation - various sources say it's the decision of partition types only.

Now what I don't know:

Partition type decision: does that mean the decision of whether to encode a single motion vector for the whole 16x16 block or divide it into partitions down to 8x16,8x8,8x4 and 4x4? Is this the only step that RD in subme 6/7 affects?

Or does RD in subme 6/7 also affect the calculation of whether to refine the motion vector to half or quarter pixel precision?

Or does RD in subme 6/7 also affect the decision of whether to encode a block as a I or P block?

Or does it also affect some other calculation?

I understand that the RDO in subme 8/9 also uses a rate distortion calculation when actually finding the motion vectors - does this mean the full pixel search AND the decision to go to half or quarter pixel precision AND the hpel/qpel search?

And lastly, if RDO for subme 6/7 only affects the partition type decision, how is it able to have such a significant positive effect as it does? Particularly when using Psy RDO, which also introduces a "complexity" metric into the mix - how is that able to have such a significant effect when all it is doing is affecting the partition type decision? How is the partition type decision all that important?

Thanks in advance.