how does x264 works on mode decision?

[rlancelot_xjb]

recently I am studying on the x264_rev270 source code,but come across many problems.The most confusing one is what the principle when decide which mode will choose for the best one in function such as x264_rd_cost_mb() in file rdo.c and where do parameter i_qp0_cost_table,i_qp0_cost2_table come from?it seems that i_qp0_cost_table is same as jm's QP2QUANT[40]=
{
1, 1, 1, 1, 2, 2, 2, 2,
3, 3, 3, 4, 4, 4, 5, 6,
6, 7, 8, 9,10,11,13,14,
16,18,20,23,25,29,32,36,
40,45,51,57,64,72,81,91
};
but I can not understand how it works.the rdo method introduced in article "Rate-Constrained Coder Control and Comparison of Video Coding Standards"(by Thomas Wiegand)
gives equation lambda_mode=0.85*pow(2,(qp-12)/3) and lambda_motion=pow(lambda_mode,0.5),it is different from x264
are there any relationship between them?
thanks! :helpful:

[superdump]

You can find the two i_qp0_cost tables in encoder/analyse.c. The values within these tables are formed using equations similar to those you mentioned in your post.

Rate distortion (hereafter known as RD) is a way of choosing the best method for doing something by evaluating a score for a method. The score is based on the bits used to encode a block using a method and the difference between the coded block and the same block within the source frame you're trying to code.

So for the mode decision, if you're using RD throughout you would fully encode the block using motion vectors (MVs) corresponding to each partition, calculate the sum of squared differences (SSD), run it through the entropy coder to get the bits used to code using that partition arrangement. Then you calculate the rate distortion via the equation RD = SSD + bits*lambda. It should be clear that we wish to minimise this value. The lambda values are looked up from the tables aforementioned. I think the values in the tables have been adjusted slightly to perform best with x264.

I hope that helps. :) Someone else might turn up and give a better description.

EDIT: Reworded second paragraph for clarity.

[Sirber]

I way too not understand :D