x264 questions about mb type selection

iamthird36 · 16th May 2007, 16:43

A few questions how x264 is performing mode selection. My overall problem is that I have different macroblock types of varying importance. For one of the types, I really don't care at all about distortion and I'd like to encode it with as few bits as possible. In this case, it seems like all these blocks should be encoded using skip (if its a P slice) or I 16x16 (if its an I slice).

I've adjusted the coder in two ways for these particular macroblocks. I either significantly increase lambda and lambda2 to force the encoder to operate with a very small rate constraint on those blocks or I adjust the ssd cost such that it is 0 for these blocks. My intuition is that by doing this, the coder should be selecting the modes with the lowest possible bitrate. This doesn't seem to be the case. I get a lot of intra 4x4 (even in p slices) and inter 16x16 blocks with pretty small motion vectors. Any thoughts?

Also, a more general question on how skip modes are determined. With fast-p-skip enabled, how does the encoder determine if a block should be a skip or a predicted block? Without any of my modifications, I seem to get a lot of predicted blocks with relatively small motion vectors where there is no change between frames.

Thanks for any feedback.

akupenguin · 16th May 2007, 17:23

lambda2 (in RDO) does trade bits vs residual. But lambda (in SAD/SATD) doesn't. lambda trades mode/mv bits vs residual, and much of the residual cost is also bits. So there's an optimal lambda per qp, and if you set it too high then motion estimation will be forced to pick a non-optimal mv (because small mvs take fewer bits) or fewer partitions, thus increasing the residual, and maybe adding more bits in dct coefs than you saved in mv.

Likewise, i16x16 isn't necessarily smaller than i4x4. i4x4 spends some bits of prediction to reduce bits of residual, which may or may not be a good trade depending on qp and the amount of detail in the mb.

So the factors you can change to influence bits vs quality are: lambda2, qp, deadzone.

fast-p-skip, low subme: Encode a skip mb. If there are no dct coefficients with magnitude >= 2, and the total number of coefs with magnitude = 1 is below some threshold, then keep the mb as skip. Else analyse inter and intra mv types.
fast-p-skip, high subme: Analyse p16x16ref0. If its mv is within +/-1 qpel of the skip mv, then check skip's coefs as above.
no-fast-pskip: Analyse all the mb types. If the chosen type is p16x16ref0 and mv is equal to skip's mv and there are sufficiently few nonzero coefs, then use skip.

iamthird36 · 16th May 2007, 18:47

So lambda2 is only used if rdo is enabled (subme >= 6)? Also, if I set lambda = 1, it will find the best matching motion vectors? How will this affect mode selection. For example, if I set lambda = 1 but lambda2 very large, will the coder calculate the best motion vectors but then select a low cost mode?

In skip mode decision, is it looking at quantized coefficients or just the transform coefficients. For very large QP, most of the quantized coefficients should be 0, so skip should be selected for small motion vectors. Also, are the skip mvs just MVp (median of the left, upper, and upper left block motion vectors)?

16th May 2007, 16:43	#1 \| Link
iamthird36 Registered User Join Date: Apr 2007 Posts: 4	x264 questions about mb type selection A few questions how x264 is performing mode selection. My overall problem is that I have different macroblock types of varying importance. For one of the types, I really don't care at all about distortion and I'd like to encode it with as few bits as possible. In this case, it seems like all these blocks should be encoded using skip (if its a P slice) or I 16x16 (if its an I slice). I've adjusted the coder in two ways for these particular macroblocks. I either significantly increase lambda and lambda2 to force the encoder to operate with a very small rate constraint on those blocks or I adjust the ssd cost such that it is 0 for these blocks. My intuition is that by doing this, the coder should be selecting the modes with the lowest possible bitrate. This doesn't seem to be the case. I get a lot of intra 4x4 (even in p slices) and inter 16x16 blocks with pretty small motion vectors. Any thoughts? Also, a more general question on how skip modes are determined. With fast-p-skip enabled, how does the encoder determine if a block should be a skip or a predicted block? Without any of my modifications, I seem to get a lot of predicted blocks with relatively small motion vectors where there is no change between frames. Thanks for any feedback.

16th May 2007, 17:23	#2 \| Link
akupenguin x264 developer Join Date: Sep 2004 Posts: 2,392	lambda2 (in RDO) does trade bits vs residual. But lambda (in SAD/SATD) doesn't. lambda trades mode/mv bits vs residual, and much of the residual cost is also bits. So there's an optimal lambda per qp, and if you set it too high then motion estimation will be forced to pick a non-optimal mv (because small mvs take fewer bits) or fewer partitions, thus increasing the residual, and maybe adding more bits in dct coefs than you saved in mv. Likewise, i16x16 isn't necessarily smaller than i4x4. i4x4 spends some bits of prediction to reduce bits of residual, which may or may not be a good trade depending on qp and the amount of detail in the mb. So the factors you can change to influence bits vs quality are: lambda2, qp, deadzone. fast-p-skip, low subme: Encode a skip mb. If there are no dct coefficients with magnitude >= 2, and the total number of coefs with magnitude = 1 is below some threshold, then keep the mb as skip. Else analyse inter and intra mv types. fast-p-skip, high subme: Analyse p16x16ref0. If its mv is within +/-1 qpel of the skip mv, then check skip's coefs as above. no-fast-pskip: Analyse all the mb types. If the chosen type is p16x16ref0 and mv is equal to skip's mv and there are sufficiently few nonzero coefs, then use skip.

16th May 2007, 18:47	#3 \| Link
iamthird36 Registered User Join Date: Apr 2007 Posts: 4	So lambda2 is only used if rdo is enabled (subme >= 6)? Also, if I set lambda = 1, it will find the best matching motion vectors? How will this affect mode selection. For example, if I set lambda = 1 but lambda2 very large, will the coder calculate the best motion vectors but then select a low cost mode? In skip mode decision, is it looking at quantized coefficients or just the transform coefficients. For very large QP, most of the quantized coefficients should be 0, so skip should be selected for small motion vectors. Also, are the skip mvs just MVp (median of the left, upper, and upper left block motion vectors)?