Two questions about x264's internal workings

[Dark Shikari]

a) Going by the code, x264 uses optimized SAD algorithms for motion searching. Would other algorithms create better results (albeit at the cost of speed), such as the DCT used in MVTools?

b) Does x264 use luma, chroma, or both for motion searching?

[akupenguin]

a) DCT will produce better results than SAD. But for any given quota of CPU-time you get better results with a wider SAD search and then refine with another metric. Which x264 does: first SAD, then SATD, then RD.

b) Fullpel SAD search: luma. Subpel SATD: both, unless you say --no-chroma-me. RD: both.

[Dark Shikari]

a) DCT will produce better results than SAD. But for any given quota of CPU-time you get better results with a wider SAD search and then refine with another metric. Which x264 does: first SAD, then SATD, then RD.

b) Fullpel SAD search: luma. Subpel SATD: both, unless you say --no-chroma-me. RD: both.
Many of the settings in x264 increase the precision at earlier search levels, performing a tradeoff of lots of speed for some quality. Would chroma search during the fullpel search improve the result at all, or has it been shown to be totally useless?

[akupenguin]

Dunno, but it's really inconvenient to implement. You can try it in lavc, which has fully configurable comparison functions for motion estimation, but as a result does not have some of x264's optimizations.