[plugh]

It has been an extremely tedious process, but I have tracked back to a specific chunk of code that gives differant results when compiled with msvc vs gcc.

(When I say tedious I mean it - tracking backwards through the code, identifying where a particular rd mode evaluation for a particular macroblock for a particular frame goes weird)

The chuck of code is

static __inline uint32_t
d_mv_bits(int x, int y, const VECTOR pred, const uint32_t iFcode, const int qpel)
{
	unsigned int bits;

	x <<= qpel;
	y <<= qpel;

	x -= pred.x;
	bits = (x != 0 ? iFcode:0);
	x = -abs(x);
	x >>= (iFcode - 1);
	bits += r_mvtab[x+63];

	y -= pred.y;
	bits += (y != 0 ? iFcode:0);
	y = -abs(y);
	y >>= (iFcode - 1);
	bits += r_mvtab[y+63];

	return bits;
}

in motion_inlines.h; r_mvtab is defined there as well.

The arguments being passed in this particular case are
x=-64 y=63 pred={x=63,y=15} iFcode=2 qpel=0

msvc produces the value 4128837
gcc produces the value 14
I manually calc it, and I get 26

The call stack is:
ModeDecision_BVOP_RD ->
SearchInterpolate_RD ->
CheckCandidateRDInt ->
the first instance in the following statement

rd += BITS_MULT * (d_mv_bits(xf, yf, data->predMV, data->iFcode, data->qpel^data->qpel_precision)
		+ d_mv_bits(xb, yb, data->bpredMV, data->iFcode, data->qpel^data->qpel_precision));

WTF!