Your formula assumes PC levels (0-255) rather than the normal 'TV' levels (16-235). Although the formula could be rewritten, the simplest solution is to use ColorYUV(levels="TV->PC") on both input clips before calling mt_lutxy, and ColorYUV(levels="PC->TV") after. But I think all your 256's probably should be 255's.
I've never heard of mt_infix - do you mean mt_polish?
The call could be written as:
Code:
mt_lutxy(a,b,mt_polish("255 - (255-x)*(255-y)/255"),Y=3, U=2, V=2)