libavcodec seems to have some trouble outputting non-mod16 video. I used Fraps to capture some video at 1680x1050 and the video looks like this when played back with ffdshow or mplayer. The problem showed up in both the uncompressed video and xvid compressed version, but disappeared when fraps own codec and xvid were used to decompress.
screencap1
screencap2
[edit] Seems like XviD used YUY2 colorspace for output and when i forced ffdshow to use it everything was fine. A Nvidia video driver bug with YV12?