I was bored, so I spent 15 minutes and wrote this patch to port x264's new VAQ to Xvid.
I haven't tested it much (only on
this video), but it seems to work surprisingly well, despite MPEG-4 ASP's drastic limitations on quantizer deltas.
Patch [updated with minor fixes]
Xvidcore.dll [updated with latest patch]
To use it with Virtualdub or any VfW interface, just overwrite the XvidCore.dll in your /Windows/System32 or similar with the one I have provided.
To activate VAQ, just check off the Adaptive Quantization box; it replaces the normal luminance masking.
Note that I could not get it working correctly with mencoder; for some reason libavcodec seems to have its own lumimasking library that's overriding my patch. It works with xvid_encraw and VfW though...
Have fun!