The VirtualDubMod project is dead for years now.
Also the Matroska support in VirtualDubMod is so much outdated that even the Matroska web-site warns you about it
Please update your outdated VirtuaDubMod to an up-to-date
VirtualDub. Then use Avisynth with
FFmpegSource2() to open the MKV file...
Alternative you may want to have look at Avidemux 2.6:
http://avidemux.org/nightly/win32/