To demux the MKV samples, you can also use mkvextract, a command line tool that is part of
mkvtoolnix. It is also included in the toolset folder of BD3D2MK3D. If you prefer a GUI, try
mkvextractGUI-2. And, as Frank wrote, you can also just replace DGMVCDecode.dll in the toolset folder of BD3D2MK3D and use it in MKV source mode.
BTW, thanks to you, and to Frank, for your efforts! And sorry. I can't help, because I can't test the filter in hw mode. :-(