Is there any solution yet? I have the same problem with "Enterprise Season 3": mkvmerge declares my .264 data as MP3!
Code:
'<SOME_PATH>\VTS_01_1_EncoderOutput.264': Using the MP2/MP3 demultiplexer.
VOB and extracted AC3 play fine though. The header of the .264 file states clearly "H.264/MPEG-4 AVC codec". I am currently encoding as XVID. I let you know if this succeeds...