How about this:
Code:
ffvideosource("blah.ts") #FFMS2 to source the video
deinterlacers()/tfm()/whateverhere()
filters()
Then just throw that at x264 with the desired bitrate/crf and you'll have a smaller mkv. If you want to keep a ts then you can use tsmuxer (and appropriate level settings in x264) to make a transport stream.
Handle the audio separately until the final muxing (if you change it at all).
You can use DGAVCIndex to demux the audio (and source the video if you like).
FFMS2
Edit: I'm not sure if you said your original tranport stream is h264 or not...
If it's mpeg2 video then use dgindex to index and source the video instead of FFMS2.