Hi r0lZ,
For the sake of your tool, I don't know if tsMuxer can have certain issues with demuxing BD iso's. I never encountered them, for 3D BD anyways.
This (de)muxer tool is rather dated, and you know filler56789 is busy with it in his tsMuxer Open Source thread.
Don't know if you were already aware of it, but as for purely demuxing, maybe a thought to look into this thread?:
https://forum.doom9.org/showthread.php?t=177053