1080p .mkv repackaging for xbox 360

[kulivontot]

Hi, I have some 1080p .mkv content that I want to repackage as .mp4 for playback on my xbox 360. I have successfully done this for a number of other 720p files and gotten it to work properly. However, my 1080p files play back on the screen with a bunch of corruption and artifcacts. I've also tried changing the profile level to 4.1 manually before repackaging as .mp4 and this allows the content to be played, but I start getting artifacts and corruption on the screen.

At first I thought the bitrate was too high, but I re-encoded a 1 minute sample at up to 20mbps and was able to get the file to play back fine. I even encoded it as high profile 5.1 at 20mbps, applied the 4.1 level fix, and it would still play just fine!

This leads me to believe that there is something wrong with the original h.264 stream that causes the 360 to play it back improperly. However, it appears that this incompatibility is not bit-rate related as I can jack that level up higher than the original file at full resolution and still have no issues. Anyone have any suggestions? I really don't want to have to spend 15+ hours per each 1080p movie and then degrade the picture quality by doing a separate encode.

Here is the process I am doing:
1. demux the .mkv using mkvextract
2. alter the .264 file using either h264info or hex editing the file
3. multiplexing a .mp4 file using mp4 box
4. stream the .mp4 over the network to my xbox

[foxyshadis]

You probably have too many reference frames used in the original, 1080p can only use a maximum of 4 on level 4.1. (There's a thread (https://forum.doom9.org/showthread.php?t=132924) about compatibility problems with b-frames and references and a patched build to alleviate that, so you can use the maximum.) No fix except to re-encode.

[kulivontot]

How can I check this sort of information? I'm struggling to find a utility that will let me see extended info on the video streams such as these reference frames you refer to.