TSM2UHD Discussion and Bug Reporting

[jdobbs]

@ocean

Thanks... so it appears they've used some of the reserved space to flag something related to HDR... now I have to figure out what the "4608" (0x1200) represents. This sure would be easier if I had the documentation. Oh well, we all have our crosses to bear I guess.

[sneaker_ger]

If you google "m2ts 4608" you find a lot of MediaInfo hits with 4608 being an ID for a PGS track. (Actually, there is one such post on the last page.) Coincidence?

[jdobbs]

If you google "m2ts 4608" you find a lot of MediaInfo hits with 4608 being an ID for a PGS track. (Actually, there is one such post on the last page.) Coincidence?You never know. But I'm was guessing there are two bit-level flags being set there:

decimal
4608

hex
0x1200

binary
0001 0010 0000 0000

[jdobbs]

If you google "m2ts 4608" you find a lot of MediaInfo hits with 4608 being an ID for a PGS track. (Actually, there is one such post on the last page.) Coincidence?But that brings out something interesting.... I just noticed that an M2TS stream I'm examining has a large group of PGS streams that use PID numbers starting at 0x12A0. That's new. The pre-v3.00 standard reserved PIDs 0x1200 - 0x129F for PGS streams (with 0x1220-0x129F being added with the 3D changes)... that has be an indicator that something new is going on.

You'd think "Ahh... new PIDs higher resolution PGS streams." -- but I thought I read in the White Paper I downloaded that the subtitles were still using 1080p PGS streams. I'll have to go back and make sure. Of course subtitles aren't the only thing a PGS stream can be used for...


[Edit] Well... all the HDR streams I've looked at so far use the same settings of 0x1200 -- so at least I can set it to that value for testing. I would guess that one flag says "UHD" and the other says "HDR". But since there's already a format flag that gets set for 2160p and a stream coding type that calls out HEVC, the first one is certainly in question. It would help if I could find an original CLPI for a stream that isn't HDR.

[Edit #2] Ok... I found one that was BT.709 and the new flag info was set to 0x0100. So I'm guessing only the first byte is really in play here. It appears that 0x12 flags BT.2020 and HDR and 0x01 flags BT.709. Still just guessing... but so far every one I've looked at is set to one of those two values.

[ocean]

Exactly, I confirm original PGS "4768" 0x12A0 movie Rise of the Planet of the Apes CLPI, after TSMuxer and TSM2UHD PGS "4608" 0x1200:

Original

"stream_PID": 4768,
"stream_coding_type": "PGS",
"language_code": "jpn",
"reserved01": 0,
"ISRC": {
"country_code": "",
"copyright_holder": "",
"recording_year": "",
"recording_number": ""

after TSMuxer and TSM2UHD


"stream_PID": 4608,
"stream_coding_type": "PGS",
"language_code": "jpn",
"reserved01": 0,
"ISRC": {
"country_code": "00",
"copyright_holder": "000",
"recording_year": "00",
"recording_number": "00000"

edit: My apologize jdobbs, I had reversed the stream.

[isoss]

hoi All,

Is there a version that wil run on a Mac OS X

greatings

isoss

[manson.reznor]

Many thanks Jdobbs for your phenomenal job on Bd Rebuilder and TSM2UHD.


TSM2UHD works fine on a Sony UBP-X700 and the Xbox One S.

I have the same problem than others > no HDR playing.

I don't have the CLPI or playlist files because I use UHD MKV files rip recoded keeping HDR with ripbot264. Then remuxed with TsMuxer bluray structure and "converted" to UHD BD with TSM2UHD.

It is possible to re-create the playlists files from a MKV or modifiy the file after remux ?

When Bd Rebuilder will be "UHD HDR Ready" ?

ps: sorry for my English - I'm French :)

[ed_co]

Please mac version!! ;)

[jdobbs]

Please mac version!! ;)Sorry. I don't do Mac.

[SeeMoreDigital]

Sorry. I don't do Mac.

I've said it before... This forum could really do with a 'like' button :)

[jdobbs]

I've updated the first post of this thread with a new release (v1.12). Changes for this release: - Added support for setting previously unknown flags that show HDR, BT.2020 and BT.709 in
the CLPI file.
- Minor and cosmetic changes.

[ocean]

Thanks jdobbs, now HDR works perfectly.

[manson.reznor]

How HDR works with you ?

I tried with a 2160p HDR mkv file converted to Bluray with TSmuxer > and TSM2UHD

No HDR output on my Xbox One S :confused:

[ocean]

Hi manson.reznor, tested with movie Rise of the Planet of the Apes structure folder, try to convert from folder instead of the container matroska.

[wolflop]

Still not working on UB 900. No Picture. Sound ok. Test with Thor Ragnarok mkv. :confused:

[jdobbs]

Still not working on UB 900. No Picture. Sound ok. Test with Thor Ragnarok mkv. :confused:Did you reencode it? I have yet to find an MKV that was UHD-BD compliant. If yours is, it is the exception.

[wolflop]

Did you reencode it? I have yet to find an MKV that was UHD-BD compliant. If yours is, it is the exception.

no reencode. No Kompression. Mvk is taken form makemkvfile. Then tsremuxxer. Then TSM2Uhd. Movie is playable with pixels in dvdfabplayer 5. In UB 900 ther is no picture, only audio

wolflop

[jdobbs]

no reencode. No Kompression. Mvk is taken form makemkvfile. Then tsremuxxer. Then TSM2Uhd. Movie is playable with pixels in dvdfabplayer 5. In UB 900 ther is no picture, only audio

wolflopWhat source was used for makemkv? Did you create the mkv yourself directly from a ripped blu-ray structure? If not -- then the odds are that it is not compliant.

[wolflop]

yes, source was the original Blu-ray structure. MKv made with makemkv.
Has someone already reported that any uhd works with the Ub 900?

[Attila1971]

I have the trouble to delay the sound. I tried some mkv files.

[no-one]

Output from TSM2UHD 1.12 on LG up970 no HDR
And when I take output from TSM2Uhd to compress uhd25G/50G by dvdfab uhdcopy. There are very mosaic and no detect hdr.

[zamengo]

yes, source was the original Blu-ray structure. MKv made with makemkv.
Has someone already reported that any uhd works with the Ub 900?

same issue here, try with a ten different discs.

[ScorpionStingz]

Just when you thought you'd get out, they pull you back in!

[d3rd3vil]

Great, so this can be used for movie-only remuxes. Does it retain Dolby Vision?

Ofc not, big surprise

[zeropc]

how do you guys even manage to use tsmuxer to remux a uhd?
i get constantly error code -2 -> reading buffer overlow

for me it's currently impossible to get a proper remux with tsmuxer of a uhd :(

[LowDead]

how do you guys even manage to use tsmuxer to remux a uhd?
i get constantly error code -2 -> reading buffer overlow

for me it's currently impossible to get a proper remux with tsmuxer of a uhd :(

Try demux first with mkvextract. Then mux with tsmuxer the separate streams.

//LD

[zeropc]

Try demux first with mkvextract. Then mux with tsmuxer the separate streams.

//LD

thanks for the tip :)

tsm2uhd though is not working for me.

i selected the newly created uhd bdmv folder and clicked start. two hours later and the progress is still at 0%

any idea?

[chin8036]

Hello, thanks for the effort to make this software, I follow the steps in instruction but the output blu-ray folder can't be played on my Sony UBP-X800 player.

I compare the folders, and found that the software only changed mpls, no other files, and it just used about 2 secs, so I thought it's not normal, do you have any advice or what can I do to fix this?

Thanks

[staina]

tsMuxeR do not work with UHD Bluray always ends with error

tsMuxeR finished with error code -2
Reading buffer overflow. Possible container streams are not syncronized. Please, verify stream fps. File name: I:\BDMV\STREAM\00045.m2ts


When will have tsMuxeR full support UHD Bluray and Dolby Vision?

Thank you staina

[jdobbs]

I compare the folders, and found that the software only changed mpls, no other files, and it just used about 2 secs
Something went wrong... the process takes about 20 mins on my system.
tsMuxeR do not work with UHD Bluray always ends with error
I've used TSMUXER on the source from UHD discs many, many times -- and have never gotten an error. The "buffer overflow" error almost always means a corrupt source (or one that hasn't been decrypted). I do always demux first (as LowDead suggested) and use the elementary streams. As for when it will have full support... probably never. But that would be up to the author. He unexpectedly provided 3D support after a long time of not hearing from him, so you never know.

[Lucius Snow]

Thanks jdobbs for your work.

Do you advice to burn the folders on BD-50 in UDF 2.5 or 2.6?

By the way, it would be nice to get a list of compatible / not compatible players models.

[Ch3vr0n]

Afaik UHD requires UDF 2.60, 2.50 is for standard BD's.

[Lucius Snow]

Thanks.

I've just made a BD HDR Rec2020 for testing. It works on my UBP-X7000. However, I don't have any HDR screen so the device displays "SDR" on the OSD. How to know if the HDR flag is present properly using this TSM2UHD tool?

EDIT : The log during the post-compilation process says "HDR collected" so it looks OK. But I have no way to be sure...

[jdobbs]

Afaik UHD requires UDF 2.60, 2.50 is for standard BD's.Really? Is that documented somewhere? I've been using UDF 2.50 and haven't noticed any issues.

[veggav]

Hey nice to find this tool avalible.

I guess all those tutorials to keep blu-ray menus when adding subtitles and audio tracks with tsmuxer will also work with tsm2uhd.
I used to do this a lot with tsmuxer and later with bdfix (which is a lot easier).

I'll give that a try in a few days and test with my OPPO 203 with russian custom firmware to check if I can add some dubs to complete UHD discs and report back.

[Lucius Snow]

Hey nice to find this tool avalible.

I guess all those tutorials to keep blu-ray menus when adding subtitles and audio tracks with tsmuxer will also work with tsm2uhd.
I used to do this a lot with tsmuxer and later with bdfix (which is a lot easier).

I'll give that a try in a few days and test with my OPPO 203 with russian custom firmware to check if I can add some dubs to complete UHD discs and report back.

I'm not sure you can build a Blu-Ray with menus and then replace the streams from H264 to HEVC to change it to Blu-Ray UHD...

[veggav]

I'm not sure you can build a Blu-Ray with menus and then replace the streams from H264 to HEVC to change it to Blu-Ray UHD...

Totally possible.
But as I see how TSM2UHD works you have to

1. demux audio, video and subtitle streams from original disk.
2. Make MKV file with audios and subtitles you want to add.
3. Create UHD BD compilant folder.
4. Change CPLI and playlist files with hex decimal to refer to the same clpi and playlist file the original disk uses.

It's actually quite simple.

TSMuxer used to have a blank spot so you could name your own mpls and clpi files isntead of default 00000 to be 0835 or something.

Maybe this could be added to TSM2UHD since it's just a number to be changed in the output files.

[Lucius Snow]

Totally possible.
But as I see how TSM2UHD works you have to

1. demux audio, video and subtitle streams from original disk.
2. Make MKV file with audios and subtitles you want to add.
3. Create UHD BD compilant folder.
4. Change CPLI and playlist files with hex decimal to refer to the same clpi and playlist file the original disk uses.

It's actually quite simple.

TSMuxer used to have a blank spot so you could name your own mpls and clpi files isntead of default 00000 to be 0835 or something.

Maybe this could be added to TSM2UHD since it's just a number to be changed in the output files.
Thanks for this tutorial but It's not such easy for a newbie like me :)

In step 2 and 3, you make a Blu-Ray HD structure from TSMuxer (from HEVC video stream source) and then patch it with TSM2UHD to make it Blu-Ray UHD compliant. Is that what you mean?

In step 4, you open in a hexadecimal editor the CLPI and MPLS from the Blu-Ray UHD

http://poubellelqt.free.fr/temp/UHD_clpi.jpg
http://poubellelqt.free.fr/temp/UHD_mpls.jpg

And here are the CLPI and MPLS from the Blu-Ray HD I authored. There's only one video stream and one menu, so the video I want to replace refer to 00001.clpi and 0001.mpls.

http://poubellelqt.free.fr/temp/HD_clpi.jpg
http://poubellelqt.free.fr/temp/HD_mpls.jpg

What CLPI and MPLS files should i change? Those from the Blu-Ray UHD ? Which values from the hexadecimal data those should i modify exactly?

How to reintegrate the original menu (from the Blu-Ray HD) to the Blu-Ray UHD structure?

Sorry but I'm a bit lost...

[Lucius Snow]

@jdobbs : Can you confirm this is possible? Thank you.

[UHD 4 Everyone]

So over the past week I've been trying to use a combination of TSM2UHD, TS Muxer, ffmpeg, Adobe Encore, and hex editing to create a UHD Disc with menus (even if the menus are 1080p) I know this is probably next to impossible. But when you're making less than $200K/year being able to purchase Scenarist UHD or KITe UHD is out of the question.

Is there anyway to retain menus from a 1080p disc and replace the streams with a UHD stream?


Here's my workflow:

1. Author disc (with menus) in Encore with 1080p streams but match audio content.
2. Render UHD content into MKV containers in x265 that are UHD compliant.
3. Remux blu-ray folder structure (one per stream, except for menu) in tsmuxer using the mpls files and replace streams with uhd h265 elementary files.
4. Drop in new m2ts files into the stream folder.
5. Drop in new mpls files.
6. Re-create entry point instructions in BDEdit
7. Drop in index.bdmv and movieobject.bdmv files from one of the remuxes.
8. Re-create instructions and references of original index and movieobject files in BDEdit.
9. Run TSM2UHD against the structure.*

* When I go through this workflow, TSM2UHD does not recognize the fact that flags need to be changed in every file. I'm not sure why it's not detecting it but the output looks like this:

========Job Started========
[11:34:16] Scanning source for UHD attributes...
- UHD attributes collected.
- Scan completed.
[11:34:16] Checking/Adjusting MPLS file...
=========Job Ended=========

So at that point I'm obviously missing a flag somewhere. It could have been overwritten when I re-created the instructions in any of the files. Or it could be that there's still an MPEG-2 stream for the menu and TSM2UHD sees it and refuses to run.

And maybe it would help if tsmuxer could remux an m2ts stream that has a PGS.

Any suggestions on what steps to take? Or where did I make a mistake (besides wasting time attempting this haha)?

[jdobbs]

Is there anyway to retain menus from a 1080p disc and replace the streams with a UHD stream?It's not impossible -- but it's not easy either. It's more than just replacing the streams. There has to be extensions and data added/provided in the index.bdmv, MovieObject.bdmv, any associated MPLS and CLPI files. You also have to change the version number of pretty much all files from v2 to v3.

[UHD 4 Everyone]

It's not impossible -- but it's not easy either. It's more than just replacing the streams. There has to be extensions and data added/provided in the index.bdmv, MovieObject.bdmv, any associated MPLS and CLPI files. You also have to change the version number of pretty much all files from v2 to v3.

So I know about the version number change that has to happen (which tsm2uhd should do).

The file I have is an 8-bit converted to 10-bit, so no new color grading just expanding the space so I'm not too worried about the data block in the extension data.

I have decrypted a retail disc I own and I'm looking at that as an example (specifically Rise of the Planet of the Apes).

After that I am looking at the extension data for the mpls files and then correcting the clpi files. And I assume that the exension data on the playlist files with ID1:3 ID2:5 is more HDR info?

I'm going to assume I don't need extension data on my MovieObject.bdmv file as I'm not seeing any on my retail disc.

As far as editing the CLPI's go, I've found the pids in the files via hex editor and bd edit. What addresses need to be changed to signify an HEVC stream?

One last question, are you using anything outside of a hex editor, bd edit, and media info to look at these flies to be able to see what's in them a little more clearly? Because I feel like I'm doing this all wrong haha.

Thanks for your all of your help and your software (seriously), sorry if I seem annoying or anything haha.

[jdobbs]

So I know about the version number change that has to happen (which tsm2uhd should do).

The file I have is an 8-bit converted to 10-bit, so no new color grading just expanding the space so I'm not too worried about the data block in the extension data.

I have decrypted a retail disc I own and I'm looking at that as an example (specifically Rise of the Planet of the Apes).

After that I am looking at the extension data for the mpls files and then correcting the clpi files. And I assume that the exension data on the playlist files with ID1:3 ID2:5 is more HDR info?

I'm going to assume I don't need extension data on my MovieObject.bdmv file as I'm not seeing any on my retail disc.

As far as editing the CLPI's go, I've found the pids in the files via hex editor and bd edit. What addresses need to be changed to signify an HEVC stream?

One last question, are you using anything outside of a hex editor, bd edit, and media info to look at these flies to be able to see what's in them a little more clearly? Because I feel like I'm doing this all wrong haha.

Thanks for your all of your help and your software (seriously), sorry if I seem annoying or anything haha.It's more than just the version. You also have to change the type and format of all references to the stream (type = 0x24, format = 8). The extension you mentioned holds HDR data. In Index.bdmv you set flags as an extension showing "this is a UHD disc". In the CLPI the mux bitrate changes as well as the type/format info (if you encoded with a bitrate higher than a standard BD uses or are using an original stream). The CLPI also has a new entry added to the program_info table. If the source is DolbyVision you also have to add an entry to the subpath table and mux the DolbyVision stream as ID 4117 (TSM2UHD doesn't do that yet). Like I said, it's not easy, and it may take more than a hex editor.

On top of everything else, there are things that have to be changed in the PMT throughout the M2TS -- and I recently discovered that TSMUXER actually hides some of the I-Frames in other frame PES packets which causes terrible sync issues. I thought it only happened on X265 encodes... but it has been reported by others that it may also occur with original streams.

I know what you mean about struggling with the "hunt and peck" method, I had to dump out files and dig around to figure out what many of the values mean.

I suggest you hang on for a few days (or maybe a little more) and use BD-RB to do all the work for you. I'm working on routines to do the M2TS muxing for you, etc.

You're using BDEDIT on UHD discs? Whenever I try it crashes.

[Lucius Snow]

Jdobbs, do you advice not to use TSmuxer at the moment because of this sync issue? Does it occur as a permanent desynchronization during all the playback? I've encoded some medias with x265 and put some patterns / tones during one frame to check the sync. It was ok.

[jdobbs]

Jdobbs, do you advice not to use TSmuxer at the moment because of this sync issue? Does it occur as a permanent desynchronization during all the playback? I've encoded some medias with x265 and put some patterns / tones during one frame to check the sync. It was ok.As I said, I personally haven't experienced the issue on original sources (it was reported by others) -- only reencodes. So I guess it comes down to individual choice.

My plan is to write a replacement for TSMUXER. I've written the module for HEVC muxing and creation of MPLS and CLPI files already so I can fix TSMUXER output in BD-RB -- but I can't predict when the entire project might be completed.

[UHD 4 Everyone]

You're using BDEDIT on UHD discs? Whenever I try it crashes.

Yeah you just have to change the version tag on each file you want to read to v2 and it will read them (albeit/obviously it can be garbled a bit since BD Edit hasn’t been updated in years). Also nothing will be listed for hevc video streams in the mpls and clpi editors.

[Lucius Snow]

As I said, I personally haven't experienced the issue on original sources (it was reported by others) -- only reencodes. So I guess it comes down to individual choice.

My plan is to write a replacement for TSMUXER. I've written the module for HEVC muxing and creation of MPLS and CLPI files already so I can fix TSMUXER output in BD-RB -- but I can't predict when the entire project might be completed.

That's good news, thanks a lot for your work. Which e-mail adress can I use to send you a donation (via Amazon gift)? I've found dvd-rb@dvd-rb.com

Do you know if the HDR metadatas not detected with some LG players is on the way to be fixed? It was mentionned there for example: https://forum.doom9.org/showthread.php?p=1835447#post1835447

[jdobbs]

That's good news, thanks a lot for your work. Which e-mail adress can I use to send you a donation (via Amazon gift)? I've found dvd-rb@dvd-rb.com

Do you know if the HDR metadatas not detected with some LG players is on the way to be fixed? It was mentionned there for example: https://forum.doom9.org/showthread.php?p=1835447#post1835447dvd-rb@jdobbs.net is probably best. Thanks for thinking about me.

Not sure about the LG issue... I can't think of any reason why it wouldn't work. Obviously I must have missed something. Have you tried running MEDIAINFO against the video to make sure the HDR is there?

[Lucius Snow]

dvd-rb@jdobbs.net is probably best. Thanks for thinking about me.

Not sure about the LG issue... I can't think of any reason why it wouldn't work. Obviously I must have missed something. Have you tried running MEDIAINFO against the video to make sure the HDR is there?
Donation done!

I don't have any LG player myself, only the Sony UBP-X700. I was just referring to comments about the LG issue. That's important to me because I'll give some Blu-Ray UHD to some people who may have any kind of brand / model of Blu-Ray UHD standalone player. I'd love a kind of universal compatibility.

I've just made a test on a HDR mastering display, the OSD of the UBP-X700 says it's still SDR. It's odd because the TSM2UHD logs detected the HDR flag.

[jdobbs]

Thought I'd post an update on the BD-RB's UHD-BD processing.

I've completed and tested the HEVC muxing routines and all is well. BD-RB corrects the TSMUXER M2TS for HEVC output and replaces the MPLS and CLPI files (corrupt due to TSMUXERs HEVC muxing issue) with those internally generated by BD-RB. The output works perfectly on my Sony UHD player.

Now I am making modifications to the import routines so they will also correct for TSMUXERs actions. It's not as easy as you may think, since I still need TSMUXER to combine multi-part playlists during import and to mux the other (non-hevc) streams. I've also done research on how BDs handle DolbyVision streams. An extra video track containing the DolbyVision data is muxed as a separate stream into the M2TS file. Since TSMUXER has no awareness of that stream (it thinks it's just another secondary video) I have to go back through my code and make changes so that it can be integrated properly into output. I'm not sure how long that will take -- but I don't expect it to be extensive.

All in all things are moving along -- albeit slowly. I may get a version out for testing in the next few days, but at the latest I'll get it out by EOY.