Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > (HD) DVD, Blu-ray & (S)VCD > DVD Rebuilder

Reply
 
Thread Tools Search this Thread Display Modes
Old 17th December 2018, 12:09   #141  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,252
Quote:
Is there anyway to retain menus from a 1080p disc and replace the streams with a UHD stream?
It's not impossible -- but it's not easy either. It's more than just replacing the streams. There has to be extensions and data added/provided in the index.bdmv, MovieObject.bdmv, any associated MPLS and CLPI files. You also have to change the version number of pretty much all files from v2 to v3.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net

Last edited by jdobbs; 17th December 2018 at 12:12.
jdobbs is offline   Reply With Quote
Old 17th December 2018, 22:22   #142  |  Link
UHD 4 Everyone
Registered User
 
Join Date: Dec 2018
Posts: 4
Quote:
Originally Posted by jdobbs View Post
It's not impossible -- but it's not easy either. It's more than just replacing the streams. There has to be extensions and data added/provided in the index.bdmv, MovieObject.bdmv, any associated MPLS and CLPI files. You also have to change the version number of pretty much all files from v2 to v3.
So I know about the version number change that has to happen (which tsm2uhd should do).

The file I have is an 8-bit converted to 10-bit, so no new color grading just expanding the space so I'm not too worried about the data block in the extension data.

I have decrypted a retail disc I own and I'm looking at that as an example (specifically Rise of the Planet of the Apes).

After that I am looking at the extension data for the mpls files and then correcting the clpi files. And I assume that the exension data on the playlist files with ID1:3 ID2:5 is more HDR info?

I'm going to assume I don't need extension data on my MovieObject.bdmv file as I'm not seeing any on my retail disc.

As far as editing the CLPI's go, I've found the pids in the files via hex editor and bd edit. What addresses need to be changed to signify an HEVC stream?

One last question, are you using anything outside of a hex editor, bd edit, and media info to look at these flies to be able to see what's in them a little more clearly? Because I feel like I'm doing this all wrong haha.

Thanks for your all of your help and your software (seriously), sorry if I seem annoying or anything haha.
UHD 4 Everyone is offline   Reply With Quote
Old 18th December 2018, 13:05   #143  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,252
Quote:
Originally Posted by UHD 4 Everyone View Post
So I know about the version number change that has to happen (which tsm2uhd should do).

The file I have is an 8-bit converted to 10-bit, so no new color grading just expanding the space so I'm not too worried about the data block in the extension data.

I have decrypted a retail disc I own and I'm looking at that as an example (specifically Rise of the Planet of the Apes).

After that I am looking at the extension data for the mpls files and then correcting the clpi files. And I assume that the exension data on the playlist files with ID1:3 ID2:5 is more HDR info?

I'm going to assume I don't need extension data on my MovieObject.bdmv file as I'm not seeing any on my retail disc.

As far as editing the CLPI's go, I've found the pids in the files via hex editor and bd edit. What addresses need to be changed to signify an HEVC stream?

One last question, are you using anything outside of a hex editor, bd edit, and media info to look at these flies to be able to see what's in them a little more clearly? Because I feel like I'm doing this all wrong haha.

Thanks for your all of your help and your software (seriously), sorry if I seem annoying or anything haha.
It's more than just the version. You also have to change the type and format of all references to the stream (type = 0x24, format = 8). The extension you mentioned holds HDR data. In Index.bdmv you set flags as an extension showing "this is a UHD disc". In the CLPI the mux bitrate changes as well as the type/format info (if you encoded with a bitrate higher than a standard BD uses or are using an original stream). The CLPI also has a new entry added to the program_info table. If the source is DolbyVision you also have to add an entry to the subpath table and mux the DolbyVision stream as ID 4117 (TSM2UHD doesn't do that yet). Like I said, it's not easy, and it may take more than a hex editor.

On top of everything else, there are things that have to be changed in the PMT throughout the M2TS -- and I recently discovered that TSMUXER actually hides some of the I-Frames in other frame PES packets which causes terrible sync issues. I thought it only happened on X265 encodes... but it has been reported by others that it may also occur with original streams.

I know what you mean about struggling with the "hunt and peck" method, I had to dump out files and dig around to figure out what many of the values mean.

I suggest you hang on for a few days (or maybe a little more) and use BD-RB to do all the work for you. I'm working on routines to do the M2TS muxing for you, etc.

You're using BDEDIT on UHD discs? Whenever I try it crashes.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net

Last edited by jdobbs; 18th December 2018 at 15:11.
jdobbs is offline   Reply With Quote
Old 18th December 2018, 13:58   #144  |  Link
Lucius Snow
Registered User
 
Join Date: Oct 2003
Location: Paris, France
Posts: 83
Jdobbs, do you advice not to use TSmuxer at the moment because of this sync issue? Does it occur as a permanent desynchronization during all the playback? I've encoded some medias with x265 and put some patterns / tones during one frame to check the sync. It was ok.
Lucius Snow is offline   Reply With Quote
Old 18th December 2018, 15:09   #145  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,252
Quote:
Originally Posted by Lucius Snow View Post
Jdobbs, do you advice not to use TSmuxer at the moment because of this sync issue? Does it occur as a permanent desynchronization during all the playback? I've encoded some medias with x265 and put some patterns / tones during one frame to check the sync. It was ok.
As I said, I personally haven't experienced the issue on original sources (it was reported by others) -- only reencodes. So I guess it comes down to individual choice.

My plan is to write a replacement for TSMUXER. I've written the module for HEVC muxing and creation of MPLS and CLPI files already so I can fix TSMUXER output in BD-RB -- but I can't predict when the entire project might be completed.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 18th December 2018, 16:24   #146  |  Link
UHD 4 Everyone
Registered User
 
Join Date: Dec 2018
Posts: 4
Quote:
Originally Posted by jdobbs View Post
You're using BDEDIT on UHD discs? Whenever I try it crashes.
Yeah you just have to change the version tag on each file you want to read to v2 and it will read them (albeit/obviously it can be garbled a bit since BD Edit hasn’t been updated in years). Also nothing will be listed for hevc video streams in the mpls and clpi editors.
UHD 4 Everyone is offline   Reply With Quote
Old 18th December 2018, 17:33   #147  |  Link
Lucius Snow
Registered User
 
Join Date: Oct 2003
Location: Paris, France
Posts: 83
Quote:
Originally Posted by jdobbs View Post
As I said, I personally haven't experienced the issue on original sources (it was reported by others) -- only reencodes. So I guess it comes down to individual choice.

My plan is to write a replacement for TSMUXER. I've written the module for HEVC muxing and creation of MPLS and CLPI files already so I can fix TSMUXER output in BD-RB -- but I can't predict when the entire project might be completed.
That's good news, thanks a lot for your work. Which e-mail adress can I use to send you a donation (via Amazon gift)? I've found dvd-rb@dvd-rb.com

Do you know if the HDR metadatas not detected with some LG players is on the way to be fixed? It was mentionned there for example: https://forum.doom9.org/showthread.p...47#post1835447

Last edited by Lucius Snow; 18th December 2018 at 17:36.
Lucius Snow is offline   Reply With Quote
Old 18th December 2018, 20:27   #148  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,252
Quote:
Originally Posted by Lucius Snow View Post
That's good news, thanks a lot for your work. Which e-mail adress can I use to send you a donation (via Amazon gift)? I've found dvd-rb@dvd-rb.com

Do you know if the HDR metadatas not detected with some LG players is on the way to be fixed? It was mentionned there for example: https://forum.doom9.org/showthread.p...47#post1835447
dvd-rb@jdobbs.net is probably best. Thanks for thinking about me.

Not sure about the LG issue... I can't think of any reason why it wouldn't work. Obviously I must have missed something. Have you tried running MEDIAINFO against the video to make sure the HDR is there?
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net

Last edited by jdobbs; 18th December 2018 at 20:31.
jdobbs is offline   Reply With Quote
Old 18th December 2018, 23:56   #149  |  Link
Lucius Snow
Registered User
 
Join Date: Oct 2003
Location: Paris, France
Posts: 83
Quote:
Originally Posted by jdobbs View Post
dvd-rb@jdobbs.net is probably best. Thanks for thinking about me.

Not sure about the LG issue... I can't think of any reason why it wouldn't work. Obviously I must have missed something. Have you tried running MEDIAINFO against the video to make sure the HDR is there?
Donation done!

I don't have any LG player myself, only the Sony UBP-X700. I was just referring to comments about the LG issue. That's important to me because I'll give some Blu-Ray UHD to some people who may have any kind of brand / model of Blu-Ray UHD standalone player. I'd love a kind of universal compatibility.

I've just made a test on a HDR mastering display, the OSD of the UBP-X700 says it's still SDR. It's odd because the TSM2UHD logs detected the HDR flag.

Last edited by Lucius Snow; 21st December 2018 at 15:07.
Lucius Snow is offline   Reply With Quote
Old 21st December 2018, 15:17   #150  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,252
Thought I'd post an update on the BD-RB's UHD-BD processing.

I've completed and tested the HEVC muxing routines and all is well. BD-RB corrects the TSMUXER M2TS for HEVC output and replaces the MPLS and CLPI files (corrupt due to TSMUXERs HEVC muxing issue) with those internally generated by BD-RB. The output works perfectly on my Sony UHD player.

Now I am making modifications to the import routines so they will also correct for TSMUXERs actions. It's not as easy as you may think, since I still need TSMUXER to combine multi-part playlists during import and to mux the other (non-hevc) streams. I've also done research on how BDs handle DolbyVision streams. An extra video track containing the DolbyVision data is muxed as a separate stream into the M2TS file. Since TSMUXER has no awareness of that stream (it thinks it's just another secondary video) I have to go back through my code and make changes so that it can be integrated properly into output. I'm not sure how long that will take -- but I don't expect it to be extensive.

All in all things are moving along -- albeit slowly. I may get a version out for testing in the next few days, but at the latest I'll get it out by EOY.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 21st December 2018, 15:33   #151  |  Link
SeeMoreDigital
Life looks better in UHD
 
SeeMoreDigital's Avatar
 
Join Date: Jun 2003
Location: Notts, UK
Posts: 11,406
That's very encouraging work
__________________
| I've been testing hardware media playback devices and software A/V encoders and decoders since 2001 | My Network Layout & A/V Gear |
SeeMoreDigital is offline   Reply With Quote
Old 21st December 2018, 18:12   #152  |  Link
Lucius Snow
Registered User
 
Join Date: Oct 2003
Location: Paris, France
Posts: 83
Quote:
Originally Posted by Lucius Snow View Post
I've just made a test on a HDR mastering display, the OSD of the UBP-X700 says it's still SDR. It's odd because the TSM2UHD logs detected the HDR flag.
It has finally worked after the firmware upgrade of the player!

EDIT : What I have noticed is that HDR flag appears in the OSD of the Sony player only if Dolby Vision is enabled in the config menu. Why is that?

Here is my logs:

========Job Started========
[15:04:54] Scanning source for UHD attributes...
- UHD and HDR attributes collected.
- Scan completed.
[15:04:54] Checking/Adjusting MPLS file...
- Found HEVC stream...
-- Type marker adjusted to UHD-BD HEVC...
-- Format marker adjusted to 2160p...
-- Version marker adjusted to be 0300...
[15:04:54] Checking/Adjusting CLPI file...
- Found HEVC stream...
-- Type marker adjusted to UHD-BD HEVC...
-- Format marker adjusted to 2160p...
-- Version marker adjusted to be 0300...
[15:04:54] Checking/Adjusting BDMV files...
- Added UHD-BD extension...
- Checked/Updated version in Index file...
- Checked/Updated version in MovieObject file...
[15:04:54] Checking/Adjusting M2TS file...
[15:04:58] TSMUXER TO UHD - SUCCESS
=========Job Ended=========

EDIT2: And the displayed colours are not good at all then. Can you confirm the Sony's OSD should mention "HDR" without Dolby Vision enabled?

EDIT3 : If I take 00000.m2ts file out from the Blu-Ray folder and copy it to a USB key that I plug in, the Sony player detects it as HDR and displays properly the colours. With "Dolby Vision" disabled. So it looks like a TSM2UHD bug during the patch?

Last edited by Lucius Snow; 22nd December 2018 at 15:04.
Lucius Snow is offline   Reply With Quote
Old 25th December 2018, 19:08   #153  |  Link
Cesarleeds
Registered User
 
Join Date: Dec 2018
Posts: 2
Quote:
Originally Posted by jdobbs View Post
Thought I'd post an update on the BD-RB's UHD-BD processing.

I've completed and tested the HEVC muxing routines and all is well. BD-RB corrects the TSMUXER M2TS for HEVC output and replaces the MPLS and CLPI files (corrupt due to TSMUXERs HEVC muxing issue) with those internally generated by BD-RB. The output works perfectly on my Sony UHD player.

Now I am making modifications to the import routines so they will also correct for TSMUXERs actions. It's not as easy as you may think, since I still need TSMUXER to combine multi-part playlists during import and to mux the other (non-hevc) streams. I've also done research on how BDs handle DolbyVision streams. An extra video track containing the DolbyVision data is muxed as a separate stream into the M2TS file. Since TSMUXER has no awareness of that stream (it thinks it's just another secondary video) I have to go back through my code and make changes so that it can be integrated properly into output. I'm not sure how long that will take -- but I don't expect it to be extensive.

All in all things are moving along -- albeit slowly. I may get a version out for testing in the next few days, but at the latest I'll get it out by EOY.
Hey jdobbs, thanks so much for your work.

I´ve been a long time tsmuxer user, our bluray market in Spain is quite abandoned, so i learnt how to create my own custom blurays which i´ve been doing, as i said, during many years.

The UHD market is even worst, we don´t even get the Disney movies (no Star Wars, no Marvel) or things like John Wick, Valerian or The Fifth Element, so my mind couldn´t stop thinking about custom uhd movies, and that is how i end up here, in this forum and in this thread.

I´ll be looking forward for your tool, and yet again, thanks so much for your work.
Cesarleeds is offline   Reply With Quote
Old 26th December 2018, 12:00   #154  |  Link
ocean
Registered User
 
Join Date: Mar 2016
Posts: 60
Happy holidays to all, the first titles were released with HDR10+, change two bit-level flags in clpi, generally it is 4608, now there is 4736:

"ProgramInfo": {
"reserved_for_word_align": 0,
"program_sequence": [
{
"SPN_program_sequence_start": 0,
"program_map_PID": 256,
"reserved01": 0,
"stream_in_ps": [
{
"stream_PID": 4113,
"stream_coding_type": "H.265",
"video_format": "2160p",
"frame_rate": "23.976",
"aspect_ratio": "16:9",
"reserved01": 0,
"cc_flag": 0,
"reserved02": 4736,
"ISRC": {
"country_code": "",
"copyright_holder": "",
"recording_year": "",
"recording_number": "" ""

decimal
4736

hex
0X1280

binary
0001 0010 1000 0000

and mpls main:

pid": 4113,
"stream_coding_type": "H.265",
"video_format": "2160p",
"frame_rate": "23.976",
"reserved_uhd": 18,
"reserved01": 16384
ocean is offline   Reply With Quote
Old 28th December 2018, 15:58   #155  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,252
@ocean: Thanks.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net
jdobbs is offline   Reply With Quote
Old 3rd January 2019, 16:47   #156  |  Link
zizux
Registered User
 
Join Date: Mar 2018
Posts: 1
First of all thank your work, some of us hope you get the program out as soon as possible so we can use it.
I have a doubt. I use scenarist but when you generate a file that enters a bd50, with some movies there are problems since they have a higher bitrate than what the disc supports and the program doesn´t automatically calculate it.
Could your program do it?


Regards.
zizux is offline   Reply With Quote
Old 3rd January 2019, 21:56   #157  |  Link
UHD 4 Everyone
Registered User
 
Join Date: Dec 2018
Posts: 4
Quote:
Originally Posted by zizux View Post
First of all thank your work, some of us hope you get the program out as soon as possible so we can use it.
I have a doubt. I use scenarist but when you generate a file that enters a bd50, with some movies there are problems since they have a higher bitrate than what the disc supports and the program doesn´t automatically calculate it.
Could your program do it?


Regards.
I haven’t used scenarist since I’m not freaking rich. But I believe it takes whatever you give it as far as input media goes. Otherwise, it costs enough that it should be able to control what Bitrate you will render to. JDobbs program does not re-encode. TSM2UHD only parses through streams and modifies certain flags and other info to reflect UHD BD “compliance”

Have you reached out to their support for assistance, you pay enough for the software that you should be able to contact them and they will walk you through it.
UHD 4 Everyone is offline   Reply With Quote
Old 4th January 2019, 14:14   #158  |  Link
jdobbs
Moderator
 
Join Date: Oct 2001
Posts: 20,252
Quote:
Originally Posted by UHD 4 Everyone View Post
I haven’t used scenarist since I’m not freaking rich. But I believe it takes whatever you give it as far as input media goes. Otherwise, it costs enough that it should be able to control what Bitrate you will render to. JDobbs program does not re-encode. TSM2UHD only parses through streams and modifies certain flags and other info to reflect UHD BD “compliance”

Have you reached out to their support for assistance, you pay enough for the software that you should be able to contact them and they will walk you through it.
There is a 32 bit value in the CLPI file called TS_recording_rate. On the UHD discs I've looked at it is set to 109 Mbps (which is the maximum for a 66GB or 100GB disc). For most standard 50GB (or 25GB) discs used for UHD the maximum should be 64 Mbps (and 48 Mbps when used for non-UHD). It wouldn't surprise me if Scenarist always sets it to 109 (since almost all commercial discs are 66/100GB and Scenarist is used commercially).

TSM2UHD doesn't change that value, since TSMUXER doesn't set it that high. But, now that I think about it, it probably should be set to 64 Mbps since many people who are reencoding might use a higher maximum bitrate than they would on a standard 1080p BD.

If anyone is going to try and change it manually -- please note that the value is actually stored as bytes per second.
__________________
Help with development of new apps: Donations.
Website: www.jdobbs.net

Last edited by jdobbs; 4th January 2019 at 14:25.
jdobbs is offline   Reply With Quote
Old 15th January 2019, 17:47   #159  |  Link
Lucius Snow
Registered User
 
Join Date: Oct 2003
Location: Paris, France
Posts: 83
Quote:
Originally Posted by Ch3vr0n View Post
Afaik UHD requires UDF 2.60, 2.50 is for standard BD's.
Some news... Scenarist UHD builds UDF 2.50 image.
Lucius Snow is offline   Reply With Quote
Old 8th March 2019, 14:35   #160  |  Link
rpareja
Registered User
 
Join Date: Nov 2011
Posts: 15
First of all, i apologize for my ignorance.

I've been a BD Rebuilder fan for a long time and just until a couple of weeks ago I am entering the 4K HDR world (new 4K UHD HDR+ TV, new 4K HDMI cables and new 4K A/V Receiver).
Yesterday I read about this program TSM2UHD.

I have some 4K UHD HDR movies with a mkv format and I want to create a blu-ray structure for each one using TSMUXER.
I AM NOT planning to burn them and use a standalone player, but I am thinking of playing the blu-ray folder structure movie using PowerDVD on my 4K laptop and connect it to my A/V Receiver with an 4K HDMI cable to watch them.

When I load the mkv file on TSMUXER, it reports the correct information about resolution, audio, etc. of each track and seems to process everything correctly creating the blu-ray folder structure.

The questions I have are:
1) Does the process of creating the structure using TSMUXER is the correct approach I need?
2) Do I still need to process each movie folder (that were created with tsmuxer) using TSM2UHD?
3) Does TSMUXER loses something (in the video or audio that were used as input) during the process of creating the folder structure or it creates everything as it was?
4) Is this step using TSM2UHD to correct some information is only needed if I plan to burn the movie on a disc and play them in a standalone player?

Thanks for any help,

RPareja

Last edited by rpareja; 8th March 2019 at 16:03.
rpareja is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 08:19.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.