Blu-ray 3D subtitle depth

[r0lZ]

I would like to know how the 3D depth is assigned to each individual subtitle in a subtitle stream of a blu-ray 3D. I know for sure that it is possible to display each subtitle at different depths (or, in other words, at different positions along the Z axis). When the player must show a specific subtitle, it must place the subtitle at slightly different X positions in the left and right eye views, so that the paralax is correct to display the subtitle at the right Z depth. I don't know how the player is informed of the X offset or Z depth to use for each subtitle. I've read somewhere that that info is provided via some kind of metadata, but there was no more precision. I would like to know where that metadata is stored, and what is its format.

Thanks for any pointer...

[HWK]

Are you referring to stereo subtitle, which combined with regular subtitle is responsible for creating 3d subtitle.

I have shown example below, what make this different is it is not usable without reading original subtitle file. Similar to how MVC video works on 3D.

http://s17.postimg.org/45c0t5umn/Tsmuxer.png

Notice, 4608 and 4609 file they tell player how to take any subtitle not just english and choose it's placing over screen to create 3D effect. Basically 4608 is one file controlling left placement and 4609 is another which control right placement. When player play subtitle in 3D it reads three files as far subtitle is concerned for this movie at least. Three files which it reads are user selected subtitle file located in base view most of the time and 4608 and 4609 file.

How they stored on disc, depends on authoring. Currently I know only two ways of storing subtitle metadata.

1. Directly inside stereo stream, not very popular though. Even with this method there are multiple ways to store it. For example Avatar, Resident Evil: Retribution both use this method, however how they store it is different. Avatar uses offset data to store and Resident Evil: Retribution doesn't.

2. Controlling via MPLS file.

This is from AVATAR stereo stream, notice subtitle file. Actually there are metadata file not actual subtitle.

General
ID : 0 (0x0)
Complete name : J:\BDMV\STREAM\00002.m2ts
Format : BDAV
Format/Info : Blu-ray Video
File size : 12.8 GiB
Duration : 2h 41mn
Overall bit rate mode : Variable
Overall bit rate : 11.4 Mbps
Maximum Overall bit rate : 48.0 Mbps

Video
ID : 4114 (0x1012)
Menu ID : 1 (0x1)
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Stereo High@L4.1
MultiView_Count : 2
Format settings, CABAC : Yes
Format settings, ReFrames : 3 frames
Codec ID : 32
Duration : 2h 41mn
Bit rate : 10.9 Mbps
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 23.976 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.219
Stream size : 12.3 GiB (96%)

Text #1
ID : 4608 (0x1200)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144
Duration : 2h 41mn
Delay relative to video : 29s 363ms

Text #2
ID : 4609 (0x1201)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144
Duration : 2h 41mn
Delay relative to video : 29s 363ms

If you were to extracted it, it would look like one below. Notice subclip size and compare it to regular subtitle. While we are it I want to let you know though extension is same, but data inside them is not.

http://s21.postimg.org/b7jjwqskn/Tsmuxer2.png

On Final note I use 00854.mpls to extract, but this lacks chapter information which is in 00850. 00850.mpls lacks 3D subtitle info, how player managed to display subtitle in 3D and still have access to chapters is interesting and I am working on it.

[r0lZ]

Thanks for the info. Seems very interesting. However, I don't understand it fully.

Yes, I'm trying to extract the depth information, to recreate the subtitles for a side-by-side or top-bottom conversion of a 3D BD. The subtitles have to be combined the same way, but the X offset is not easy to figure out.

If I understand correctly, in your example, PGS streams #0 to #6 all use the same metadata file, stored in streams with IDs 4608 and 4609. That seems strange as stream 4608's language code is "eng" and 4609 is "fre". For me, they are just different streams in different language. Do you mean that they contain also the 3D offsets to apply to all other subtitles streams? So, if the player must display, say, the Italian subtitles, it has to read, of course, the graphics from the "ita" stream #4611, plus the offsets from the "eng" and "fre" streams #4608 and #4609? Correct? If it's the case, why are the 4608 and 4609 numbers printed only with the last PGS stream ("eng" #4614) in the tsMuxeR output?

Also, what is the meaning of the "3D plane: zero"? I suppose that in this example, since the 3D plane is always #0, that means that all streams use the same 3D offsets. Correct?

Anyway, I have already noticed the subtitle streams in the MVC M2TS file in the Avatar DB. But when I try to display it with BDSup2Sub, that stream seems to be exactly identical to the stream stored with the AVC M2TS. The positions of the subtitles are identical in both files, and if there is some additional metadata in the subtitle stored in the MVC stream, BDSup2Sub can't use it. I guess I'll have to bit-compare them one by one. Any idea of the way that offsets are stored in the stream?

Also, if Avatar and some other BDs have subtitle streams in the AVC and MVC M2TS files, most 3D BDs have subtitles only in the AVC M2TS. There must therefore be somewhere a metadata stream with the Z depth or X offset values. Any idea?

Anyway, thanks again. I'll have a look to the Avatar 3DBD tomorrow...

[HWK]

If I understand correctly, in your example, PGS streams #0 to #6 all use the same metadata file, stored in streams with IDs 4608 and 4609. That seems strange as stream 4608's language code is "eng" and 4609 is "fre".

This code of PGS is not relevant, same file can be used with all subtitle. When I was working with Roman to add support we came to conclusion. So It goes like this

Subtitle + 4608(left depth) +4609 (right depth) = 3D subtitle.


For me, they are just different streams in different language. Do you mean that they contain also the 3D offsets to apply to all other subtitles streams?

It applies to all language, it has nothing to do with language code of stream. However in Resident Evil: Retribution there are 9 subtitle and also there are 9 more in dependent view. In this case you theory about having each file will only work with that one only is correct, basically there are two streams for each subtitle language. I can post info from that as well if you want to compare.

So, if the player must display, say, the Italian subtitles, it has to read, of course, the graphics from the "ita" stream #4611, plus the offsets from the "eng" and "fre" streams #4608 and #4609? Correct?

Yes, it is correct. Remember name of 4608 and 4609 doesn't matter.

If it's the case, why are the 4608 and 4609 numbers printed only with the last PGS stream ("eng" #4614) in the tsMuxeR output?

Roman, can answer this better. Since If you open avatar actual mpls for that is 00850, and in this case you wouldn't see 4614 (S_HDMV/PGS, "J:\BDMV\PLAYLIST\00854.mpls", fps=23.976, track=4614, lang=eng) in output of tsmuxer and as a result you would not see stereo stream either.

Also, what is the meaning of the "3D plane: zero"? I suppose that in this example, since the 3D plane is always #0, that means that all streams use the same 3D offsets. Correct?

Roman, is working on document to explain this and other option better, after you asked about documentation.

Anyway, I have already noticed the subtitle streams in the MVC M2TS file in the Avatar DB. But when I try to display it with BDSup2Sub, that stream seems to be exactly identical to the stream stored with the AVC M2TS. The positions of the subtitles are identical in both files, and if there is some additional metadata in the subtitle stored in the MVC stream, BDSup2Sub can't use it.

Correct, BDSup2Sub can't read it. You will get message please turn on you 3D glasses... for each frame and yes it applies to each file. Instead of subtitle you would expect from base view file you normally see. (also which mpls you used to open Avatar if at all)

I guess I'll have to bit-compare them one by one. Any idea of the way that offsets are stored in the stream?

Also, if Avatar and some other BDs have subtitle streams in the AVC and MVC M2TS files, most 3D BDs have subtitles only in the AVC M2TS. There must therefore be somewhere a metadata stream with the Z depth or X offset values. Any idea?

Anyway, thanks again. I'll have a look to the Avatar 3DBD tomorrow...

When player plays in this case it reads 00850.mpls, since it has chapters info and other info as well. However stereo subtitle will not appear in 00850.mpls file, but they can be found under 00854.mpls file but no chapters.

I am working on how to have everything together.

In the mean time I am uploading screenshots to help you further in understanding this matter.

First and second screenshot of normal subtitle file, From Avatar.

http://s16.postimg.org/rjtk4htfp/dsg.png
http://s23.postimg.org/3tgd96biz/3423.png

Second two from Avatar as well, this time it is from stereo subtitle stream.

http://s22.postimg.org/8wjqxty3l/dsg2.png
http://s9.postimg.org/6035n0s0f/dsg3.png

Notice change in time stamp and how subtitle changes in base view (first two screenshots), but dependent view subtitle remain same (last two screenshot) and no forced found in dependent view subtitle stream. Even though Avatar has close to 81 forced.

* This one show from 4608 file which control left depth, but right file is same as well.

[r0lZ]

OK, thanks. It's still very confuse for me, but it's becoming more clear.

It applies to all language, it has nothing to do with language code of stream. However in Resident Evil: Retribution there are 9 subtitle and also there are 9 more in dependent view. In this case you theory about having each file will only work with that one only is correct, basically there are two streams for each subtitle language. I can post info from that as well if you want to compare.Yes, please do it. It seems that there are many different cases, and I need to know them all.

Roman, can answer this better. Since If you open avatar actual mpls for that is 00850, and in this case you wouldn't see 4614 (S_HDMV/PGS, "J:\BDMV\PLAYLIST\00854.mpls", fps=23.976, track=4614, lang=eng) in output of tsmuxer and as a result you would not see stereo stream either.

Roman, is working on document to explain this and other option better, after you asked about documentation.Yes, I can see that most MPLS files that reference 00001.ssif do not have that info. (It's also the case of 00800.mpls, that I've used to do my tests.)

It's damn complex. How can I know in advance what MPLS I must analyse? In my BD, there are 10 MPLS files referencing 00001.ssif, and only two of them have that stereo information: 00804.mpls and 00854.mpls. I notice also that that 2 mpls files have all 3d-plane values set to zero, but the other mpls files have non-zero 3d-plane values. Anyway, I'll wait for the updated doc to try to understand the usage of that 3d-plane value.

When player plays in this case it reads 00850.mpls, since it has chapters info and other info as well. However stereo subtitle will not appear in 00850.mpls file, but they can be found under 00854.mpls file but no chapters.I see. To know everything it is therefore necessary to get information from different MPLS files! It's a real nightmare.

I am working on how to have everything together.What do you mean? Will it be possible to see the whole information about a specific title in one shot? Currently, it seems difficult to parse the output of tsMuxeR.

In the mean time I am uploading screenshots to help you further in understanding this matter.

First and second screenshot of normal subtitle file, From Avatar.

Second two from Avatar as well, this time it is from stereo subtitle stream.

Notice change in time stamp and how subtitle changes in base view (first two screenshots), but dependent view subtitle remain same (last two screenshot) and no forced found in dependent view subtitle stream. Even though Avatar has close to 81 forced.

* This one show from 4608 file which control left depth, but right file is same as well.I see. I begin to understand how it works. But I still don't know hot the depth is stored in that "control files". Have you an idea of the file format?

[HWK]

Yes, please do it. It seems that there are many different cases, and I need to know them all.


This is from Resident Evil Base View Media Info

Text #1
ID : 4608 (0x1200)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144
Delay relative to video : 2s 961ms

Text #2
ID : 4609 (0x1201)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144

Text #3
ID : 4610 (0x1202)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144

Text #4
ID : 4611 (0x1203)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144

Text #5
ID : 4612 (0x1204)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144

Text #6
ID : 4613 (0x1205)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144

Text #7
ID : 4614 (0x1206)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144

Text #8
ID : 4615 (0x1207)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144

Text #9
ID : 4616 (0x1208)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144



This is from Resident Evil Dependent View Media Info

Text #1
ID : 4608 (0x1200)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144
Delay relative to video : 2s 961ms

Text #2
ID : 4609 (0x1201)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144

Text #3
ID : 4610 (0x1202)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144

Text #4
ID : 4611 (0x1203)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144

Text #5
ID : 4612 (0x1204)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144

Text #6
ID : 4613 (0x1205)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144

Text #7
ID : 4614 (0x1206)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144

Text #8
ID : 4615 (0x1207)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144

Text #9
ID : 4616 (0x1208)
Menu ID : 1 (0x1)
Format : PGS
Codec ID : 144



Notice how they have same number of subtitle files. I have extracted subtitle from Resident Evil: Retribution for this method to work, I had to use individual m2ts files. Since tsmuxer can't extract subtitle from dependent view, if they are done like this.

http://s24.postimg.org/y0cmjaxmd/Resident_Evil.jpg

[Zorg]

What happens, when I choose srt file to render it for 3D BD? There's no "3D plane" dropbox for srt though...
What happens with 2D sup files muxed to 3D BD, for which I can choose a "3D plane" even when they were never 3D?

[r0lZ]

Thanks HWK. I don't have Resident Evil, but I have other BDs with the same subtitle streams in the base and dep view M2TS. I have tried to understand how that streams are used but I don't understand. When I extract the subtitles from the base view and from the dep view (with tsMuxeR or eac3to) and then I compare them, they are EXACTLY identical, bit per bit! That means that there is no difference between the left and right subtitles, and that the depth offsets are not saved in each file. Also, I wonder what is the advantage in duplicating a subtitle streams in the MVC m2ts, since it is already present and totally identical in the AVC m2ts.

Can you confirm that tsMuxeR doesn't remove that 3D depth information when it demuxes the subtitle streams?

Can you explain where and how exactly that information is stored in the subtitle file? What is its format?

Also, any chance to have an explanation about the 3D-planes mystery?


@Zorg: As you have noticed, I'm new to this stuff, and I don't understand it yet. But I think that the 3D-depth information is saved as offsets, and you can have several "sets" of 3D offsets, for different subtitle streams. On a BD, all subtitles can share the same set, and on another, each language can have its own set. Perhaps the 3D-plane settings is used to specify the "set" that a specific subtitle stream must use, but of course, I'm not sure. I don't think you can specify a 3D depth for SRT subtitles, but again, I may be wrong.

[Zorg]

No, it's not possible in tsmuxer. However when it is possible for pgs I don't see (so far) why not to enable it for srt as well. Good thing is, pgs from 2D bluray muxed into 3D are then for both eyes :) I could make some experiments with the same subtitle and changed 3d plane,if I find some time... :)
Perhaps physic or hwk should explain, how the 3d subtitle engine in tsmuxer works. We would appreciate it much :rolleyes:
But so far - according to my (and yours) experiments - my best bet is that the subtitles are rendered just once to the ssif/m2ts stream, the player then uses them for both eyes, and there is some (magical) flag - 3D plane - which stores the depth aka paralax (L/R distance)...

[r0lZ]

I don't think it's simply a flag. It's probably a table of paralax values, with at least one value per subtitle. It's necessary, as the Z position of a specific subtitle depends of the depth of its particular shot.
I have tried to discover where that table is stored in the subtitle streams, but currently without success. I hope also that physic or HWK will be able to explain...

[HWK]

Ok, I just recompiled disc again and I will share results with you folks. In my case for resident evil subtitle extract from mvc is same size, but contents are not same and yes BDSuptoSub, eac3to and tsmuxer will not able to distinguish difference in them. None of them are able read it correctly at all, all of them reads 2D portion of stream, though they are fully functional 3D according to TMT, powerdvd and other players. Simply put for each stream in base view there is one in MVC stream as well which is sup file with 3D effect applied already to it, similar to hardcoding subtitle in video. What makes this is interesting it doesn’t rely on offset for 3D to work. You can confirm my findings if you can access such a stream and rebuild blu-ray disc or any other disc and use media info to see that subtitle are in base view where they should be in dependent view. Tsmuxer can’t handle such subtitle and treat it as a normal subs, instead of 3D subtile and putting in dependent view. During playback it is different story altogether, since they will show in 3D if taken from MVC view and 2D if taken form AVC view. If you have both tsmuxer gone put both with one in 3D and one in 2D, depends on order of importing.

Player decides which subtitle to decode based on playback environment. Which explain why 3D subtitle are attached to dependent view and why nine of them for resident evil.

If you look other way, when playing in 2D player reads base view for all assets, however in 3D except audio which is still coming from base view. All other assets are read from avc and mvc as well.

2D
Video +Audio +Subtitle (all from base view)

3D
Video(Avc) + Video(MVC) + Audio(Base file) + Subtitle(Dependent view)

Disclaimer: This apply to Resident Evil only, others do differently like Avatar.

Let say we have 0001.sup in English and 0001.sup also in English, although both file are in SUP, but second one is in 3D where first one is in 2D. Offset is calculated during authoring and final result is in sup format with 3D part done.

Only difference in my case is second one is stored in mvc view, not avc, so players can adjust based on what to play.

In regards 3d offset I have to ask Roman about this or do some work on my own. As you can see there is no single way off handling 3D subtitle for movie.

[r0lZ]

OK, I see. But how is it possible that in my BD, the streams in MVC view file are identical to the ones in AVC? The DOS command COMP reports no differences at all. I'm confused. Is it because tsMuxeR cannot extract the 3D part of the stream?

I will try to find Resident Evil to analyse it. In the meantime, can you explain how Avatar works? It seems even more complicated.

[HWK]

I'm confused. Is it because tsMuxeR cannot extract the 3D part of the stream?


Sort off, tsmuxer on it's own with mpls file will not able to read subtitle from dependent view. However if done separately then it will read it and extract sup file fine, which already has 3D applied to it. Actual problem comes when putting it back or creating disc structure, then it regards as a 2d subtitle, though it is still in 3D and put it in base view instead of dependent view. Players still play in 3D if stream is taken form mvc, despite tsmuxer putting in base view.

This is from post 6
Notice how they have same number of subtitle files. I have extracted subtitle from Resident Evil: Retribution for this method to work, I had to use individual m2ts files.

If mpls file is used, in that case it will not extract 9 subtitle which are in 3D from dependent view stream. Do it separately then it works, but problem lies with authoring phase in tsmuxer not able to distinguish Sup file in 3D or 2D. Since both are sup files with identical size and attributes, but they are not same by any means.

[r0lZ]

Yes, I've noticed that txMuxeR can extract the subtitle streams from the dep view M2TS. It's what I did for my tests, but as I wrote above, they were identical to the corresponding stream in the base view M2TS. For that reason, I have demuxed them again from the two views but this time with eac3to. Same result: the files from both views are identical. Maybe I've used a badly authored BD.

Note that strangely, the files produced by eac3to and the files produced by tsMuxeR are different! I don't know why, but anyway, in both cases, the base and dep files were identical. As a consequence, I can't compare them to discover where and how the 3D info is stored. I will investigate with other BDs...

[HWK]

they were identical to the corresponding stream in the base view M2TS. For that reason, I have demuxed them again from the two views but this time with eac3to. Same result: the files from both views are identical. Maybe I've used a badly authored BD.

Note that strangely, the files produced by eac3to and the files produced by tsMuxeR are different! I don't know why, but anyway, in both cases, the base and dep files were identical. As a consequence, I can't compare them to discover where and how the 3D info is stored. I will investigate with other BDs...

Just heads up, file I have send you is same in size. If open with BDsupto sub you may not able to tell difference. When I remuxed back and choose to play in full disc mode (iso) TMT 5 displayed subtitle in 3D.

[r0lZ]

In my tests, I've compared the files with the DOS command COMP, and nothing was different, so I'm sure it is useless to swap them. But I did my tests on another BD. I'll test your files the same way, and other BDs, if I can find some time.

I need to know where the differences are, to try to understand how the 3D works. I need that for my BD3D2MK3D (http://forum.doom9.org/showthread.php?p=1596672#post1596672) project. Currently, it can convert existing subtitles to 3D SBS or Top/Bottom, but you have to give a constant 3D depth, and it is difficult to figure out if it is correct without encoding the whole file. If I can understand how the BD players know at what depth (or with what parallax offsets) each sub must be displayed, I'll be able to convert them much better, without user intervention.

[r0lZ]

Just heads up, file I have send you is same in size. If open with BDsupto sub you may not able to tell difference. When I remuxed back and choose to play in full disc mode (iso) TMT 5 displayed subtitle in 3D.
There must be something else, because the two subtitle files you sent me are absolutely identical. No differences at all, as confirmed by the COMP command. I have even written a little program to compare them myself, as I'm not totally confident in the way Microsoft has programmed its COMP command. Same result with my program: both files are absolutely identical.

So, the info about the parallax offsets is probably not stored in the subtitle stream itself, or, it it's the case, it is identical in the two versions of the stream, or it is stripped by tsMuxeR.

I'm stuck now, and I still need some useful info on how the subtitle 3D depth info is stored in the blu-rays. I hope physic will read this thread and explain. Thanks anyway for having tried to help.

[bigotti5]

AFAIK subtitle depth is generated by using predefined (up to 32) offset metadata sequences stored in dependent view stream as mvc_scalable_nesting SEI message (0x000000010625, ts transport priority flag "on").
In extension data of mpls (STN_table_ss) pg stream points to specific offset sequence in SEI.

[r0lZ]

Thanks bigotti5. Nice to see you're still active.
I suppose that offset metadata sequences are what tsMuxeR calls "3D-Planes". Correct?
I still need to understand how a specific subtitle is tied to the corresponding offset in the sequence, but I'm on the way. And do you know if the offset is simply a displacement of the subtitle in pixels along the X axis, or something else?
Thanks again.

[bigotti5]

Not familiar with new tsMuxeR.
Just created two 3d BDs with differences in offset sequences only.
Here (https://www.dropbox.com/s/flk18l4d5cc1enh/3D_DIFF.rar) (ca 195 MB)

First project:
offset sequence 1: -19 (closer)
offset sequence 2: +19 (further)
ger stream points to seq 1
eng stream points to seq 2

Second project:
offset sequence 9: -4 (closer)
offset sequence 10: +4 (further)
ger stream points to seq 9
eng stream points to seq 10

diff mpls
http://members.aon.at/video.digital/Diff_mpls.png

diff m2ts
http://members.aon.at/video.digital/Diff_m2ts.png

[bigotti5]

Just tested tsMuxeR
I suppose that offset metadata sequences are what tsMuxeR calls "3D-Planes". Correct?
Yes
Planes are offset sequences in SEI
Zerro sets 0xFF in mpls (== 2d sub)

[r0lZ]

Thanks a lot! I'll have a look tomorrow...

[HWK]

In the meantime I will work on Avatar, since it doesn't rely on tsmuxer offsets to work.

[Zorg]

Just tested tsMuxeR

Yes
Planes are offset sequences in SEI
Zerro sets 0xFF in mpls (== 2d sub)

So the sequence is taken 1:1 from mpls when remuxing? What if there's only one (or less sequences than I intend to use), i.e. there's only "#0" and I use "#1", "#2", "#3"?

[minhjirachi]

I think we can't set the subtitle further on the tsmuxer. It allows to set the subtitle closer only. Is it right?

[bigotti5]

Selecting a plane does not mean further or closer.
It depends on what is defined in SEI message, you can only select a plane.
TsMuxeR let you select plane but neither shows values (further,closer) nor change these values.

[r0lZ]

TsMuxeR let you select plane but neither shows values (further,closer) nor change these values.It's my problem. I need a tool that can retrieve and display these values. Currently, I'm not familiar enough with the internal format of the MPLS and M2TS files to do it myself. Do you know a command line tool that can display that info? Or a site with info on how to compute the position of the tables myself (something similar to what mpucoder did for the IFO and VOB file formats)?

[bigotti5]

Did some investigations on SEI messages

Example is first project posted above

MVC scalable nesting SEI
...
4F 46 4D 44 - OFMD
81 - 1b marker_bit + 3b reserved + 4b framerate
framerate 1==23.976
6==50
7==59.94
only these are allowed in 3D
00 86 6F F9 80 A0 - 5b reserved + 36b PTS (3+15+15, 3 marker bits) + 7b number of planes (not sure about # of planes)

(0000 0)000 (1)000 0110 0110 1111 (1)111 1001 1000 0000 (1) 010 0000
res mb mb mb # of planes

000000011001101111111100110000000 = 0x337F980 = 54.000.000 / 90.000 = 600 = 00:10:00.000

18 - number of frames (metadata are valid for one GOP only)
80 - number of PG offset sequences
00 - number of IG offset sequences

13 13 13 .... - PG offset sequences (1 Byte per picture in GOP, display order)
1b - offset direction flag + 7b - offset value
offset direction 0 = from display to the user side
1 = behind display
offset value - # of pixels to shift (0 to 127)


not sure for all but a beginning...
Now trying to descrample offset metadata part in extension data from mpls...

[r0lZ]

Nice findings! I have also deduced some of these assignments from the examples I have here, but I was not as precise as you. As far as I can tell, the "OFMD" string is always present, the number of frames is correct, as well as the offset sequences (the actual data I'm interested in). I think also that you are correct about the number of planes. At least, it's coherent with what I see here. Huge thanks for that finding. It's important for me, and I was wondering where that value was hidden! With bit 7 removed, it makes sense.

I haven't paid attention to the other values, as I don't need them.

I don't know what means "PG" and "IG". Can you explain? What are the IG offsets used for and when are they present in the message? (In my examples, the two numbers seem to be always 0x80 and 0x00, but I have to verify.)

Also, I'm a newbie to that "SEI messages" and I don't understand exactly what they contain (after the 0x00010625 and before the OFMD string). Can you point me to the right direction?

I have also to check if the values of the offsets are really displacements of the subpics in pixels along the X axis, but I suppose so. I'll try to apply the offsets to a concrete example to see if they act as expected.

[bigotti5]

PG = Presentation Graphics = Image Subtitles
IG = Interactive Graphics = Menu

SEI messages - MPEG2 TS Spec you can download pdf here (http://www.itu.int/rec/T-REC-H.264-201304-I/en)
For 3D SEI messages look at Annex H.13

[r0lZ]

OK, thanks! I'm interested only in PG offsets, and I suppose that in regular movie, the number of IG offsets is always 0. Best to verify though. ;-)

[r0lZ]

I wrote a script to retrieve all 3d offsets from the Avatar blu-ray, and I have analysed the offsets.

There is something strange. When I extract the MVC stream with tsMuxeR, the log says "Processed 232,608 video frames". That's coherent with the duration of the movie and eac3to reports the same number of frames. But when I retrieve the 3d offsets (for any of the 7 3d-planes present in that BD), I get only 232,540 frames (68 frames less). Also, when I get a set of offsets corresponding to the frames during which a particular subtitle is displayed, the set contains often 2 different offset values. For example, it can be offset 4 during 20 frames, and then offset 3 during the remaining frames. I suppose it's not correct, as that change during the display of a subtitle means that it should jump from one plane to another when it is displayed. Or is the player supposed to take into account only the first value of the set?

The "cut" point where the jump occurs seems to change slowly from subtitle to subtitle, as if there was a sort of desync problem, similar to what we get when we mux an audio @25fps with a video @24 fps. I have tried to syncronize the offsets by using slightly different frame rates (24000/1001, 23.976 and 24), but the problem persists.

I don't have that problem when I extract the 3d-offsets from the 2 short examples you gave me, bigotti5. But they are very short sequences, and if there is a desync problem, it may not be noticeable.

I don't think it's a bug in my code (although of course it's possible), and I'm wondering if someone has an explanation for that strange things.

Also, strangely, the time code in the 3d-offset sei message of the first GOP corresponds to "01:10:00". A really strange time code for the beginning of a movie. Is it normal?

[bigotti5]

Also, when I get a set of offsets corresponding to the frames during which a particular subtitle is displayed, the set contains often 2 different offset values.

Every frame has its own offset value.
If you have 7 planes and 22 frames per SEI message you should see 154 values.
22 values for plane 1, followed by 22 values for plane 2, followed by 22 values for plane 3 .....

I suppose it's not correct, as that change during the display of a subtitle means that it should jump from one plane to another when it is displayed
Depth of subtitle is changed - not plane.
Plane is set in MPLS...

Also, strangely, the time code in the 3d-offset sei message of the first GOP corresponds to "01:10:00". A really strange time code for the beginning of a movie. Is it normal?
My release of avatar shows start PTS 00:00:02.000

[r0lZ]

Every frame has its own offset value.
If you have 7 planes and 22 frames per SEI message you should see 154 values.
22 values for plane 1, followed by 22 values for plane 2, followed by 22 values for plane 3 .....
I know that. What's strange is that the depth change during the display of a particular subtitle. I was expecting the same depth values for all frames of a subtitle. (Here "particular subtitle" means one text string, not a subtitle stream.)

But I have experienced further, and I think I know now why the values change. First of all, I have discovered a bug in my code, and now, if I assume 24fps instead of 23.976, the changes of depth within a plane are coherent with the subtitles timings.

But there are still abrupt changes during the display of some subtitles. When I've watched the movie at that subtitles, I can see that the change in depth occurs at a cut, or when a character or an object appears in the image, in front of all other objects. For example, when the subtitle "Wait!" is displayed (during +/-2 seconds), there is a cut and the second shot is much closer to the spectator than the first one. That explains why the values are 4 during approx 1 seconds, and 17 during the next second. The cut occurs exactly when the values change. I suppose that that means that the subtitle is moved along the Z axis by the player when the values change, although I have never noticed this.

In my case, since I have to convert the subtitle for Side by Side or Top/Bottom, I have to pick one value only. Currently, my program picks the highest value, to be sure that no subtitle is displayed "inside" an object. My PC is computing a SBS rendering, and I will verify this evening if it's correct. In particular, I'm still not sure that the depth is equal to the displacement of the subtitle in the dep view along the X axis. I think it's the case, but I'll verify soon...

Depth of subtitle is changed - not plane.
Plane is set in MPLS...Oh, yes. It's what I mean.

My release of avatar shows start PTS 00:00:02.000I'll verify here. Maybe I did a mistake when I've converted it. Anyway, 00:00:02.000 is also strange. It should be 0:00:00.000 at the beginning of the movie, no?

[bigotti5]

It should be 0:00:00.000 at the beginning of the movie, no?

No - never
Just like on DVD you have to allow for muxing delay, buffers have to be filled etc.
DTS has to be < PTS , 0:00:00.000 would result in negative DTS.

Authoring programs take different values, Scenarist for example uses 00:10:00.000

[r0lZ]

OK, thanks. Maybe I'll use the time codes from the sei messages later. Currently, I count the frames, and when 24fps is assumed, that works pretty well.

I've just finished a fast SBS encode with hardcoded subtitles, and it seems that the subs are globally correct (except one, I have to verify why). However, they are usually a bit too near the screen, so IMO, it is better to add 2 to all 3D offsets, and they should be ideally placed. I'll do a new encode tomorrow, when I will have found why one subtitle is wrong. (I guess it's because the offsets are for English subs, and I've encoded the French forced subs. The sentence that doesn't work well is much longer in French, and passes inside objects in the right side of the image. It's a minor problem anyway.)