Blu-ray 3D subtitle depth

[bigotti5]

Just tested tsMuxeR

I suppose that offset metadata sequences are what tsMuxeR calls "3D-Planes". Correct?

Yes
Planes are offset sequences in SEI
Zerro sets 0xFF in mpls (== 2d sub)

[r0lZ]

Thanks a lot! I'll have a look tomorrow...

[HWK]

In the meantime I will work on Avatar, since it doesn't rely on tsmuxer offsets to work.

[Zorg]

Originally Posted by bigotti5

Just tested tsMuxeR

Yes
Planes are offset sequences in SEI
Zerro sets 0xFF in mpls (== 2d sub)

So the sequence is taken 1:1 from mpls when remuxing? What if there's only one (or less sequences than I intend to use), i.e. there's only "#0" and I use "#1", "#2", "#3"?

[minhjirachi]

I think we can't set the subtitle further on the tsmuxer. It allows to set the subtitle closer only. Is it right?

[bigotti5]

Selecting a plane does not mean further or closer.
It depends on what is defined in SEI message, you can only select a plane.
TsMuxeR let you select plane but neither shows values (further,closer) nor change these values.

[r0lZ]

Originally Posted by bigotti5

TsMuxeR let you select plane but neither shows values (further,closer) nor change these values.

It's my problem. I need a tool that can retrieve and display these values. Currently, I'm not familiar enough with the internal format of the MPLS and M2TS files to do it myself. Do you know a command line tool that can display that info? Or a site with info on how to compute the position of the tables myself (something similar to what mpucoder did for the IFO and VOB file formats)?

[bigotti5]

Did some investigations on SEI messages

Example is first project posted above

MVC scalable nesting SEI
...
4F 46 4D 44       - OFMD
81                - 1b marker_bit + 3b reserved + 4b framerate
                    framerate  1==23.976
                               6==50
                               7==59.94
                               only these are allowed in 3D
00 86 6F F9 80 A0 - 5b reserved + 36b PTS (3+15+15, 3 marker bits)  + 7b number of planes (not sure about # of planes)

                    (0000 0)000 (1)000 0110 0110 1111 (1)111 1001 1000 0000 (1)   010 0000  	
                       res       mb                    mb                    mb   # of planes
  
                    000000011001101111111100110000000 = 0x337F980 = 54.000.000 / 90.000 = 600 = 00:10:00.000

18                - number of frames (metadata are valid for one GOP only)
80                - number of PG offset sequences
00                - number of IG offset sequences

13 13 13 ....     - PG offset sequences (1 Byte per picture in GOP, display order)
                    1b - offset direction flag + 7b - offset value
					offset direction 0 = from display to the user side
					                 1 = behind display
					offset value - # of pixels to shift (0 to 127)

not sure for all but a beginning...
Now trying to descrample offset metadata part in extension data from mpls...

[r0lZ]

Nice findings! I have also deduced some of these assignments from the examples I have here, but I was not as precise as you. As far as I can tell, the "OFMD" string is always present, the number of frames is correct, as well as the offset sequences (the actual data I'm interested in). I think also that you are correct about the number of planes. At least, it's coherent with what I see here. Huge thanks for that finding. It's important for me, and I was wondering where that value was hidden! With bit 7 removed, it makes sense.

I haven't paid attention to the other values, as I don't need them.

I don't know what means "PG" and "IG". Can you explain? What are the IG offsets used for and when are they present in the message? (In my examples, the two numbers seem to be always 0x80 and 0x00, but I have to verify.)

Also, I'm a newbie to that "SEI messages" and I don't understand exactly what they contain (after the 0x00010625 and before the OFMD string). Can you point me to the right direction?

I have also to check if the values of the offsets are really displacements of the subpics in pixels along the X axis, but I suppose so. I'll try to apply the offsets to a concrete example to see if they act as expected.

[bigotti5]

PG = Presentation Graphics = Image Subtitles
IG = Interactive Graphics = Menu

SEI messages - MPEG2 TS Spec you can download pdf here
For 3D SEI messages look at Annex H.13

[r0lZ]

OK, thanks! I'm interested only in PG offsets, and I suppose that in regular movie, the number of IG offsets is always 0. Best to verify though. ;-)

[r0lZ]

I wrote a script to retrieve all 3d offsets from the Avatar blu-ray, and I have analysed the offsets.

There is something strange. When I extract the MVC stream with tsMuxeR, the log says "Processed 232,608 video frames". That's coherent with the duration of the movie and eac3to reports the same number of frames. But when I retrieve the 3d offsets (for any of the 7 3d-planes present in that BD), I get only 232,540 frames (68 frames less). Also, when I get a set of offsets corresponding to the frames during which a particular subtitle is displayed, the set contains often 2 different offset values. For example, it can be offset 4 during 20 frames, and then offset 3 during the remaining frames. I suppose it's not correct, as that change during the display of a subtitle means that it should jump from one plane to another when it is displayed. Or is the player supposed to take into account only the first value of the set?

The "cut" point where the jump occurs seems to change slowly from subtitle to subtitle, as if there was a sort of desync problem, similar to what we get when we mux an audio @25fps with a video @24 fps. I have tried to syncronize the offsets by using slightly different frame rates (24000/1001, 23.976 and 24), but the problem persists.

I don't have that problem when I extract the 3d-offsets from the 2 short examples you gave me, bigotti5. But they are very short sequences, and if there is a desync problem, it may not be noticeable.

I don't think it's a bug in my code (although of course it's possible), and I'm wondering if someone has an explanation for that strange things.

Also, strangely, the time code in the 3d-offset sei message of the first GOP corresponds to "01:10:00". A really strange time code for the beginning of a movie. Is it normal?

[bigotti5]

Also, when I get a set of offsets corresponding to the frames during which a particular subtitle is displayed, the set contains often 2 different offset values.

Every frame has its own offset value.
If you have 7 planes and 22 frames per SEI message you should see 154 values.
22 values for plane 1, followed by 22 values for plane 2, followed by 22 values for plane 3 .....

I suppose it's not correct, as that change during the display of a subtitle means that it should jump from one plane to another when it is displayed

Depth of subtitle is changed - not plane.
Plane is set in MPLS...

Also, strangely, the time code in the 3d-offset sei message of the first GOP corresponds to "01:10:00". A really strange time code for the beginning of a movie. Is it normal?

My release of avatar shows start PTS 00:00:02.000

[r0lZ]

Originally Posted by bigotti5

Every frame has its own offset value.
If you have 7 planes and 22 frames per SEI message you should see 154 values.
22 values for plane 1, followed by 22 values for plane 2, followed by 22 values for plane 3 .....

I know that. What's strange is that the depth change during the display of a particular subtitle. I was expecting the same depth values for all frames of a subtitle. (Here "particular subtitle" means one text string, not a subtitle stream.)

But I have experienced further, and I think I know now why the values change. First of all, I have discovered a bug in my code, and now, if I assume 24fps instead of 23.976, the changes of depth within a plane are coherent with the subtitles timings.

But there are still abrupt changes during the display of some subtitles. When I've watched the movie at that subtitles, I can see that the change in depth occurs at a cut, or when a character or an object appears in the image, in front of all other objects. For example, when the subtitle "Wait!" is displayed (during +/-2 seconds), there is a cut and the second shot is much closer to the spectator than the first one. That explains why the values are 4 during approx 1 seconds, and 17 during the next second. The cut occurs exactly when the values change. I suppose that that means that the subtitle is moved along the Z axis by the player when the values change, although I have never noticed this.

In my case, since I have to convert the subtitle for Side by Side or Top/Bottom, I have to pick one value only. Currently, my program picks the highest value, to be sure that no subtitle is displayed "inside" an object. My PC is computing a SBS rendering, and I will verify this evening if it's correct. In particular, I'm still not sure that the depth is equal to the displacement of the subtitle in the dep view along the X axis. I think it's the case, but I'll verify soon...

Originally Posted by bigotti5

Depth of subtitle is changed - not plane.
Plane is set in MPLS...

Oh, yes. It's what I mean.

Originally Posted by bigotti5

My release of avatar shows start PTS 00:00:02.000

I'll verify here. Maybe I did a mistake when I've converted it. Anyway, 00:00:02.000 is also strange. It should be 0:00:00.000 at the beginning of the movie, no?

[bigotti5]

It should be 0:00:00.000 at the beginning of the movie, no?

No - never
Just like on DVD you have to allow for muxing delay, buffers have to be filled etc.
DTS has to be < PTS , 0:00:00.000 would result in negative DTS.

Authoring programs take different values, Scenarist for example uses 00:10:00.000

[r0lZ]

OK, thanks. Maybe I'll use the time codes from the sei messages later. Currently, I count the frames, and when 24fps is assumed, that works pretty well.

I've just finished a fast SBS encode with hardcoded subtitles, and it seems that the subs are globally correct (except one, I have to verify why). However, they are usually a bit too near the screen, so IMO, it is better to add 2 to all 3D offsets, and they should be ideally placed. I'll do a new encode tomorrow, when I will have found why one subtitle is wrong. (I guess it's because the offsets are for English subs, and I've encoded the French forced subs. The sentence that doesn't work well is much longer in French, and passes inside objects in the right side of the image. It's a minor problem anyway.)