PDA

View Full Version : .bdmv, .clpi, .mpls and .m2ts - Exact specs of AVCHD/BD structure and files


kaid
22nd September 2008, 17:55
Okay, since I was simply unable to find anything specific on the structure and binary contents of the files on a BD/AVCHD, i thought it warrants its own thread (sorry if there already is one, I couldn't find anything upon searching!).

On making my AVCHD menu generator (http://forum.doom9.org/showthread.php?t=141173) I learned a few things about the structure and the files, but the majority of the bytes remain a mystery to me. Since it's near impossible to get roman76 to react to queries we also don't know how he generates his files in tsMuxer when using the BD mux option. The official AVCHD Info site (http://www.avchd-info.org/) doesn't tell you shit about exact specs, all they give you is some vague info how the video streams should look like..

So here's my findings:
1) Structure:
Root is always the folders BDMV and CERTIFICATE. On FAT32 this needs to be within a folder called AVCHD and on optical media BDMV and CERTIFICATE need to be on the root level.
CERTIFICATE seems to be mandatory (probably for keyfiles) and contains an (empty) BACKUP folder for redundancy, AFAIK it is always empty for self-authored discs.

BDMV is where it's at, and in there we have:
AUXDATA: Auxilliary Data judging by the name. Probably bonus material for your computer like screensavers etc goes here.
BACKUP: Contains backups of the CLIPINFO and PLAYLIST directories along with the two .bdmv files in the BDMV directory. For redundancy
BDJO: Probably has to do with the Bluray Java, but I'm not sure in how far. This NES emulator for PS3 has a .bdjo file in there
CLIPINF: This contains one .clpi file for each .m2ts stream (also split ones) of the same name with exact specs about the stream
JAR: The Java Applets probably go in here.
META: Dunno. Metadata? D'oh, what kind of Metadata?
PLAYLIST: Contains a .mpls file for every track (not stream!) on the disc. So if a disc made with my script has 4 tracks, there's gonna be 4 mpls files in there, one for each, no matter into how many chunks the stream was split up (plus there must be -also with m2ts and clpi- 00000.mpls, which is what the BD/AVCHD starts with, in my case the menu track!)
STREAM: All .m2ts streams go in here. These are the actual content!

This structure must be followed 100% as far as I know, so even though they might be empty all these directories need to be there in this structure, with the exact names in caps!

2) Differences between AVCHD FAT32/UDF:
The only difference between AVCHD stored on a FAT32 volume and optical media lies in the filenames, because 8.3 naming is used on FAT32, and it's all caps:
*.clpi (UDF) becomes *.CLP (FAT32), *.mpls becomes *.MPL, *.m2ts becomes *.MTS, MovieObject.bdmv becomes MOVIEOBJ.BDM and index.bdmv becomes INDEX.BDM
Even though the .mpls files reference the .m2ts files internally, it seems there is some fault tolerance, probably required by the spec, so there's no need to change the string in the .mpls files accordingly from M2TS to MTS and vice versa.

3) File contents:
Unfortunately I was only able to find out something about the structure of .mpls playlist files by looking at it in a binary editor:
First there's a header, "MPLS0200", which is probably a file identifier with version number. Some bytes later (not always the same offset!), the M2TS file that is played first is referenced, (aways with a P infront).. Around 60 bytes later seems to be the language code, preceded with an "a" (so e.g. it's aeng for english). All of this minus the header is repeated in the same mpls file if one track has more chunks to be played sequentially.
The rest of the bytes is mostly zeroes, so reverse-engineering it should be possible...
.clpi files don't give much away, there's a header again (HDMV0200) and further down after lots of zeroes there's that language code with 'a' infront again..
index.bdmv is the same thing: Header (INDX0100) and mostly zeros...
MovieObject.bdmv: Header (MOBJ0100) and mostly zeros...

So. Does anyone know where to get further specs about this? I'd especially like to know how the menu m2ts (00000.m2ts) is structured so I could inject my own graphics in the menu generator! ;-) I would guess it is rather similar to the structure of those VTSx_0.VOBs on a DVD that contain the track's menu gfx!...

kaid
24th September 2008, 14:14
Something else that would be of great interest are the contents of .clpi files, as I would like to be able to generate my own, so all that's needed for my script is m2ts files as input...

I just found this (http://www.freepatentsonline.com/EP1821310.html), a patent for recordable BDs that tells a lot about the filestructure, very very interesting:

An example of the above-mentioned file structure is shown in FIG. 2. The file structure according to the present invention includes one or more BD directories (BDMV) in a single root directory. The BD directory includes an index file "index.bdmv" and an object file "MovieObject.bdmv" acting as a general file capable of ensuring user interactivity. The file structure includes a variety of directories for storing information of actual data recorded in a disc and other information associated with a method for reproducing the data, for example, a playlist directory (PLAYLIST), a clip information directory (CLIPINF), a stream directory (STREAM), and an auxiliary directory (AUXDATA). The above-mentioned directories and a variety of files included in the directory will hereinafter be described.

The AUXDATA directory includes an additional data file for reproducing data of the disc. For example, the AUXDATA directory includes a "Sound.bdmv" file for providing a user with sound data when an interactive graphic function is executed, and an "aaaaa.otf" file for providing the user with font information when data of the disc is reproduced. For the convenience of description, the AUXDATA directory will herein be omitted in figures since FIG. 6A.

The stream directory (STREAM) includes a plurality of AV (Audio and Video) stream files recorded in a disc according to a specific format, and also includes a file associated with interactive graphic streams. Generally, individual streams are recorded using an MPEG-2 based transport packet. Therefore, the stream directory (STREAM) uses extension names of stream files (1111.m2ts, 2222.m2ts, and 33333.m2ts) as a specific extension name "*.m2ts". The above-mentioned stream may also be called a clip file in the BD standard. Particularly, if video/audio/graphic information from among the above-mentioned streams are multiplexed, the multiplexed information is called an AV stream, and a single title is configured as at least one AV stream file.

The clip information (Clip-info) directory (CLIPINF) is composed of a plurality of clip-info files (11111.clpi, 22222.clpi, and 33333.clpi) connected to the above-mentioned stream files on a one-to-one basis. Particularly, the clip-info files (*.clpi) record attribute information and timing information of the stream files (i.e., clip files) therein. The clip-info files (*.clpi) connected to the stream files (*.m2ts) on a one-to-one basis are generically named a "Clip". In other words, this means that a single stream file (*.m2ts) must correspond to a clip-info file (*.clpi).

A clip associated with the main data recorded in the disc is called a main clip. A clip associated with downloaded sub-data stored in a local storage is called a sub-clip.

The playlist directory (PLAYLIST) includes a plurality of playlist files (*.mpls). Each playlist file (*.mpls) includes one or more playitems (PlayItem) for designating a playing interval during which a specific clip is reproduced. The playitem (PlayItem) includes information associated with a specific clip to be reproduced, i.e., information associated with a reproduction start time (IN-Time) and other information associated with a reproduction termination time (OUT-Time) of a clip determined to be a clip name (Clip__Information_File_name) contained in the playitem (PlayItem).

The playlist file (*.mpls) is used as a basic reproduction management file unit contained in an overall file structure, such that it can reproduce a desired clip using a combination of one or more playitems. The playlist file (*.mpls) may include data different from reproduction data formed by the playitem. The playlist file (*.mpls) may include a sub-playitem (SubPlayItem) distinguished from the playitem (PlayItem).

FIG. 3A is a relationship among files used for reproducing a specific title according to the present invention. Upon receiving a title reproduction command from a user in association with a specific title applied to an index table of an index file, a method for reproducing the specific title will hereinafter be described.

The index file (index.bdmv) includes first playback information "First Playback" indicative of information associated with a first reproduction image when a corresponding disc is loaded, top menu information "Top_Menu" for providing a menu image, and at least one title information "Title #1 ~Title #n".

If the user selects a specific menu from among a plurality of menus of a specific title or a menu image, a specific playlist is reproduced according to a command contained in individual objects "mobj" contained in an object file (MovieObject.bdmv) of a file structure, and a specific clip (denoted by "ClipInfo + AV stream") is reproduced by a playitem contained in the playlist.

turbojet
26th September 2008, 15:00
The way I understand and recently experienced is AVCHD is a digital camera format which always uses the FAT32 structure changes you mention, it doesn't have the FAT32 4GB restriction. It can be burned onto a DVD or BD-R and work on most BD players. Also from what I gather AVCHD has different specs then BD, for example no DTS support and different max video bitrate.
BD is the format that has BDMV\CERTIFICATE in root, what tsmuxer outputs and what your program is intended for.
I know it's easy to get these 2 confused because some programs output BD and call it AVCHD, and some players display AVCHD, when its technically BD.

kaid
27th September 2008, 17:43
The way I understand and recently experienced is AVCHD is a digital camera format which always uses the FAT32 structure changes you mention, it doesn't have the FAT32 4GB restriction. It can be burned onto a DVD or BD-R and work on most BD players. Also from what I gather AVCHD has different specs then BD, for example no DTS support and different max video bitrate.

Yeah... Though PS3 still does DTS on AVCHD nonetheless! ;-)
From what i gathered AVCHD isn't always on FAT32, when it's a mini-DVD-R based camcorder it isn't, because then it uses UDF and the different naming scheme (atleast that's what the spec says!)..
But if the camcorder is using SD it HAS to use FAT32 (NTFS on Flashmedia isn't supported by anything!) and FAT32 *always* has the 4GB-limit!

BD is the format that has BDMV\CERTIFICATE in root, what tsmuxer outputs and what your program is intended for.

Well, the PS3 says it's an AVCHD, not a BD. As long as I don't go overboard with the bitrate (and don't use DTS) it's the same anyway, right? ;-)

I know it's easy to get these 2 confused because some programs output BD and call it AVCHD, and some players display AVCHD, when its technically BD.

Well, if the bitrate, use of DTS and the presence of a /CERTIFICATE folder are all that sets the two apart, I can understand the confusion! ;-)
Or is there some other differences I don't know about yet?

turbojet
28th September 2008, 07:19
The few avchd streams I've seen from avchd cameras have this (http://upload.wikimedia.org/wikipedia/en/f/fb/AVCHD_actual_file_structure.jpg) structure.

frank
3rd October 2008, 10:58
I want to clarify:

AVCHD headers
index.bdmv - INDX0100
MovieObject.bdmv - MOBJ0100
PLAYLIST *.mpls - MPLS0100
CLIPINF *.clpi - HDMV0100

So NERO does it right.

Blu-ray headers on BD-RE
index.bdmv - INDX0200
MovieObject.bdmv - MOBJ0200
PLAYLIST *.mpls - MPLS0200
CLIPINF *.clpi - HDMV0200

ATTENTION!
tsMuxeR mixed up headers, it sets mpls and clpi headers as Blu-ray = WRONG!
You have to patch *.mpls and *.clpi to AVCHD compliant headers.

smnckl
29th January 2009, 03:29
@kaid

I know its been a while since anyone has contributed to this thread... and I am just looking for info about the layout of the *.clpi file(s).

My issue is that when using tsMuxeR to create a BD structure, it has on many occasions messed up the AR info found in the .clpi file. I have muxed source files which had original ARs 2.35 or 2.4 and tsMuxeR has it set to 4:3 in the .clpi file. When these are burned to disc and played on a standalone, I get a "good" picture on the left half of my 16:9 HDTV and a ghost-like similar image on the right, as tho it was 2 tracks running side-by-side.

I have tried using BDedit but can not edit the .clpi file. I am hoping that you, in your research, have found where in the .clpi the AR info is stored and could help me with the off-set so I could try hexing that to see whether I can get the correct AR without having to re-encode the source x264 file with borders.

TIA, James

deank
29th January 2009, 08:37
Usually the AR value is at offset 0x0109 if you use tsMuxer's output.

smnckl
30th January 2009, 00:41
Usually the AR value is at offset 0x0109 if you use tsMuxer's output.

:thanks: for the info, deank.
I took a look at the .clpi from a movie-only rip, using HxD and it shows a value of 30h at offset 109.
MediaInfo tells me that the movie has original AR of 2.4:1.

Where can I find which hex value equals 16:9 and which equals 4:3?

Thanks in advance for any pointers.

James

deank
30th January 2009, 01:06
In AVCHD or BD you can have only 4:3 and 16:9 ARs.

Try changing $30 to $20.

foolproof
31st January 2009, 22:16
to deank

I have mailed you before time links to download samples of AVCHD structure made by your (great) program and made by Panasonic HDWriter 2.5e for the same stream.
I hope you can see differences and this will helpful in your researche.
Your structure is on PanasonicTV unfortunately functionless, from HDWriter is OK. Has Panasonic in his structure some one secret byte or what ?:mad:

deank
31st January 2009, 22:26
The difference is in main INDEX/MOVIEOBJ files. I'm still looking at the folders you've sent me. Another user also sent me some files so now I have like 5-6 different configurations that play on Panasonic TVs.

I think pannies use shortened format and no 'real' playlists. I will try to post a link with a sample for you to test these days (before end of next week).

Dean

edit: Is there a way I can get hands on this Panasonic HD writer program? (is it free?)

smnckl
1st February 2009, 06:33
In AVCHD or BD you can have only 4:3 and 16:9 ARs.

Try changing $30 to $20.

:thanks: deank. I will try as you suggested.
I understand the limitation on AR but it seems that after I have backed up the movie, the 16:9 info in the stream is missing and gets replaced by an AR calculated from the actual pixels. The non-AVCHD/BD AR appears after I have run the movie through tsMuxeR.

Thanks again. I will get back here and let you know whether the AR change is successful.

James

smnckl
1st February 2009, 21:38
In AVCHD or BD you can have only 4:3 and 16:9 ARs.

Try changing $30 to $20.

deank, your tip about the hex values worked. When value at offset 109h is $30, BDedit shows me that the DAR is 16:9 while $20 at offset 109h is shown as 4:3.

I am running a re-encode now to see whether x264 and/or tsMuxeR will accept the change and re-encode the stream as 16:9.

Thanks again.

James

mrr19121970
9th February 2009, 07:43
I wrote a little utility that can do this for you, incase you're not too comfortable with a HEX editor:

Editing CLIPINF (*.CLPI or *.CPI) (http://forum.slysoft.com/showthread.php?t=26930)

twazerty
6th May 2009, 11:10
Sorry for kicking this topic but I do have a question. tsMuxer can only handle 1 title. I used Nero Vision to create a multi title disc. Lets say I've put 8 items on a disc with a menu and intro. These files do have the right AVCHD headers etc. But I want to burn the files on a blu-ray disc. Are the only differences the headers? Or are there more things that are different?

So is changing the headers enough to make it fully 100% Blu-ray compatible?

*Also renaming those files but I already know that.

deank
6th May 2009, 11:57
1) You could have selected blu-ray output in Nero
2) No need to change anything in nero's avchd output (just filenames to long format)

What player will you be using with this bd disc?

twazerty
6th May 2009, 13:27
1) You could have selected blu-ray output in Nero
2) No need to change anything in nero's avchd output (just filenames to long format)

What player will you be using with this bd disc?

Don't have a blu-ray burner so no Blu-ray output. Only AVCHD output. I have the Samsung BD-P1500.

I have to tacke this possible problem for AVCHDCoder. If changing the Header to 0200 is enough it would be nice.

deank
23rd August 2009, 20:53
Just a reviver note, inspired by an email correspondence.

Although I posted about 20 or 30 times in different threads it seems that it is missed again and again.

0100 header value for clpi, mpls and bdmv files is a BLU-RAY VERSION ID. Changing it to 0200 does NOTHING (except in the cases you really know why you change/author it). 0100 is both for AVCHD and Blu-ray (profile 1.0, 1.1). If you by any chance self-author a BD-J BD or use advanced authoring (with an application you created) then may be 0200 may be a proper header. In all other cases you shouldn't play with these values (not that it will make a difference). This overrated discussion has to come to an end, although certain users decided to confuse people over and over again.

tsMuxeR (1.10.+) output for

BLU-RAY:
* is compliant blu-ray output

AVCHD:
* is BLU-RAY compliant output with AVCHD index.bdmv (so it is a mix)

Now, with AVCHD output from tsMuxeR (which was AVCHD-only since earlier versions) came a lot of questions, troubles, player compatibility reports, etc.

People started playing with deleting folders, changing headers, adding and changing bytes here and there.

So - play with them... And you/we will waste another 2 years finding out what really works.

And a final note: AVCHD is much more complicated format, compared to standard BDMV/BDAV and even BD-J. That is the reason a lot of manufacturers dropped AVCHD support in their players in recent firmware updates. In Blu-ray mode, firmware updates change/update the JAVA core errors and some advanced playlist handling, but the AVCHD part remains the same. It is just something set by sony and panasonic some time ago. You can't update millions of camcoders and TVs around the world with 'ethernet-firmware-update'.

Dean

xpro
1st June 2011, 20:56
Hello!

Anyone who has info about how the .M2TS files are structured? How to parse them to be able to demux the video/audio/PGS? Do you know any open source code for demuxing a M2TS / MTS file?

Regards, Tobias

kaefert
28th July 2018, 16:29
I have ton's of BMDV folders coming from a Panasonic video camera. I would like to parse the timestamp of each recorded clip from the clpi files. (I know the date of each recording, but not time.)

I'm trying to identify possible byte addresses of that possibly stored timestamp within the clpi files by comparing clpi files from the same day versus comparing clpi files from different days vs from different years. (I'm using "vbindiff" to make the differences visible)

Addresses that look more different the farther apart two clips are should be good candidates for timestamps.

I'm documenting my findings here, maybe they will be useful for others too:

0x17 & 0x1B: single bytes that always have different values, 0x1B seems to be equal to the value of 0x17 plus 4.
0x39 - 0x3B: 3 bytes always different, didn't find a pattern yet.
0xE2 - 0xEA: Most clips: ASCII "00001M2TS" where the number is counting (and is always the number in the filename+1), Those clips have a 0x01 at address 0xDD. Clips with a 0x00 at 0xDD don't have this ASCII string. Some of the clips without ASCII string have value 0x00F6 instead of 0x0106 @ 0xE & 0xF. Those files are missing 0x10 bytes between 0xC3 & 0xEF. So for those clips 0x10 must be subtracted for any address higher than 0xEF for the value to match the discription.
0x100 - 0x105: Clips on the same day mostly only vary on 0x101 & 0x103 - 0x105, clips from different days also vary in 0x100.
0x119 contains the 0x30 described by others above as coding for aspect ratio at address 0x109 (which is the address of this 0x30 for clpi files without the ASCII string mentioned 3 lines above.)
0x146 - 0x147, 0x151 - 0x153, 0x15B: more changes. I can't see any patterns here..

videofan3d
28th July 2018, 16:55
I have ton's of BMDV folders coming from a Panasonic video camera. I would like to parse the timestamp of each recorded clip from the clpi files. (I know the date of each recording, but not time.)



Maybe try AVCHDInfo (http://forum.doom9.org/showthread.php?t=174563) ...

videoh
28th July 2018, 18:41
There's no need to guess or reverse-engineer; the bluray specs are readily available online if you search for them (yes, free copies).

kaefert
28th July 2018, 22:04
oh thanks, I'll check it out.
update: hmm, this seems to be windows only.. doesnt work with mono..
And I did search for specs of those clpi files, this thread was the only thing I found.

UPDATE: yess! that worked! With wine AVCHDInfo.exe can extract Timestamps for each frame of an m2ts file, so it doesn't need the clpi files to get that info but only the video file itself :)
UPDATE2: ahh, and the MPLS2JSON.exe tool shows me that I've been looking for a non existent needle in the haystack, no timestamp infos in there ;)

neil wilkes
5th August 2018, 15:27
There's no need to guess or reverse-engineer; the bluray specs are readily available online if you search for them (yes, free copies).

Where, please?
According to the official specs site, I need to have these books and a license just to use the perishing logo (see Authoring House Logo Usage (http://www.blu-raydisc.info/need-license/logo-usage-ah.php)) section at the BDA Licensing office (http://www.blu-raydisc.info/index.php)

It doesn't stop there either, as I also need the BD-RE2 specs to cover any Java issues - most of the good information on BD-J is long since taken down (probably because Netflix & Amazon use a very similar system on their pay per view models) and even HD Cookbook site is history these days.

So if you could kindly point me in the right direction it would be greatly appreciated