Suggested XML key storage format

[noclip]

I'd like to propose that future versions of BackupHDDVD and BackupBluRay use a common XML format to store keys.

<?xml version="1.0" encoding="UTF-8"?>
<KeyIndex version="1.0">
<Format>HD DVD</Format>
<KeySet> <!-- Volume key example -->
<Description>Movie name</Description>
<Timestamp>YYYY-MM-DD</Timestamp>
<Origin>US</Origin>
<Key type="TKFMAC">00000000000000000000000000000000</Key>
<Key type="Volume">00000000000000000000000000000000</Key>
</KeySet>
<KeySet> <!-- Title key example -->
<Description>A different name</Description>
<Timestamp>YYYY-MM-DD</Timestamp>
<Origin>US</Origin>
<Key type="TKFMAC">00000000000000000000000000000000</Key>
<Key type="Title" title="1">00000000000000000000000000000000</Key>
<Key type="Title" title="2">00000000000000000000000000000000</Key>
</KeySet>
<KeySet> <!-- Barebones example -->
<Key type="TKFMAC">00000000000000000000000000000000</Key>
<Key type="Volume">00000000000000000000000000000000</Key>
</KeySet>
<KeySet>
...
</KeySet>
...
</KeyIndex>

The root element should always be KeyIndex. The format tag is mandatory and its value must be one of "HD DVD" or "Blu-ray". Each KeySet must contain at least a TKFMAC key and a decryption (title, volume) key. Date, region, and description are optional.

[honai]

My suggestion:


<?xml version="1.0" encoding="UTF-8">
<DiscIndex majorVersion="1" minorVersion="0">
<Disc index="[index]"> <!-- index is optional -->
<Title>[movie title]</Title>
<ScreenDate>[YYYY]</ScreenDate> <!-- optional -->
<Origin>[US|EU]</Origin>
<Release format="[HDDVD|BD]" discId="[discid]">
<Timestamp>[YYYY-MM-DD|YYYY]</Timestamp> <!-- optional -->
<Key type="[tkfmac|title|volume]">[key]</Key> <!-- optional -->
</Release>
<Release discId="[discid]">
...
</Release>
</Disc>
<Disc>
...
</Disc>
...
</DiscIndex>


Example:


<?xml version="1.0" encoding="UTF-8">
<DiscIndex majorVersion="1" minorVersion="0">
<Disc index="0">
<Title>Pitch Black</Title>
<ScreenDate>2000</ScreenDate>
<Origin>US</Origin>
<Release format="HDDVD" discId="FCEB4CA6C9614759A86A63E92BB87D0F">
<Timestamp>2006-08-01</Timestamp>
<Key type="title">9D82A55BF2DAC3995AD24B40B802D71F</Key>
</Release>
<Release format="HDDVD" discId="C4ECEA1CC4FB4CEF9E4D4D51231D84EE">
<Timestamp>2006-09-25</Timestamp>
<Key type="title">01BE7C9DE9C83DF7966A13BA621C1830</Key>
</Release>
</Disc>
<Disc index="1">
<Title>Corpse Bride</Title>
<ScreenDate>2005</ScreenDate>
<Origin>US</Origin>
<Release format="BD" discId="8F6B0E90A94011DAA7460800200C9A66">
<Timestamp>2006-10-12</Timestamp>
<Key type="volume">3E36749ADB6DBDE0A2238817FB3EBF14BA09F0AC</Key>
</Release>
</Disc>
</DiscIndex>

[tonyp12]

All the information to be inputed could be automated
exepect the region I think.

Can you read some info on the disc to get hold of that information?

How does a person in Germany that buys a disc and extract keys to
be inputed know that he acctually not have the same disc that is released in USA?

If that is the case, I say leave region out.

[He-Man]

I haven't looked in the Blu-Ray specs yet, but is there also a TKF MAC field there (maybe named something else) or do they use a different key file format?

[noclip]

From what Janvitos has said Blu-ray also uses a TKF MAC.



- Format should be the type of the KeySet

I considered this, but decided against it. It would be simpler to store each format's keys in its own file.



- move the movie title name into the KeySet as an attribute

I think we should stick to only putting mandatory values in attributes.

[He-Man]

<Date>0000-00-00</Date>
How are you supposed to fill out this filed?
What comes first dateor month?
There's so many different date formats used around the World.
Should it be YYYY-MM-DD or YYYY-DD-MM? I think you should change the zero with letters like this.

I also think you should change the filed "Date" to "Disc production date", or something like that so people don't think it's the key extraction date or something else.


I'm not a native English speaker, but isn't the term "Movie title" more common than "Movie name"? At least it is if you Google it.

[honai]

@He-Man

It's YYYY-MM-DD, that's the standard in all software applications.

[noclip]

I'm not a native English speaker, but isn't the term "Movie title" more common than "Movie name"? At least it is if you Google it.

I recommend sticking with "description" as it's much more flexible (What if it's a TV show and not a movie?).

[He-Man]

@He-Man

It's YYYY-MM-DD, that's the standard in all software applications.
Ok fine, that's also what I thought, I just wanted to make sure since there's so many different way of writing the same date in different countries.
If I look in Explorer it's DD-MM-YYYY, but I think that varies from language version to language version.

[He-Man]

I recommend sticking with "description" as it's much more flexible (What if it's a TV show and not a movie?).
Yes leave movie out of it.
But isn't it better to just call it "Title" then or maybe "Disc title"?
"Description" is too general IMO and can refer to many other things than the ttitle.

[noclip]

Yes leave movie out of it.
But isn't it better to just call it "Title" then or maybe "Disc title"?
"Description" is too general IMO and can refer to many other things than the ttitle.

Title may get it confused with title keys or titles of the disk.

[He-Man]

Title may get it confused with title keys or titles of the disk.
How about "Disc name" then?

[honai]

@noclip

Why don't we already write a common structure for both HD-DVD and Blu-ray? So just move the format as an attribute into KeySet. The other elements should be common for both formats.

[noclip]

How about "Disc name" then?

My only objection to this is that it could confuse users into thinking that they had to use the exact disk title.

@noclip

Why don't we already write a common structure for both HD-DVD and Blu-ray? So just move the format as an attribute into KeySet. The other elements should be common for both formats.

It feels cleaner to me to use two separate files considering they will be read by two separate programs located in different directories. The structure should, of course, be the same for both HD and Blu.

[He-Man]

My only objection to this is that it could confuse users into thinking that they had to use the exact disk title.
Yes isn't that what they are supposed to? What else should they write?

Anyway the title is inside one of the XML files at I posted in another topic, so it can get extrated automatically along with TKF MAC and possibly also the Timestamp.

[noclip]

Yes isn't that what they are supposed to? What else should they write?

Anyway the title is inside one of the XML files at I posted in another topic, so it can get extrated automatically along with TKF MAC and possibly also the Timestamp.

Problems could arise with some multiple-disk sets using the same name for each disk. TV show seasons on DVD are notorious for this problem. A name like "24 Season 4 Episodes 12-18" would be much more helpful than "24 Season 4" or even "24 Season 4 disk 3".

[honai]

It feels cleaner to me to use two separate files considering they will be read by two separate programs located in different directories.

Well, if it's the same structure then that'd be argument *for* using the same DTD/XSD, and writing a single XML document parser that handles both formats, thus minimizing the codebase. And it really doesn't make sense to keep two different DTDs updated if you're going to basically use the same format.

[noclip]

Well, if it's the same structure then that'd be argument *for* using the same DTD/XSD, and writing a single XML document parser that handles both formats, thus minimizing the codebase. And it really doesn't make sense to keep two different DTDs updated if you're going to basically use the same format.

I'm all for a unified XSD. I'm saying there should be two different files on end users' drives ("BluKeyIndex.xml" and "HDKeyIndex.xml").

[honai]

A name like "24 Season 4 Episodes 12-18" would be much more helpful than "24 Season 4" or even "24 Season 4 disk 3".

The "standard" for that is "[Series title] S[season]E[episode] Disc [disc]", e.g. "24 S04E12 Disc 1" or "24 S04E12-18 Disc 3". There is no series running over 99 seasons or 99 episodes per season.

[honai]

I'm all for a unified XSD. I'm saying there should be two different files on end users' drives

Ah ok, that clarifies it. But you should still move the <Format> element as an attribute into <KeyIndex>, makes it a bit easier for parser writers and cleans up the semantics of the XSD as well.

[He-Man]

From what Janvitos has said Blu-ray also uses a TKF MAC.
I looked in the AACS Blu-Ray speac now and Blu-Ray doesn't use a TKF MAC, first of all because there's no file named Title Key File.

There's two kind of CMAC used on Blu-Ray, but from what I read they are both optinal depending on different things.

The two CMAC's on Blu-Ray discs:
MAC of PMSN field
MAC of Device Binding ID


AACS Blu-ray Disc Pre-recorded Book v0.912 (http://www.aacsla.com/specifications/AACS_Spec_BD_Prerecorded_0.912.pdf)
MAC of PMSN field
MAC of PMSN field contains the 16-byte MAC of Pre-Recorded Media Serial Number by using CPS Unit Key
for each CPS Unit. The MAC of PMSN is generated as follows:
CMAC( Kcu, Pre-recorded Media Serial Number )
(Note) In case that the CPS Unit is not bound to the Pre-Recorded Media Serial Number, the MAC of PMSN
field shall be set to all-zero. In other words, this field on the BD-ROM disc is always set to all-zero. Practically,
this field is used only in the case that the Virtual File System is used for downloaded content in local storage.
For the Virtual File System, refer to Section 4.1.

3.6 Pre-Recorded Media Serial Number
For purpose of using On-line Connections, the Pre-Recorded Media Serial Number is defined and is used for
generating the MAC of PMSN. The Pre-Recorded Media Serial Number is optional for BD-ROM disc. For
BD-ROM, the Pre-Recorded Media Serial Number shall be stored in the BCA record of the BD-ROM disc.
Player shall use 128bits value in a Data Unit as a Pre-recorded Media Serial Number, where the first 8 bits of
the value is set to 00000100.


MAC of Device Binding ID
MAC of Device Binding ID field contains the 16-byte MAC of Device Binding ID by using CPS Unit Key for
each CPS Unit. The MAC of Device Binding ID is generated as follows:
CMAC( Kcu, Device Binding ID )
(Note) In case that the CPS Unit is not bound to the Device, the MAC of Device Binding ID field shall be set to
all-zero. In other words, this field on the BD-ROM disc is always set to all-zero. Practically, this field is used
only in the case that the Virtual File System is used for downloaded content in local storage. For the Virtual
File System, refer to Section 4.1.
Device Binding ID is used only when the Binding_Type defined in Section 3.9.4.4 is set to Device/Content
Binding or Device/Media Binding. The value of Device Binding ID is implementation dependent and is not
specified in this specification.

Based on this can a CMAC value always be used for Blu-Ray? It's not all clear to me.

[He-Man]

It feels cleaner to me to use two separate files considering they will be read by two separate programs located in different directories.
What makes you think it will be two different programs?
I'm convinced BackupHDDVD and BackupBluRay will eventually be integrted into on aplication capable of decrypting both HD DVD and Blu-Ray and extracting keys, tiltes, date and CMAC from the disc.

[noclip]

As far as I know all BD disks so far contain a PMSN-MAC, so we should be able to rely on this.

What makes you think it will be two different programs?
I'm convinced BackupHDDVD and BackupBluRay will eventually be integrted into on aplication capable of decrypting both HD DVD and Blu-Ray and extracting keys, tiltes, date and CMAC from the disc.

I'm not so sure of that. Why have one big, unwieldy codebase for the benefit of the vast minority of users? Changes in Blu-ray AACS and HD DVD AACS are likely to happen at different times, so keeping them two separate programs seems smarter to me.

[He-Man]

As far as I know all BD disks so far contain a PMSN-MAC, so we should be able to rely on this.
In my quotes above you can read that: "The Pre-Recorded Media Serial Number is optional for BD-ROM disc."
If no PMSN is used, the PMSN-MAC field will be all zeros as it also syas in the quote above.

So the quation is if PMSN (Pre-Recorded Media Serial Number) is always used since it's optional? If PMSN is not always used then we can't use PMSN-MAC since it will be all zeros on discs without PMSN.

[He-Man]

I'm not so sure of that. Why have one big, unwieldy codebase for the benefit of the vast minority of users? Changes in Blu-ray AACS and HD DVD AACS are likely to happen at different times, so keeping them two separate programs seems smarter to me.
Well you also use the same player for audio, DVD, HD DVD, Blu-Ray etc. even though different formats and codecs are updated at differnt times.
I'm quite convinced they will eventually be integrated inot one to supoort both Blu-Ray and HD DVD, at least the same GUI, maybe with different DLL's.
But make sure the XML format will support both stand alone Blu-Ray and HD DVD backup applications and integrated applications. Then it can always be decided later wehter the tools should be integrated into one, that's not something we should decide now.

[noclip]

I made a schema, ideally programmers should make sure their applications' generated files validate against this.

http://backuphddvd.sourceforge.net/schema/XKI-1.00.xsd

[Zero1]

This is probably a flawed idea (it just popped into my head); but how about making the software able to upload keys to a webserver so a database can be built? I'm thinking along the lines of a CDDB for HD-DVD keys. The only problem with this though, is I suppose some company would get the server/ISP shut down?

It would be similar to storing the keys in that first forum post on here, but automated. Perhaps an end user could sync their keyfile or something.

[honai]

Great! A couple of notes:


<xs:element name="Origin" type="xs:string"/>


Please use an enumeration for possible values ("US", "EU", etc).


<xs:element name="Key">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:pattern value="[A-Z0-9]{32}"/>
</xs:restriction>
</xs:simpleType>
</xs:element>


Please use <xsd:hexBinary> instead.

And use <xsd:token> instead of <xsd:string>.

[noclip]

<xs:element name="Origin" type="xs:string"/>


Please use an enumeration for possible values ("US", "EU", etc).

I didn't want to limit it this way because we don't know what the actual regions are.

As for the keys, I'm a little confused. If I apply a length restriction, should it be 32 or 128?

[honai]

The length delimiter is lexical, i.e. 32 (1 character for each octet).

Region codes for Blu-ray:

http://www.emedialive.com/articles/readarticle.aspx?articleid=11760#ixa
http://en.wikipedia.org/wiki/Blu-ray_Disc#Region_codes

[melakai]

@noclip

Why don't we already write a common structure for both HD-DVD and Blu-ray? So just move the format as an attribute into KeySet. The other elements should be common for both formats.

This was my first inclination also, as I dislike the idea of multiple files for describing the same/similar things. If this were a relational database, one would probably use a flag for type instead of multiple tables.

Furthermore, I suspect a single backup binary will eventually be developed.

[melakai]

This is probably a flawed idea (it just popped into my head); but how about making the software able to upload keys to a webserver so a database can be built? I'm thinking along the lines of a CDDB for HD-DVD keys. The only problem with this though, is I suppose some company would get the server/ISP shut down?

It would be similar to storing the keys in that first forum post on here, but automated. Perhaps an end user could sync their keyfile or something.

There is another thread discussing a website for keys: http://forum.doom9.org/showthread.php?t=120664

The site is located at http://hdkeys.com/ and is already set up for automated retrieval. It would be trivial to implement key submission.

[HyperHacker]

I would imagine the two being combined into one program; clean, well-written code should avoid the issue of changes to one's protection scheme being a problem. That said, I support the idea of having one file for both types. I'd also move some things to attributes of the KeySet node, and shrink a few names a bit, to save space, EG:
<KeySet fmt="BD" desc="Cool Blu-ray Movie" date="2007-01-21" origin="US">
This is a much more efficient format; once you start adding a lot of movies to the file, it may turn out to be worthwhile.

As I mentioned elsewhere I can post an XML parser/writer class for C++ if anyone wants it.

[Zotty]

So where do we stand on this subject? Until now I've been using the format found in the initial post of this thread. But seeing that CMAC values are nowhere near as widespread as the SHA-1 hashes of titlekey files, I've been abusing the TKFMAC field for the SHA hashes. This can cause confusion and that's not something you'd want. I'd love to change this so the hash value is no longer in the MAC field. Just an extra key type to store it in would be enough.

Combined with what I've read in posts above, it more or less boils to the following; BR has no titlekey file, so naming it KTFMAC would be a wrong name for the CMAC field.
Also there's no field to store the hash value. Given the current situation I do think it would be nice to have it one way or the other, even if it's optional. Given it's a filehash, it doesn't matter if the file comes from a HD DVD titlekey file or some BR file.

Boils down to this:

....
<KeySet>
<Description>Movie name</Description>
<Timestamp>YYYY-MM-DD</Timestamp>
<Origin>US</Origin>
<Key type="CMAC">00000000000000000000000000000000</Key>
<Key type="Hash">00000000000000000000000000000000</Key>
<Key type="Volume">00000000000000000000000000000000</Key>
</KeySet>
....


If I missed anything say so, but until now there isn't a final format and time is kinda running out. There are more and more keys coming, so instead of leaving this issue I think it's a good idea to come to at least some sort of agreement before custom formats start popping up.

[honai]

I don't see "custom formats" as a problem as long as specific implementations come with a parser.

That said, if you want to settle this just write your own version and ship it with a working and pluggable parser.

[tonyp12]

I think 2-4 people should decide the final format.
The same people that are coding the new versions of BackupHDDVD.

I think people will still post VUK in a single line posts
So the Parser should look at the lenght (or padding) of the first field
and decide if it's a cmac or sha1

30F8DC87B137A1607C7F2A731FF7B6BC c.m.a.c=King Kong |V|09/18/06| 802F78B1B20D1183638D84E1A96D6EDD

I will implement that in my next update of TitleSorter (http://forum.doom9.org/showthread.php?p=943657#post943657), that as of now also outputs to XML.

[The_ByteMaster]

I think 2-4 people should decide the final format.
The same people that are coding the new versions of BackupHDDVD.

I think people will still post VUK in a single line posts
So the Parser should look at the lenght (or padding) of the first field
and decide if it's a cmac or sha1

30F8DC87B137A1607C7F2A731FF7B6BC c.m.a.c=King Kong |V|09/18/06| 802F78B1B20D1183638D84E1A96D6EDD

I will implement that in my next update of , that as of now also outputs to XML.

The reason CMAC / TKF-MAC was brought up originally was just an easy way to verify the validity of the Volume Unique Key, see
here (http://forum.doom9.org/showthread.php?p=938202#post938202).

This is of course still valid. Having the "supposed' TKF-MAC available in the xml file is a plus, as should decryption with the supplied key for some reason fail, the program can then compare the TKF-MAC in the xml file with the one it found on the disc.

Assuming the user can either download, or have online access to, a copy of the xml file with all known keys to date, then basically, IMHO, what is needed in the xml file boils down to:

1. a machine readable unique Disc ID
2. key necessary to decrypt
3. a human readable, perhaps non-unique Disc ID

The machine readable unique Disc ID might perhaps be a combination of a few things, like the SHA1, the TKF-MAC, the file creation date and md5sum of the VTKF.AACS / VTKF%%%.AACS file, etc.

[tonyp12]

Sha1 or cmac does not really matter when it comes identify the disc.

But why not go with later as it also can be used to verify that the key is good.

Muslix64 probably did not know about cmac or that this value is unique to every released title.

Now that we have 100 titles with hash, it's getting harder to change over.

[honai]

I don't have the specs, but the unique Disc ID is defined in the DISCID.DAT file, offset 44, 16 bytes long.

I already wrote a library to extract that record (along with an XPL parser), it will be released this week, hopefully.

[Emon]

I propose the following format for the KeyDB


<?xml version="1.0" encoding="UTF-8"?>
<TitleIndex Version="1.0" Format="HDDVD">
<Titles>
<Title Description="12 Monkeys"
Origin="US"
ReleaseDate="YYYY-MM-DD"
Hash="1828B68D292D2EA1E9EEA1C7044DC864FDBC3EB6"
VolumeKey="2662C05B5238B0C50BD1BDF693223712" />

<Title Description="16 Blocks"
Origin="US"
ReleaseDate="YYYY-MM-DD"
Hash="7E0FE374FA11E4007228606E95E1A9B374D8EA69"
VolumeKey="DB8702D6FC60E08F27633EC2399371F5" />
</Titles>
</TitleIndex>

[hajj_3]

that looks good, is the region info located on the disc itself tho? like US or EUROPE?? if not you could have a drop down box with the 3 or so regions for the user to manually select.

[Zotty]

While we're at it suggesting ;)

All the fuzz is about storing the volume keys in a way they can be linked to the correct disc, nothing more, nothing less. So two values we're dealing with; a disc identifier and a VolumeKey, on a per disc basis;


<?xml version="1.0" encoding="UTF-8"?>
<KeyIndex version="1.0">
<Disc>
<Description>Movie name</Description>
<Timestamp>YYYY-MM-DD</Timestamp>
<Origin>EU</Origin>
<Id type="xxx">00000000000000000000000000000000</Id>
<Key>00000000000000000000000000000000</Key>
</Disc>

<Disc>
<Description>Movie name</Description>
<Timestamp>YYYY-MM-DD</Timestamp>
<Origin>EU</Origin>
<Id type="xxx">00000000000000000000000000000000</Id>
<Key>00000000000000000000000000000000</Key>
</Disc>

...
</KeyIndex>


Where type can be TKFMAC, SHA-1 hash, a general CMAC, some hash, a DISCID.DAT value, whatever.

[honai]

I've updated my suggestion:

http://forum.doom9.org/showthread.php?p=941290#post941290

This is likely the format I'll be using in my upcoming toolkit.

[melakai]

I've updated my suggestion:

http://forum.doom9.org/showthread.php?p=941290#post941290

This is likely the format I'll be using in my upcoming toolkit.

Looks like a good start, but I think it needs some fixing. here is my suggestion with reasons in comments:


<?xml version="1.0" encoding="UTF-8">
<ReleaseIndex majorVersion="1" minorVersion="0"> <!-- DiscIndex is not semantically correct; we can have multiple discs per release. fixed below -->
<Release format="[HDDVD|BD]" timestamp="[YYYY-MM-DD|YYYY]" Orgin="[US|EU]"> <!-- index removed, unneeded -->
<Title>[movie title]</Title>
<ScreenDate>[YYYY]</ScreenDate> <!-- optional; more movie details like imdb link can be added, all optional -->
<Disc discId="[discid]" DiscTitle="[Disc Title]"> <!-- allows multiple discs per release (i.e. TV series). DiscTitle optional, allows for "Episode 5-8" -->
<Key type="[tkfmac|title|volume]">[key]</Key> <!-- allows multiple keys per disc -->
</Disc>
</Release>
</DiscIndex>

This mostly fixes some issues arising from multiple-disc releases. It would be possible in the other format, but I think this would reduce duplicated information in those cases.

[Emon]

I think we won't ever come to an agreement on this. The best route would be to come up with a frontend for this DB on every different type of tools that we are coming up with.

[honai]

@melakai

Excellent work! I guess that fixes most of the issues of my suggestion.

[blutach]

I think we won't ever come to an agreement on this. The best route would be to come up with a frontend for this DB on every different type of tools that we are coming up with.Good idea. Eventually, I hope all the fine work can converge or else we are going to have more solutions than they have formats :)

Regards

[noclip]

There is absolutely no reason for any of the wackiness proposed by some of the posts here. The only two pieces of information needed to do 99% of what needs to be done are the TKF MAC and the volume key. Grouping multiple disk sets or creating separate "release" elements is just silly. The format in my original post and the main schema has a 1:1 disk to KeySet ratio, which is simple and functional.

[noclip]

I've updated my suggestion:

http://forum.doom9.org/showthread.php?p=941290#post941290

This is likely the format I'll be using in my upcoming toolkit.

Ugh. Why needlessly complicate things?

All that's really needed is a disk identifier/volume key pair, everything else is just for show and should therefore be optional. Grouping by release and mixing attributes and child elements is complicating things more than they need to be. If a disk gets another release, is from another region, or is part of a box set it will have a different TKF MAC. That's all the information needed to distinguish between disks in need of different volume keys.

[honai]

There is absolutely no reason for any of the wackiness proposed by some of the posts here. The only two pieces of information needed to do 99% of what needs to be done

"No one will ever need more than 640K RAM."

But seriously, there are many reasons to make the format as liberal (read: generic and adaptable) as possible and as strict as necessary. If you think you'll only need a few of the proposed elements then just write your parser that way and ignore the rest. Having more optional elements doesn't inflict any complexity on your own parser unless you decide to implement them.