Suggested XML key storage format

[noclip]

I'd like to propose that future versions of BackupHDDVD and BackupBluRay use a common XML format to store keys.

<?xml version="1.0" encoding="UTF-8"?>
<KeyIndex version="1.0">
<Format>HD DVD</Format>
<KeySet> <!-- Volume key example -->
<Description>Movie name</Description>
<Timestamp>YYYY-MM-DD</Timestamp>
<Origin>US</Origin>
<Key type="TKFMAC">00000000000000000000000000000000</Key>
<Key type="Volume">00000000000000000000000000000000</Key>
</KeySet>
<KeySet> <!-- Title key example -->
<Description>A different name</Description>
<Timestamp>YYYY-MM-DD</Timestamp>
<Origin>US</Origin>
<Key type="TKFMAC">00000000000000000000000000000000</Key>
<Key type="Title" title="1">00000000000000000000000000000000</Key>
<Key type="Title" title="2">00000000000000000000000000000000</Key>
</KeySet>
<KeySet> <!-- Barebones example -->
<Key type="TKFMAC">00000000000000000000000000000000</Key>
<Key type="Volume">00000000000000000000000000000000</Key>
</KeySet>
<KeySet>
...
</KeySet>
...
</KeyIndex>

The root element should always be KeyIndex. The format tag is mandatory and its value must be one of "HD DVD" or "Blu-ray". Each KeySet must contain at least a TKFMAC key and a decryption (title, volume) key. Date, region, and description are optional.

[honai]

My suggestion:


<?xml version="1.0" encoding="UTF-8">
<DiscIndex majorVersion="1" minorVersion="0">
<Disc index="[index]"> <!-- index is optional -->
<Title>[movie title]</Title>
<ScreenDate>[YYYY]</ScreenDate> <!-- optional -->
<Origin>[US|EU]</Origin>
<Release format="[HDDVD|BD]" discId="[discid]">
<Timestamp>[YYYY-MM-DD|YYYY]</Timestamp> <!-- optional -->
<Key type="[tkfmac|title|volume]">[key]</Key> <!-- optional -->
</Release>
<Release discId="[discid]">
...
</Release>
</Disc>
<Disc>
...
</Disc>
...
</DiscIndex>


Example:


<?xml version="1.0" encoding="UTF-8">
<DiscIndex majorVersion="1" minorVersion="0">
<Disc index="0">
<Title>Pitch Black</Title>
<ScreenDate>2000</ScreenDate>
<Origin>US</Origin>
<Release format="HDDVD" discId="FCEB4CA6C9614759A86A63E92BB87D0F">
<Timestamp>2006-08-01</Timestamp>
<Key type="title">9D82A55BF2DAC3995AD24B40B802D71F</Key>
</Release>
<Release format="HDDVD" discId="C4ECEA1CC4FB4CEF9E4D4D51231D84EE">
<Timestamp>2006-09-25</Timestamp>
<Key type="title">01BE7C9DE9C83DF7966A13BA621C1830</Key>
</Release>
</Disc>
<Disc index="1">
<Title>Corpse Bride</Title>
<ScreenDate>2005</ScreenDate>
<Origin>US</Origin>
<Release format="BD" discId="8F6B0E90A94011DAA7460800200C9A66">
<Timestamp>2006-10-12</Timestamp>
<Key type="volume">3E36749ADB6DBDE0A2238817FB3EBF14BA09F0AC</Key>
</Release>
</Disc>
</DiscIndex>

[tonyp12]

All the information to be inputed could be automated
exepect the region I think.

Can you read some info on the disc to get hold of that information?

How does a person in Germany that buys a disc and extract keys to
be inputed know that he acctually not have the same disc that is released in USA?

If that is the case, I say leave region out.

[He-Man]

I haven't looked in the Blu-Ray specs yet, but is there also a TKF MAC field there (maybe named something else) or do they use a different key file format?

[noclip]

From what Janvitos has said Blu-ray also uses a TKF MAC.



- Format should be the type of the KeySet

I considered this, but decided against it. It would be simpler to store each format's keys in its own file.



- move the movie title name into the KeySet as an attribute

I think we should stick to only putting mandatory values in attributes.

[He-Man]

<Date>0000-00-00</Date>
How are you supposed to fill out this filed?
What comes first dateor month?
There's so many different date formats used around the World.
Should it be YYYY-MM-DD or YYYY-DD-MM? I think you should change the zero with letters like this.

I also think you should change the filed "Date" to "Disc production date", or something like that so people don't think it's the key extraction date or something else.


I'm not a native English speaker, but isn't the term "Movie title" more common than "Movie name"? At least it is if you Google it.

[honai]

@He-Man

It's YYYY-MM-DD, that's the standard in all software applications.

[noclip]

I'm not a native English speaker, but isn't the term "Movie title" more common than "Movie name"? At least it is if you Google it.

I recommend sticking with "description" as it's much more flexible (What if it's a TV show and not a movie?).

[He-Man]

@He-Man

It's YYYY-MM-DD, that's the standard in all software applications.
Ok fine, that's also what I thought, I just wanted to make sure since there's so many different way of writing the same date in different countries.
If I look in Explorer it's DD-MM-YYYY, but I think that varies from language version to language version.

[He-Man]

I recommend sticking with "description" as it's much more flexible (What if it's a TV show and not a movie?).
Yes leave movie out of it.
But isn't it better to just call it "Title" then or maybe "Disc title"?
"Description" is too general IMO and can refer to many other things than the ttitle.

[noclip]

Yes leave movie out of it.
But isn't it better to just call it "Title" then or maybe "Disc title"?
"Description" is too general IMO and can refer to many other things than the ttitle.

Title may get it confused with title keys or titles of the disk.

[He-Man]

Title may get it confused with title keys or titles of the disk.
How about "Disc name" then?

[honai]

@noclip

Why don't we already write a common structure for both HD-DVD and Blu-ray? So just move the format as an attribute into KeySet. The other elements should be common for both formats.

[noclip]

How about "Disc name" then?

My only objection to this is that it could confuse users into thinking that they had to use the exact disk title.

@noclip

Why don't we already write a common structure for both HD-DVD and Blu-ray? So just move the format as an attribute into KeySet. The other elements should be common for both formats.

It feels cleaner to me to use two separate files considering they will be read by two separate programs located in different directories. The structure should, of course, be the same for both HD and Blu.

[He-Man]

My only objection to this is that it could confuse users into thinking that they had to use the exact disk title.
Yes isn't that what they are supposed to? What else should they write?

Anyway the title is inside one of the XML files at I posted in another topic, so it can get extrated automatically along with TKF MAC and possibly also the Timestamp.

[noclip]

Yes isn't that what they are supposed to? What else should they write?

Anyway the title is inside one of the XML files at I posted in another topic, so it can get extrated automatically along with TKF MAC and possibly also the Timestamp.

Problems could arise with some multiple-disk sets using the same name for each disk. TV show seasons on DVD are notorious for this problem. A name like "24 Season 4 Episodes 12-18" would be much more helpful than "24 Season 4" or even "24 Season 4 disk 3".

[honai]

It feels cleaner to me to use two separate files considering they will be read by two separate programs located in different directories.

Well, if it's the same structure then that'd be argument *for* using the same DTD/XSD, and writing a single XML document parser that handles both formats, thus minimizing the codebase. And it really doesn't make sense to keep two different DTDs updated if you're going to basically use the same format.

[noclip]

Well, if it's the same structure then that'd be argument *for* using the same DTD/XSD, and writing a single XML document parser that handles both formats, thus minimizing the codebase. And it really doesn't make sense to keep two different DTDs updated if you're going to basically use the same format.

I'm all for a unified XSD. I'm saying there should be two different files on end users' drives ("BluKeyIndex.xml" and "HDKeyIndex.xml").

[honai]

A name like "24 Season 4 Episodes 12-18" would be much more helpful than "24 Season 4" or even "24 Season 4 disk 3".

The "standard" for that is "[Series title] S[season]E[episode] Disc [disc]", e.g. "24 S04E12 Disc 1" or "24 S04E12-18 Disc 3". There is no series running over 99 seasons or 99 episodes per season.

[honai]

I'm all for a unified XSD. I'm saying there should be two different files on end users' drives

Ah ok, that clarifies it. But you should still move the <Format> element as an attribute into <KeyIndex>, makes it a bit easier for parser writers and cleans up the semantics of the XSD as well.

[He-Man]

From what Janvitos has said Blu-ray also uses a TKF MAC.
I looked in the AACS Blu-Ray speac now and Blu-Ray doesn't use a TKF MAC, first of all because there's no file named Title Key File.

There's two kind of CMAC used on Blu-Ray, but from what I read they are both optinal depending on different things.

The two CMAC's on Blu-Ray discs:
MAC of PMSN field
MAC of Device Binding ID


AACS Blu-ray Disc Pre-recorded Book v0.912 (http://www.aacsla.com/specifications/AACS_Spec_BD_Prerecorded_0.912.pdf)
MAC of PMSN field
MAC of PMSN field contains the 16-byte MAC of Pre-Recorded Media Serial Number by using CPS Unit Key
for each CPS Unit. The MAC of PMSN is generated as follows:
CMAC( Kcu, Pre-recorded Media Serial Number )
(Note) In case that the CPS Unit is not bound to the Pre-Recorded Media Serial Number, the MAC of PMSN
field shall be set to all-zero. In other words, this field on the BD-ROM disc is always set to all-zero. Practically,
this field is used only in the case that the Virtual File System is used for downloaded content in local storage.
For the Virtual File System, refer to Section 4.1.

3.6 Pre-Recorded Media Serial Number
For purpose of using On-line Connections, the Pre-Recorded Media Serial Number is defined and is used for
generating the MAC of PMSN. The Pre-Recorded Media Serial Number is optional for BD-ROM disc. For
BD-ROM, the Pre-Recorded Media Serial Number shall be stored in the BCA record of the BD-ROM disc.
Player shall use 128bits value in a Data Unit as a Pre-recorded Media Serial Number, where the first 8 bits of
the value is set to 00000100.


MAC of Device Binding ID
MAC of Device Binding ID field contains the 16-byte MAC of Device Binding ID by using CPS Unit Key for
each CPS Unit. The MAC of Device Binding ID is generated as follows:
CMAC( Kcu, Device Binding ID )
(Note) In case that the CPS Unit is not bound to the Device, the MAC of Device Binding ID field shall be set to
all-zero. In other words, this field on the BD-ROM disc is always set to all-zero. Practically, this field is used
only in the case that the Virtual File System is used for downloaded content in local storage. For the Virtual
File System, refer to Section 4.1.
Device Binding ID is used only when the Binding_Type defined in Section 3.9.4.4 is set to Device/Content
Binding or Device/Media Binding. The value of Device Binding ID is implementation dependent and is not
specified in this specification.

Based on this can a CMAC value always be used for Blu-Ray? It's not all clear to me.

[He-Man]

It feels cleaner to me to use two separate files considering they will be read by two separate programs located in different directories.
What makes you think it will be two different programs?
I'm convinced BackupHDDVD and BackupBluRay will eventually be integrted into on aplication capable of decrypting both HD DVD and Blu-Ray and extracting keys, tiltes, date and CMAC from the disc.

[noclip]

As far as I know all BD disks so far contain a PMSN-MAC, so we should be able to rely on this.

What makes you think it will be two different programs?
I'm convinced BackupHDDVD and BackupBluRay will eventually be integrted into on aplication capable of decrypting both HD DVD and Blu-Ray and extracting keys, tiltes, date and CMAC from the disc.

I'm not so sure of that. Why have one big, unwieldy codebase for the benefit of the vast minority of users? Changes in Blu-ray AACS and HD DVD AACS are likely to happen at different times, so keeping them two separate programs seems smarter to me.

[He-Man]

As far as I know all BD disks so far contain a PMSN-MAC, so we should be able to rely on this.
In my quotes above you can read that: "The Pre-Recorded Media Serial Number is optional for BD-ROM disc."
If no PMSN is used, the PMSN-MAC field will be all zeros as it also syas in the quote above.

So the quation is if PMSN (Pre-Recorded Media Serial Number) is always used since it's optional? If PMSN is not always used then we can't use PMSN-MAC since it will be all zeros on discs without PMSN.

[He-Man]

I'm not so sure of that. Why have one big, unwieldy codebase for the benefit of the vast minority of users? Changes in Blu-ray AACS and HD DVD AACS are likely to happen at different times, so keeping them two separate programs seems smarter to me.
Well you also use the same player for audio, DVD, HD DVD, Blu-Ray etc. even though different formats and codecs are updated at differnt times.
I'm quite convinced they will eventually be integrated inot one to supoort both Blu-Ray and HD DVD, at least the same GUI, maybe with different DLL's.
But make sure the XML format will support both stand alone Blu-Ray and HD DVD backup applications and integrated applications. Then it can always be decided later wehter the tools should be integrated into one, that's not something we should decide now.

[noclip]

I made a schema, ideally programmers should make sure their applications' generated files validate against this.

http://backuphddvd.sourceforge.net/schema/XKI-1.00.xsd

[Zero1]

This is probably a flawed idea (it just popped into my head); but how about making the software able to upload keys to a webserver so a database can be built? I'm thinking along the lines of a CDDB for HD-DVD keys. The only problem with this though, is I suppose some company would get the server/ISP shut down?

It would be similar to storing the keys in that first forum post on here, but automated. Perhaps an end user could sync their keyfile or something.

[honai]

Great! A couple of notes:


<xs:element name="Origin" type="xs:string"/>


Please use an enumeration for possible values ("US", "EU", etc).


<xs:element name="Key">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:pattern value="[A-Z0-9]{32}"/>
</xs:restriction>
</xs:simpleType>
</xs:element>


Please use <xsd:hexBinary> instead.

And use <xsd:token> instead of <xsd:string>.

[noclip]

<xs:element name="Origin" type="xs:string"/>


Please use an enumeration for possible values ("US", "EU", etc).

I didn't want to limit it this way because we don't know what the actual regions are.

As for the keys, I'm a little confused. If I apply a length restriction, should it be 32 or 128?

[honai]

The length delimiter is lexical, i.e. 32 (1 character for each octet).

Region codes for Blu-ray:

http://www.emedialive.com/articles/readarticle.aspx?articleid=11760#ixa
http://en.wikipedia.org/wiki/Blu-ray_Disc#Region_codes

[melakai]

@noclip

Why don't we already write a common structure for both HD-DVD and Blu-ray? So just move the format as an attribute into KeySet. The other elements should be common for both formats.

This was my first inclination also, as I dislike the idea of multiple files for describing the same/similar things. If this were a relational database, one would probably use a flag for type instead of multiple tables.

Furthermore, I suspect a single backup binary will eventually be developed.

[melakai]

This is probably a flawed idea (it just popped into my head); but how about making the software able to upload keys to a webserver so a database can be built? I'm thinking along the lines of a CDDB for HD-DVD keys. The only problem with this though, is I suppose some company would get the server/ISP shut down?

It would be similar to storing the keys in that first forum post on here, but automated. Perhaps an end user could sync their keyfile or something.

There is another thread discussing a website for keys: http://forum.doom9.org/showthread.php?t=120664

The site is located at http://hdkeys.com/ and is already set up for automated retrieval. It would be trivial to implement key submission.

[HyperHacker]

I would imagine the two being combined into one program; clean, well-written code should avoid the issue of changes to one's protection scheme being a problem. That said, I support the idea of having one file for both types. I'd also move some things to attributes of the KeySet node, and shrink a few names a bit, to save space, EG:
<KeySet fmt="BD" desc="Cool Blu-ray Movie" date="2007-01-21" origin="US">
This is a much more efficient format; once you start adding a lot of movies to the file, it may turn out to be worthwhile.

As I mentioned elsewhere I can post an XML parser/writer class for C++ if anyone wants it.

[Zotty]

So where do we stand on this subject? Until now I've been using the format found in the initial post of this thread. But seeing that CMAC values are nowhere near as widespread as the SHA-1 hashes of titlekey files, I've been abusing the TKFMAC field for the SHA hashes. This can cause confusion and that's not something you'd want. I'd love to change this so the hash value is no longer in the MAC field. Just an extra key type to store it in would be enough.

Combined with what I've read in posts above, it more or less boils to the following; BR has no titlekey file, so naming it KTFMAC would be a wrong name for the CMAC field.
Also there's no field to store the hash value. Given the current situation I do think it would be nice to have it one way or the other, even if it's optional. Given it's a filehash, it doesn't matter if the file comes from a HD DVD titlekey file or some BR file.

Boils down to this:

....
<KeySet>
<Description>Movie name</Description>
<Timestamp>YYYY-MM-DD</Timestamp>
<Origin>US</Origin>
<Key type="CMAC">00000000000000000000000000000000</Key>
<Key type="Hash">00000000000000000000000000000000</Key>
<Key type="Volume">00000000000000000000000000000000</Key>
</KeySet>
....


If I missed anything say so, but until now there isn't a final format and time is kinda running out. There are more and more keys coming, so instead of leaving this issue I think it's a good idea to come to at least some sort of agreement before custom formats start popping up.

[honai]

I don't see "custom formats" as a problem as long as specific implementations come with a parser.

That said, if you want to settle this just write your own version and ship it with a working and pluggable parser.

[tonyp12]

I think 2-4 people should decide the final format.
The same people that are coding the new versions of BackupHDDVD.

I think people will still post VUK in a single line posts
So the Parser should look at the lenght (or padding) of the first field
and decide if it's a cmac or sha1

30F8DC87B137A1607C7F2A731FF7B6BC c.m.a.c=King Kong |V|09/18/06| 802F78B1B20D1183638D84E1A96D6EDD

I will implement that in my next update of TitleSorter (http://forum.doom9.org/showthread.php?p=943657#post943657), that as of now also outputs to XML.

[The_ByteMaster]

I think 2-4 people should decide the final format.
The same people that are coding the new versions of BackupHDDVD.

I think people will still post VUK in a single line posts
So the Parser should look at the lenght (or padding) of the first field
and decide if it's a cmac or sha1

30F8DC87B137A1607C7F2A731FF7B6BC c.m.a.c=King Kong |V|09/18/06| 802F78B1B20D1183638D84E1A96D6EDD

I will implement that in my next update of , that as of now also outputs to XML.

The reason CMAC / TKF-MAC was brought up originally was just an easy way to verify the validity of the Volume Unique Key, see
here (http://forum.doom9.org/showthread.php?p=938202#post938202).

This is of course still valid. Having the "supposed' TKF-MAC available in the xml file is a plus, as should decryption with the supplied key for some reason fail, the program can then compare the TKF-MAC in the xml file with the one it found on the disc.

Assuming the user can either download, or have online access to, a copy of the xml file with all known keys to date, then basically, IMHO, what is needed in the xml file boils down to:

1. a machine readable unique Disc ID
2. key necessary to decrypt
3. a human readable, perhaps non-unique Disc ID

The machine readable unique Disc ID might perhaps be a combination of a few things, like the SHA1, the TKF-MAC, the file creation date and md5sum of the VTKF.AACS / VTKF%%%.AACS file, etc.

[tonyp12]

Sha1 or cmac does not really matter when it comes identify the disc.

But why not go with later as it also can be used to verify that the key is good.

Muslix64 probably did not know about cmac or that this value is unique to every released title.

Now that we have 100 titles with hash, it's getting harder to change over.

[honai]

I don't have the specs, but the unique Disc ID is defined in the DISCID.DAT file, offset 44, 16 bytes long.

I already wrote a library to extract that record (along with an XPL parser), it will be released this week, hopefully.

[Emon]

I propose the following format for the KeyDB


<?xml version="1.0" encoding="UTF-8"?>
<TitleIndex Version="1.0" Format="HDDVD">
<Titles>
<Title Description="12 Monkeys"
Origin="US"
ReleaseDate="YYYY-MM-DD"
Hash="1828B68D292D2EA1E9EEA1C7044DC864FDBC3EB6"
VolumeKey="2662C05B5238B0C50BD1BDF693223712" />

<Title Description="16 Blocks"
Origin="US"
ReleaseDate="YYYY-MM-DD"
Hash="7E0FE374FA11E4007228606E95E1A9B374D8EA69"
VolumeKey="DB8702D6FC60E08F27633EC2399371F5" />
</Titles>
</TitleIndex>

[hajj_3]

that looks good, is the region info located on the disc itself tho? like US or EUROPE?? if not you could have a drop down box with the 3 or so regions for the user to manually select.

[Zotty]

While we're at it suggesting ;)

All the fuzz is about storing the volume keys in a way they can be linked to the correct disc, nothing more, nothing less. So two values we're dealing with; a disc identifier and a VolumeKey, on a per disc basis;


<?xml version="1.0" encoding="UTF-8"?>
<KeyIndex version="1.0">
<Disc>
<Description>Movie name</Description>
<Timestamp>YYYY-MM-DD</Timestamp>
<Origin>EU</Origin>
<Id type="xxx">00000000000000000000000000000000</Id>
<Key>00000000000000000000000000000000</Key>
</Disc>

<Disc>
<Description>Movie name</Description>
<Timestamp>YYYY-MM-DD</Timestamp>
<Origin>EU</Origin>
<Id type="xxx">00000000000000000000000000000000</Id>
<Key>00000000000000000000000000000000</Key>
</Disc>

...
</KeyIndex>


Where type can be TKFMAC, SHA-1 hash, a general CMAC, some hash, a DISCID.DAT value, whatever.

[honai]

I've updated my suggestion:

http://forum.doom9.org/showthread.php?p=941290#post941290

This is likely the format I'll be using in my upcoming toolkit.

[melakai]

I've updated my suggestion:

http://forum.doom9.org/showthread.php?p=941290#post941290

This is likely the format I'll be using in my upcoming toolkit.

Looks like a good start, but I think it needs some fixing. here is my suggestion with reasons in comments:


<?xml version="1.0" encoding="UTF-8">
<ReleaseIndex majorVersion="1" minorVersion="0"> <!-- DiscIndex is not semantically correct; we can have multiple discs per release. fixed below -->
<Release format="[HDDVD|BD]" timestamp="[YYYY-MM-DD|YYYY]" Orgin="[US|EU]"> <!-- index removed, unneeded -->
<Title>[movie title]</Title>
<ScreenDate>[YYYY]</ScreenDate> <!-- optional; more movie details like imdb link can be added, all optional -->
<Disc discId="[discid]" DiscTitle="[Disc Title]"> <!-- allows multiple discs per release (i.e. TV series). DiscTitle optional, allows for "Episode 5-8" -->
<Key type="[tkfmac|title|volume]">[key]</Key> <!-- allows multiple keys per disc -->
</Disc>
</Release>
</DiscIndex>

This mostly fixes some issues arising from multiple-disc releases. It would be possible in the other format, but I think this would reduce duplicated information in those cases.

[Emon]

I think we won't ever come to an agreement on this. The best route would be to come up with a frontend for this DB on every different type of tools that we are coming up with.

[honai]

@melakai

Excellent work! I guess that fixes most of the issues of my suggestion.

[blutach]

I think we won't ever come to an agreement on this. The best route would be to come up with a frontend for this DB on every different type of tools that we are coming up with.Good idea. Eventually, I hope all the fine work can converge or else we are going to have more solutions than they have formats :)

Regards

[noclip]

There is absolutely no reason for any of the wackiness proposed by some of the posts here. The only two pieces of information needed to do 99% of what needs to be done are the TKF MAC and the volume key. Grouping multiple disk sets or creating separate "release" elements is just silly. The format in my original post and the main schema has a 1:1 disk to KeySet ratio, which is simple and functional.

[noclip]

I've updated my suggestion:

http://forum.doom9.org/showthread.php?p=941290#post941290

This is likely the format I'll be using in my upcoming toolkit.

Ugh. Why needlessly complicate things?

All that's really needed is a disk identifier/volume key pair, everything else is just for show and should therefore be optional. Grouping by release and mixing attributes and child elements is complicating things more than they need to be. If a disk gets another release, is from another region, or is part of a box set it will have a different TKF MAC. That's all the information needed to distinguish between disks in need of different volume keys.

[honai]

There is absolutely no reason for any of the wackiness proposed by some of the posts here. The only two pieces of information needed to do 99% of what needs to be done

"No one will ever need more than 640K RAM."

But seriously, there are many reasons to make the format as liberal (read: generic and adaptable) as possible and as strict as necessary. If you think you'll only need a few of the proposed elements then just write your parser that way and ignore the rest. Having more optional elements doesn't inflict any complexity on your own parser unless you decide to implement them.

[cyberpass]

i agree with honai. Some sort of XML standard would help speed development alot more.

[arnezami]

Well since we're at it I would like to suggest to add an optional VolumeID. They could become quite important as additional ammo for hackers at a later stage (they will have to try to retrieve keys by memory searching/debugging new versions of software players and the volumeid plays quite an important role in the decryption process and may be helpful to have) Right now we haven't been able to get any yet but that can change. ;)

For BD discs (who need a better unique disc/content identifier (http://forum.doom9.org/showthread.php?p=947595#post947595)) we might use the discid (in the discid.dat file although im not sure if ADV_OBJ is always present on every disc) or something in the Content Certificate (like the Content Sequence Number or better the Certificate itself or a hash of the CC file).

I think a general "id" tag and a general "key" is useful because it will make it very flexible and since we might need even more types of keys/ids in the future (like segment keys, volume/title variant keys etc) Zotty's suggestion sounds like a good idea although multiple variant keys might have to be grouped/numbered which may require some extension in the key tag.

Just my 2 cents... :)

[The_ByteMaster]

there are many reasons to make the format as liberal (read: generic and adaptable) as possible and as strict as necessary. If you think you'll only need a few of the proposed elements then just write your parser that way and ignore the rest. Having more optional elements doesn't inflict any complexity on your own parser unless you decide to implement them.

I agree with honai.

What we all do agree on is that some things are need-to-have and some things are nice-to-have. We disagree on what is nice-to-have and indeed, if it's nice at all to have.

So for the XML for HDDVD 1.0 'standard' let's first agree on the required elements before discussing the optional elements.

REQUIRED
- machine readable identifier
- volume unique key

Volume unique key is well-defined so that is already taken care of.

We need to discuss what can/will be used to uniquely identify a disc, keeping in mind that that our "opponent" might do whatever it can to thwart backuphddvd from functioning. For instance, on a 2-disc release like M:I 3 opponent would it be possible to keep the same SHA1 and TKF MAC, but using a different volume unique key to encrypt the disc?
If not, TKF MAC would IMHO the 'best' choice as it can also be used to validate the volume unique key, and validating the TKF MAC is of course done automagically if your disc is recognized.
For this to work we need to be absolutely positive that TKF MAC is and will remain to be truely unique among different pressings of the same movie/region/edition.

I am assuming we will see more re-pressings because when the movie goes out-of-stock but not out-of-print, the pressing plant might be required to update the revocation lists that are included on discs.

[arnezami]

So for the XML for HDDVD 1.0 'standard' let's first agree on the required elements before discussing the optional elements.

REQUIRED
- machine readable identifier
- volume unique key
Agreed.

Volume unique key is well-defined so that is already taken care of.
Agreed.

We need to discuss what can/will be used to uniquely identify a disc, keeping in mind that that our "opponent" might do whatever it can to thwart backuphddvd from functioning. For instance, on a 2-disc release like M:I 3 opponent would it be possible to keep the same SHA1 and TKF MAC, but using a different volume unique key to encrypt the disc?Its not possible to have the same TKF MAC with two different Volume Unique Keys (while both title key files being "meaningful" **). The MAC (http://en.wikipedia.org/wiki/Message_authentication_code) is - in our case of a title key file with large parts that have to be meaningful (eg the binding macs inside it) - essentially a "one-way hash" on both the data (title key file) and the volume unique key. So if you have a TKF MAC there can only be one valid volume key that has produced that MAC.

If not, TKF MAC would IMHO the 'best' choice as it can also be used to validate the volume unique key, and validating the TKF MAC is of course done automagically if your disc is recognized.
Agreed. Of course this only applies to HD DVD. Since only they use a TKF MAC. BD does not.

For this to work we need to be absolutely positive that TKF MAC is and will remain to be truely unique among different pressings of the same movie/region/edition.
Yes. The TKF MAC can only have one valid volume key. So if a disc if encrypted using a different volume key the TFK MAC will be different. Keep in mind that technically they could encrypt two different movies using the same volume key and the same title key file (unlikely but possible) so two discs would have the same TFK MAC. But that doesn't hurt us: the volume key is the same on both discs so we can decrypt them both. No problem there. It would even make our job easier since with one volume key we can now decrypt two discs! The (small) problem is that we can't distinguish the two discs using the TFK MAC anymore. But I'm not sure if that's important/desired. Don't think this is "REQUIRED" though...

(btw using the hash of the title key file has the same properties as the MAC since the title key file contains the MAC and the hash of the TKF is thus as unique as the MAC: muslix64 probably knew this too ;))

Ok HD solved :).

Now BD:

What is not practical is (only) using the hash of the title key file if there is no MAC in it. This is the case with BD only. The problem is that they could use the same title key file for all discs from now on and use different volume keys (as there have been examples of this already) and we would have to try each found key associated with that TKF Hash and see if something useful is decrypted (how do you quickly and reliably check whether a decrypted file is random or video/audio/png etc?). The origin of this problem is really this: BD doesn't give us a title key file MAC (using a volume key) nor does it give us a Binding MAC for each title key (using a title key itself). So we cannot quickly check whether a volume/title key is correct or not.

For this problem we need a solution: we need to somehow differentiate discs encrypted with a different CPS (=title) key without having the title/CPS key itself. To my knowledge the only way to do that properly is to use the hash(es) of the encrypted content (since content is encrypted with a certain CPS key it is unique for every CPS key).

A (robust) way would be to (literally) hash the encrypted files (in a standardized order) and use that as an identifier: this however takes a very long time for a standard PC and is thus impractical.

Another idea is to use the Content Hashes already made for us. These Content hashes are inside the Content Hash Tables (ContentHash00x.tbl: where x = 1 or 2 depending on the nr of layers). So a hash of all CHT's should suffice and would identify "encrypted content+key". These CHT's can be read and (if there is more than one) simply hashed in order (they are numbered). So that would work.

The (small) problem with this idea though is that they could on a new pressing (for example) shuffle all content hashes inside the Content Hash Table. That would mean there would be different disc identifiers with the same key. And that would be a bit of a shame on the size of our keylist (but this is limited by the amount of different pressings they are willing to make so I guess its not that bad). In fact its much more likely they will encrypt the new pressing with a new key which is far more effictive. In other words: using the hash of all Content Hash Files is a good way to identify BD discs.

As for revocation: revocation only works if they encrypt a disc with a different key. And if its encrypted with a different key then it has different content hashes (keep in mind they have hashed the encrypted content not the decrypted content). This of course means a single movie will have multiple versions (and thus multiple keys). But there is nothing we can do about that anyway. Pressings that are decrypted with a different key should have their own place in our key list.

Hope that helps.

Regards,

arnezami

** For those who like to know. Technically there is a way to create a different title key file with a different volume key resulting in the same MAC. However at least one of these title key files would have to be complete rubbish (as in: random bits). And since they would never release corrupt and unplayable discs (because of the corrupt title key file) we can be assured this is not a problem.

[cyberpass]

"I am assuming we will see more re-pressings because when the movie goes out-of-stock but not out-of-print, the pressing plant might be required to update the revocation lists that are included on discs."

As long as u have the volume key, this would not make a difference...The worry is they change the title keys in which case we need to identify the discs by version maybe?

[cyberpass]

arnezami - thanks!

[Ishan]

identification by date of manufacture would be better IMO

[noclip]

How is my idea not generic and adaptable?

<?xml version="1.0" encoding="UTF-8"?>
<KeyIndex majorVersion="1" minorVersion="0">
<Format>HD DVD</Format>
<KeySet> <!-- Volume key-based -->
<Description>Movie name</Description>
<Timestamp>YYYY-MM-DD</Timestamp>
<Origin>US</Origin>
<Key type="TKFMAC">00000000000000000000000000000000</Key>
<Key type="Volume">00000000000000000000000000000000</Key>
</KeySet>
<KeySet> <!-- Minimum, everything else is optional -->
<Key type="TKFMAC">00000000000000000000000000000000</Key>
<Key type="Volume">00000000000000000000000000000000</Key>
</KeySet>
</KeyIndex>

The system in my first post is about as flexible as it gets, you can have as little as 2 elements or as many as necessary to include all the data you want, including storing both title keys and the volume key for each disk. I am suggesting the format be as fundamental as possible to avoid any problems. What's the disagreement here?

Grouping multiple disk sets and using "releases" and then using "disk" sub-elements just serves to make the format more convoluted. Conversely, one-level elements grouped by volume key are much more readable and easier to code processing of in an application. Each VUK/TKFMAC pair getting its own element is just so much cleaner. Then you can optionally describe the disk that the pair is associated with.

If you start mandating inclusion of the date and titles, you run into problems. What if there are two different pressings of a movie, months apart whose keys are identical? This scenario is almost a certainty. You are suggesting including a "title" field while I am suggesting a "description" field. Description is more generic of the two, so why are you insisting on calling it a title?

As for the format, I still am not comfortable with using the same file on a user's drive for both. If some future generation HD DVD decrypting app is installed at C:\Program Files\HD Backup\ and a seperate Blu-ray app is installed at C:\Program Files\Blu Backup\, where would the KeySets.xml file be located? Most likely each app would have its own copy of the file with different key lists for each format. They would still use the same layout and schema, but they would be two separate physical files.

identification by date of manufacture would be better IMO

AACS-LA already handed us a perfect identifier (in fact, one commercial HD DVD players use) with the TKF MAC. It's consistently located, reliable, and is going to be grabbed from the disk anyway in order to test the validity of the VUK.

[honai]

Grouping multiple disk sets and using "releases" and then using "disk" sub-elements just serves to make the format more convoluted. Conversely, one-level elements grouped by volume key are much more readable and easier to code processing of in an application. Each VUK/TKFMAC pair getting its own element is just so much cleaner.

That's not true. You have to think from a progammer's point of view: How much complexity is needed to retrieve/update items in the XML database? Using your approach you'll have to write a lot *more* code because of the higher ambiguity of your DTD/Schema. Using the above consensus of melakai/me you'll only need a discrete XPath expression to retrieve any given element without further disambiguation.

If you start mandating inclusion of the date and titles, you run into problems.

You got that wrong, too. Those elements are also optional.

Now let me point out the problems of your suggestion:

<Format>HD DVD</Format>

If you plan on having two separate files, anyway, then that element should go into an attribute of the root element. But the problem of your approach remains: why define a global format when you could just have a common file for both formats, and set the format for each title?

<KeySet>

KeySet is the wrong context because the container entity shall not be a specific key but a specific release of a movie, including different discs. You simply got the hierarchy wrong.

Now the biggest problem:


<Description>Movie name</Description>
<Timestamp>YYYY-MM-DD</Timestamp>
<Origin>US</Origin>
<Key type="TKFMAC">00000000000000000000000000000000</Key>
<Key type="Volume">00000000000000000000000000000000</Key>


So how do you deal with movies/titles spanning several discs? This is where you'll have to introduce a lot of disambiguation in the software, namely: parsing the description/title strings and determining common patterns, because in the end you will want to group the list in a meaningful way to the end-user. But why work on the string/literal representation to introduce structure when the XML format already provides structured grouping?

Also, if you used your format and simply said, "Why, let's just give each disc another entry," you'd end up *replicating* a lot of information common to the release itself. On the other hand, if you simply said, "Ok, but each disc of the release has its own TKFMAC, anyway," then how do you extract the semantics of each disc when you simply work on <Key> elements?

Using melakai/mine approach you'll already have the title grouping inherent, and each disc will have its own sub-title ("Disc 3 of 4" or something like that).

Also, as a by-product the ReleaseIndex format above serves another purpose: maintaining a database of disc id's which can be neatly derived from the DISCID.DAT files. No need to write a TKFMAC calculator, but you simply extract the unique id from one file, which leads to *less* complexity, not more. Identifying a disc and getting the corresponding VUK is much easier with ReleaseIndex than KeySet.

[melakai]

As for the format, I still am not comfortable with using the same file on a user's drive for both. If some future generation HD DVD decrypting app is installed at C:\Program Files\HD Backup\ and a seperate Blu-ray app is installed at C:\Program Files\Blu Backup\, where would the KeySets.xml file be located? Most likely each app would have its own copy of the file with different key lists for each format. They would still use the same layout and schema, but they would be two separate physical files.

Everything else aside, mandating separate files due to schema is not flexible. By using other suggestions already posted (i.e. disc type is a property of the disc, rather than a property of the file) that allow for single file, you at least have an option of deciding if you want a single file (let's say, stored in %appdata%), or alternatively, separate files (i.e. %programfiles%\appname).

There's going to be people that want it both ways, and because of this, let's go with something flexible enough to work in either case.

[cwl7454]

Everything else aside, mandating separate files due to schema is not flexible. By using other suggestions already posted (i.e. disc type is a property of the disc, rather than a property of the file) that allow for single file, you at least have an option of deciding if you want a single file (let's say, stored in %appdata%), or alternatively, separate files (i.e. %programfiles%\appname).

There's going to be people that want it both ways, and because of this, let's go with something flexible enough to work in either case.

Have been sitting by watching all the different beta's and contoversy surrounding what to include and what not to include.
I concur w/ blutach on the converging of the different applications. I must say that Ripit4me was a godsend to us out here; It has all the plugins you need and links to the other needed apps, it has options; but default settings are already incorporated and set, as you will need them 99.9% ott anyways. If not protected disc, just use an alternative program.
Would love to see a GUI as mentioned above that gives you the option of 1 step or wizard (options). Willing to wait patiently for such an app just as we waited for RI4M. Majority of us are waiting for the price to come down substantually and still not decided which format to lean towards anyways.
As for me, I have so many versions of the different betas in my folder, starting to lose track of what goes with what.

Kudos to all who have gotten it this far. No need to rush it out just to get it out.
Thanks, once again to you all for all your time and effort so far, it IS appreciated.