BD-J platform development

[loo3aem3ON]

BD-J does not only provide the menu/gui for convenient movie playback. The BD-J applications (Xlets) also interact with the content code (BD+) via TRAP_ApplicationLayer. A basic BD-J platform implementation is therefor required to properly repair BD+ corrupted movies. Hopefully the project can be extended to a fully working BD-J platform allowing interactive menus in linux.

Our repository is hosted at launchpad: https://launchpad.net/bdj
Latest binary snapshot: download (http://uploaded.to/?id=d9etpf) (2009-03-01)
Sample applications from "The Day The Earth Stood Still" for testing: download (http://uploaded.to/?id=odplqu)
java platform (source): download (http://download.java.net/mobileembedded/phoneme/advanced/phoneme_advanced-mr1-rel-src-b06-10_nov_2006.zip) (don't compile this! instead use the libcvm_g.so from the binary snapshot)

I provide a precompiled library of of the java compact virtual machine (libcvm_g.so) for intel x86 compatible architectures because the compilation process (phoneme-advanced-mr1 based) is a bit difficult. Furthermore available are the core components of a working BD-J platform implementation (application manager, etc.) in form of (non-obfuscated) java bytecode (bdj.jar) from a licensed player.

completed tasks:
- [done] manually starting and running BD+ handshake application

current tasks:
- [started] parse the index.bdvm to obtain the playfirst BD-J object
- [testing] porting BD-J object parser/reader from java to C++
- [open] automatically load applications into cache
- [open] integrate index.bdmv and BD-J object handling
- [started] properly implement the native functions called via the java native interface

future tasks:
- [open] reverse engineer BD-J components implemented in java
- [open] menu support

[stottle]

loo3aem3ON-

Very nice work, and I hope to be able to help. I've got code for many of the blu-ray files (ported from hdcookbook to c++), but I need to convert from Visual c++ to linux.

Thanks for all of the hard work so far.

[kkloster21]

@loo3aem3ON:

you mentioned that the pre-compiled jvm will work with x86 architectures - will it work on amd64 archs (including intel 64-bit processors)? I'm guessing it won't?

and thanks by the way for all your hard work on this! i haven't stayed up on it as much as i'd like to have (i was able to get BD+ movies to play back when you guys first got it integrated into DumpHD but i haven't tested it since) but it'd be great to have a guide on how to get it all going.

[loo3aem3ON]

I've got code for many of the blu-ray files (ported from hdcookbook to c++), but I need to convert from Visual c++ to linux.
Thanks but i have to improve my C++ "skills" anyway. You could look over the code of the BD-J object reader (bdjo/ subdirectory) to see if it can be improved (bugs, memory leaks, whatever...).

you mentioned that the pre-compiled jvm will work with x86 architectures - will it work on amd64 archs (including intel 64-bit processors)? I'm guessing it won't?
Of course it will run on amd64 system but you will need some additional 32-bit libraries (and kernel support for 32-bit "emulation"). I have tried to create an amd64 version of the libcvm_g.so but that is a bit difficult as well because some parts of the code are written in assembler.

it'd be great to have a guide on how to get it all going.
I know there is a demand for a proper guide but i've decided to integrate the BD-J object reader first. Otherwise disc specific data (from "The Day The Earth Stood Still") which is currently hardcoded into the interface would be passed to the application manager (inside bdj.jar) regardless of what disc you have inserted. I'm currently working on the "index.bdmv" reader. If everything works well integration could be done in a few days.

In the meantime you can download the binary snapshot and check if the platform initialization goes well by executing "sh run.sh" and following the console messages. This test doesn't require a disc. You should see a black/white window in the left top corner of the screen.

[stottle]

Thanks but i have to improve my C++ "skills" anyway. You could look over the code of the BD-J object reader (bdjo/ subdirectory) to see if it can be improved (bugs, memory leaks, whatever...).
I took a quick look, and it looks like a pretty straightforward conversion from java to c++, without changing much.

Couple of notes:
readISO646String vs. readUTF8String - I'm not sure what the right functions are for reading UTF8 strings in linux, but I'm pretty sure what you have is incorrect. UTF8 allows for international characters that require more than 8 bits per character. If the bytes read in are all english, you will still get the right answer out, but non-ansi characters will break your code. If you already knew that and just hadn't gotten to fix that part, no problem, but I couldn't tell.

readShort/readInt etc - you might want to take a look at ntohs and ntohl, which will do the conversion probably faster than your hand coded byte handling. Nothing wrong with your code, just nice to know those functions are there, if you don't know about them. Ok, this really is a nit, but it seemed strange to just have one note....

On a larger scale, I found it necessary to change the architecture from what was in java. The code works fine for reading an existing object, but won't let you write your own out. Or make changes to the object and then write it out. It is also not very error resistant, and I wanted to be able to detect problems with the input better than that. So I think my implementation offers some benefits over a strict port from java to c++.

[loo3aem3ON]

The code works fine for reading an existing object, but won't let you write your own out. Or make changes to the object and then write it out.
The code is not supposed to modify any of the files on the disc so i removed the redundant methods. The average user can't modify a BD-ROM anyway. I've heard of some people who "open" BD-ROM discs but i don't think they can read ever read them again afterwards. :rolleyes:
It is also not very error resistant, and I wanted to be able to detect problems with the input better than that.
Yes i have some problems with the exception handling. It would be interesting to see your implementation.
So I think my implementation offers some benefits over a strict port from java to c++.
Thank you for your comments. We can use your code when it is available. In the meantime i will push forward with the integration of the BD-J object reader.

[stottle]

The code is not supposed to modify any of the files on the disc so i removed the redundant methods. The average user can't modify a BD-ROM anyway. I've heard of some people who "open" BD-ROM discs but i don't think they can read ever read them again afterwards. :rolleyes:
Ah, interesting comment. I was thinking more along the lines of TSMuxer, TSRemux or the hdcookbook code, which can create blu-rays. Possible example - people creating their own (unencrypted) blu-rays from home video.

[loo3aem3ON]

Ah, interesting comment. I was thinking more along the lines of TSMuxer, TSRemux or the hdcookbook code, which can create blu-rays. Possible example - people creating their own (unencrypted) blu-rays from home video.
What you mean is BD-J application development. This thread is about BD-J platform development. It works as follows:

1. read the BDMV/index.bdmv and get the PlayFirst entry (BD-J object id if BD-J type)
2. load the BD-J object BDMV/BDJO/?????.bdjo (????? is BD-J object id)
3. startup the application manager (bdj.jar) and pass configuration data as requested over JNI (java native interface)
4. copy the jar files mentioned in the AppCacheInfo section of the ?????.bdjo to the local cache (currently /tmp/BDJRoot/).
5. issue initTitle command over JNI
6. pass the data from from the ?????.bdjo to the application manager over JNI as requested

The application is now loaded from the cache and executed. The application did load other files directly from the disc. I've done all these steps manually before and now i'm trying to automate the process so it will work for other discs too.

[stottle]

What you mean is BD-J application development. This thread is about BD-J platform development.
:) I know. Reading the index.bdmv, movieobjects.bdmv and bdjo files will be necessary for either development, though. While my examples had more to do with app development (which is what got me started in the first place), I am interested in platform development which is why I posted....

[loo3aem3ON]

I am interested in platform development which is why I posted....
Well you are hereby invited to join the project and help. Please register at launchpad if you haven't done this already.

You will need the phonme-advanced-mr1 source snapshot for the headers. See my first posting (java platform) for the link.

[loo3aem3ON]

The BD-J object reader is now part of the interface (AppsDatabase) but the path is currently fixed to "/media/cdrom/BDMV/BDJO/07777.bdjo". Both the 77770.jar and 99900.jar still have to be manually copied to /tmp/BDJRoot/. Likewise a directory /DOCR/ with user write permission has to be created.

A new binary snapshot is available. You will need to press enter several times (not too fast) as part of the content code emulation during the handshake. The console output should look like this (http://uploaded.to/?id=k2cqn8).

[loo3aem3ON]

Both the index and the bdjo reader seem to be working now. For those who wanted to help testing a new build snapshot is available (see first posting of this thread -> direct link (http://uploaded.to/?id=d9etpf)).

Instructions:
1. mount the disc at /media/cdrom
2. extract the build snapshot into a directory
3. run "sh run.sh" from that directory

Some directories (/tmp/...) will be created and the applications (*.jar) will be loaded automatically into the cache (/tmp/BDJRoot/).

It will only start the applications associated with the firstPlay BD-J index object. The latest batch of BD+ corrupted discs should have something to execute there.

Nothing spectacular will happen (no graphics/menu/sound and the like). It should execute the BD+ handshake if there is any. Remember to press enter in that case.

Post your results here please.

[stottle]

Ok, I've got things started. I had to download a 32 bit version of qt, and add the install directory to the LD_LIBRARY_PATH in the script, but now I get a black window on the screen before the script locks up.

I'm running against "The Happening" since I don't have tdtess (in particular, I don't have a copy of index.bdmv for it). The problem looks to be in AppsDatabase.cpp, which is looking for a certificate file in /media/cdrom0/. Note, cdrom0, not cdrom. That causes the file not to be found, which is the error I get before the program stalls.

I'll try fixing this on my end, but I thought I'd pass this along.

BTW, the LD_LIBRARY_PATH line (in run.sh) gives an error because it has $(LD_LIBRARY_PATH) instead of ${LD_LIBRARY_PATH}.

[stottle]

Fixing the code was easy (just changed cdrom0 to cdrom) and compiling worked fine once I installed an i386 version of glibc (for crti.o) and changed the location of the phoneme source in the makefile. Nice job putting all of this together. I'm still going through the output, but it seems to be working so far.

[lbalan79]

Just to note while compiling I get the following:

src/bdj.cpp:261:2: warning: no newline at end of file
g++: -lbdjinterface: linker input file unused because linking not done
g++: -lcvm_g: linker input file unused because linking not done
g++: -ldl: linker input file unused because linking not done
g++: -lpthread: linker input file unused because linking not done
g++: -lm: linker input file unused because linking not done


Probably this warning does not affect the end result but I thought to post it here just in case.

[lbalan79]

My compiled version throws the following exception. It is true though that I run it under a 64 bit machine. So any help would be appreciated.

./run.sh
CVM Configuration:
Java stack chunk size (stackChunkSize): 1024
Java stack minimum size (stackMinSize): 1024
Java stack maximum size (stackMaxSize): 131072
GC[SS]: Initialized semi-space gen for generational GC
Size of *each* semispace in bytes=1048576
Limits of generation = [0xf72c6200,0xf74c6200)
First semispace = [0xf72c6200,0xf73c6200)
Second semispace = [0xf73c6200,0xf74c6200)
GC[MC]: Initialized mark-compact gen for generational GC
Size of the space in bytes=4194304
Limits of generation = [0xf74c6200,0xf78c6200)
GC[generational]: Auxiliary data structures
heapBaseMemoryArea=[0xf72c6008,0xf78c6208)
cardTable=[0x80521f0,0x80551f0)
objectHeaderTable=[0x80551f8,0x80581f8)
summaryTable=[0x8058200,0x8064200)
"/home/liviu/bdj/lib/basis.jar" is not a valid classpath zip file or directory.
JVM created
Using library: /home/liviu/bdj/bin/libbdjinterface.so
Reading /media/cdrom/BDMV/index.bdmv...
Reading /media/cdrom/BDMV/BDJO/07777.bdjo
Calling initBDJ...
Java.com.cl.bdj.helper.CUtil.pGetRegistryInt: ENABLE_LOGOTESTING, 0 -> 0
Java.com.cl.bdj.helper.CUtil.pGetRegistryInt: PaintingMaxFPS, 80 -> 80
Java.com.cl.bdj.helper.CUtil.pGetRegistryInt: DisableBDJGraphic, 0 -> 0
Java.com.cl.bdj.helper.CUtil.pGetRegistryInt: PaintingLatency, 100 -> 100
Java.com.cl.bdj.helper.CUtil.pGetRegistryInt: LogThreshold, 100 -> 0
Java.com.cl.bdj.helper.CUtil.pGetRegistryInt: LogShowTime, 0 -> 0
Java.com.cl.bdj.helper.CUtil.pGetRegistryInt: LogFullClassname, 0 -> 0
Java.com.cl.bdj.helper.CUtil.pGetRegistryInt: BDJTraceGraphicCommand, 0 -> 0
Java.com.cl.bdj.helper.CUtil.pGetRegistryInt: BDJTraceKeyEvent, 0 -> 0
Java.com.cl.bdj.helper.CUtil.pGetRegistryInt: BDJTraceAll, 0 -> 1
Java.com.cl.bdj.helper.CUtil.pGetRegistryInt: BDDefProfile, 3 -> 3
http.proxyHost = null
http.proxyPort = null
https.proxyHost = null
https.proxyPort = null
socksProxyHost = null
socksProxyPort = null
called Java.com.cl.bdj.helper.AppsDatabaseHelper.pGetAppCacheBase
created one file output stream : /home/liviu/javaLog.txt


ä¸*æLog started: Mon Mar 02 19:30:03 GMT-05:00 2009
called Java.com.cl.bdj.helper.CUtil.pGetRegistryString -> LogFilter, *
[BDJAppManager.initial()] initialLogging() finished
called Java.java.awt.QtImage.initIDs -> NOT IMPLEMENTED!
QPainter::begin: Paint device cannot be null
QPainter::setClipRegion: Will be reset by begin()
QPainter::setClipping: Will be reset by begin()
QPainter::setPen: Will be reset by begin()
QPainter::setBrush: Will be reset by begin()
QPainter::setBrush: Will be reset by begin()
QPainter::setPen: Will be reset by begin()
[BDRootScene.getRootScene()] BDRootScene.notifyDirty
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 1
[HGraphicsHelper.init()] m_image.getWidth 1920
[HGraphicsHelper.init()] m_image.getHeight 1080
[HGraphicsHelper.init()] m_image.type 1
JMFProperties: readProperties
[BDJAppManager.initial()] initial() finished
called Java.com.cl.bdj.helper.AppsDatabaseHelper.pGetInitHAVIConfigId()
called Java.com.cl.bdj.helper.BDJPlaneControlHelper.pGetScreenSize
Current Free Mem = 4694512
[BDJCFacade.executeCommand()] Start init Title
called com.cl.bdj.helper.BDJTitleInfoHelper.pGetCurrentDiscIdFromC
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 2
called Java.com.cl.bdj.helper.BDJVFSHelper.pGetStorageRoot -> 3
called Java.com.cl.bdj.helper.BDJMessageHelper.pSetRequest -> 9, 0 -> NOT IMPLEMENTED!
invoke_executeCommand returned -357152
java.lang.OutOfMemoryError: [B
at sun.misc.CharacterDecoder.decodeBuffer(Ljava/lang/String;)[B(Unknown Source)
at com.cl.bdj.helper.BDJMessageHelper.SetRequest(JII[B)[B(Unknown Source)
at com.cl.bdj.helper.BDJMessageHelper.isTimeStampReserved()Z(Unknown Source)
at com.cl.bdj.main.BDJCFacade.checkOnlineCapability()V(Unknown Source)
at com.cl.bdj.main.BDJCFacade.initTitle(I)V(Unknown Source)
at com.cl.bdj.main.BDJCFacade.executeCommand(I[Ljava/lang/Object;)I(Unknown Source)
called Java.java.awt.QtImage.pGetQImageBits -> NOT IMPLEMENTED!
called Java.java.awt.QtImage.pGetQImageBytesPerLine -> NOT IMPLEMENTED!
Exception occurred during event dispatching:
java.lang.UnsatisfiedLinkError: com.cl.bdj.helper.GraphicsPlaneImp.pClearFrameImp(JIIII)I
at com.cl.bdj.helper.GraphicsPlaneImp.clearImage(Ljava/awt/image/BufferedImage;)V(Unknown Source)
at com.cl.bdj.helper.GraphicsPlaneImp.clearFrame(Ljava/awt/image/BufferedImage;)V(Unknown Source)
at com.cl.bdj.main.BDRootScene.clearFrame()V(Unknown Source)
at com.cl.bdj.main.BDRootScene.updateAll()V(Unknown Source)
at com.cl.bdj.main.BDRootScene.updateFrame()V(Unknown Source)
at com.cl.bdj.main.BDRootScene.syncGraphics()V(Unknown Source)
at com.cl.bdj.appmanager.BDJPluginImp.toolkitSync()Z(Unknown Source)
at java.awt.QtToolkit.sync()V(Unknown Source)
at java.awt.Component.dispatchEventImpl(Ljava/awt/AWTEvent;)V(Unknown Source)
at java.awt.Window.dispatchEventImpl(Ljava/awt/AWTEvent;)V(Unknown Source)
at java.awt.Component.dispatchEvent(Ljava/awt/AWTEvent;)V(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Ljava/awt/AWTEvent;)V(Unknown Source)
at java.awt.EventQueueProxy.dispatchEvent(Ljava/awt/AWTEvent;)V(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEvent(I)Z(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(ILjava/awt/Conditional;)V(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Ljava/awt/Conditional;)V(Unknown Source)
at java.awt.EventDispatchThread.run()V(Unknown Source)
at java.lang.Thread.startup(Z)V(Unknown Source)
Current Free Mem = 4947384
Current Free Mem = 4946872

[loo3aem3ON]

Just to note while compiling I get the following:
The warning about the missing newline at the end of the bdj.c is correct. I don't know about the other warnings but these libraries are required and you should get lots of "undefined symbol" error without them.
Probably this warning does not affect the end result but I thought to post it here just in case.
Try "make clean" before "make" to see if the linker produces a proper "bdj" and "libbdjinterface.so" in the subdirectory "bin/"
My compiled version throws the following exception. It is true though that I run it under a 64 bit machine. So any help would be appreciated.
I've so far failed to create a amd64 version of the "libcvm_g.so" so you currently need to use the i386 version. This is why you have to compile everything with the -m32 switch to produce 32-bit x86 code. As far as i understand it's not possible to link a 32-bit library to a 64-bit binary.

Your first step should be to try to get the build/binary snapshot working. You will probably need to install some extra 32-bit libraries. Then you can proceed to compile the sources yourself.

Are you getting the "OutOfMemoryError" message with my build/binary snapshot too? What changes have you done to the source and the makefile?

[jansen]

When running run.sh with "The dark knight" mounted at /dev/cdrom I see this results:

CVM Configuration:
Java stack chunk size (stackChunkSize): 1024
Java stack minimum size (stackMinSize): 1024
Java stack maximum size (stackMaxSize): 131072
GC[SS]: Initialized semi-space gen for generational GC
Size of *each* semispace in bytes=1048576
Limits of generation = [0xb7350200,0xb7550200)
First semispace = [0xb7350200,0xb7450200)
Second semispace = [0xb7450200,0xb7550200)
GC[MC]: Initialized mark-compact gen for generational GC
Size of the space in bytes=4194304
Limits of generation = [0xb7550200,0xb7950200)
GC[generational]: Auxiliary data structures
heapBaseMemoryArea=[0xb7350008,0xb7950208)
cardTable=[0x8051248,0x8054248)
objectHeaderTable=[0x8054250,0x8057250)
summaryTable=[0x8057258,0x8063258)
"/home/joep/Documents/blueray/1/trunk/lib/basis.jar" is not a valid classpath zip file or directory.
JVM created
Using library: /home/joep/Documents/blueray/1/trunk/bin/libbdjinterface.so
Reading /media/cdrom/BDMV/index.bdmv...
Error: FirstPlaybackObject type is not BD-J.

Does this mean The dark knight is not a BD-J disk?
Greetings,
Jansen

[KenD00]

Well, it means that the First Playback Object is not a BD-J Object but a HDMV Object. A disc can have many objects (like Title Sets on a DVD), either HDMV or BD-J. My disc has 4 objects and the last one is a BD-J object.

:rolleyes:

[loo3aem3ON]

I observed that the BD-J/BD+ handshake takes place very early during initialization long time before the menu gets loaded. The BD-J object performing the handshake was referenced by the FirstPlaybackObject entry in the index.bdmv. If this entry doesn't refer to a BD-J object it doesn't mean the disc doesn't contain BD-J programs but you will see this error message currently.
The code has to be connected to libbluray somehow (either merged into a single process or two communicating processes) to repair the newer BD+ corrupted movies.