HDDVDBackup C/C++ Port

[NghtShd]

I had some free time today, so I did a C++ port. The code is mostly just written like straight C. If anyone wants to see if it works or if it offers any speed advantage here the app is:

http://www.sendspace.com/file/sv2i90 (http://www.sendspace.com/file/sv2i90)

No GUI yet and, as usual, I have no actual HD-DVD files to test with so it may not work on the real thing.

BTW, I used Brian Gladman's AES libs. They are supposed to be fast, but I'm not sure which version of his libs are best for this application.

If it's confirmed that it works then I'll release the code after a few tweaks. I'll probably slap a gui on it too, but I've kept the backend code fairly generic so it sould be easy enough to port to other platforms.

[arnezami]

I had some free time today, so I did a C++ port. The code is mostly just written like straight C. If anyone wants to see if it works or if it offers any speed advantage here the app is:

http://www.sendspace.com/file/c36a9s (http://www.sendspace.com/file/c36a9s)

No GUI yet and, as usual, I have no actual HD-DVD files to test with so it may not work on the real thing.

BTW, I used Brian Gladman's AES libs. They are supposed to be fast, but I'm not sure which version of his libs are best for this application.

If it's confirmed that it works then I'll release the code after a few tweaks. I'll probably slap a gui on it too, but I've kept the backend code fairly generic so it sould be easy enough to port to other platforms.

If this works it would be great because we could recompile it (for the Cell/PPC processor) and try to decrypt a movie using the PS3 itself :). After confirmation the source would indeed be very useful.

[LokiHD]

so, whos up to a gui on the C++ ?

[MrDVD]

If this works it would be great because we could recompile it (for the Cell/PPC processor) and try to decrypt a movie using the PS3 itself :). After confirmation the source would indeed be very useful.

I wonder how you want mount the hddvd drive a ripped iso on your PS3. As i know there is no UDF2.5 support for linux :(

[NghtShd]

so, whos up to a gui on the C++ ?

I'm actually working on something now. Thinking about a good way to hook into the base code (for progress feedback and such) without windowsifying it too much.

[NghtShd]

I'm pretty far along in the process of adding a GUI. In the process I decided to do some rewriting of a lot of the backend code and I found and fixed one bug in the pack decryption setup, so I know the current release doesn't decrypt properly, though it could still be useful in finding out if C/C++ offers a speed advantage.

On that note, I'm also doing some optimizations. There's a lot of uneccessary memory copying going on, due to the code being so directly ported from a pointerless language. I don't know how much it will help to speed things up, but it can't hurt.

[XKCorp]

it would be nice if you could release your source code when it will run fine :)

[LokiHD]

I'm pretty far along in the process of adding a GUI. In the process I decided to do some rewriting of a lot of the backend code and I found and fixed one bug in the pack decryption setup, so I know the current release doesn't decrypt properly, though it could still be useful in finding out if C/C++ offers a speed advantage.

On that note, I'm also doing some optimizations. There's a lot of uneccessary memory copying going on, due to the code being so directly ported from a pointerless language. I don't know how much it will help to speed things up, but it can't hurt.

alright when done, ill take it for spin

[kad77]

Hey NightShd,

I was testing AES in a home-brew tftp setup a while back, and I found Gladman's implementation to be really fast (plus he seems to devote a *lot* of time to refining it). There are generic c/c++ routines that are portable, and of course his assembler routines which you could optionally link with #ifdefs.

You can also look at using the Christhophe Devine's tidy XySSL AES and SHA1 routines, they are more generic, short and probably written to run well on more than just x86/x64.

I haven't benchmarked them, but I'm sure they are quite fast. Given the drive data load delays, you may be fine using them even though they may not be as cycle efficient as Gladman's.

see: http://xyssl.org/code/source/aes/

keep on hacking! keep in mind that BackupHDDVD's decryption is broken! cwm9 seems to be working on it, anyone talk to him?

Everyone else is beating around the bush... :sly:

[Momotte]

Also, I haven't looked closely, but I think openssl contains all the necessary routines for AES, SHA1, etc... it is in c/c++ and cross platform. Optimization are also done in assembly in that library. Might be worth to take a look.

[NghtShd]

If anyone wants to test, I'd love to know 1) if it works, 2) if it's fast/slow, 3) if changing the chunksize has any effect on speed. The chunk sizes available are 32k, 64k, 128k, and 256k. I don't see much difference between them, but I only have my test files to work with. I also tested with 8K and 1024K, both of which were slow.

Note that there could be bugs with the double buffering code. It checks out whith my test files, but all output blocks are identical in those files (I should write something to generate test files with random data, but I'm lazy).

There is probably a lot more that can be done to speed up the code, assuming it isn't disk bound. Also, the code has to wait on reads right now. I'm thinking about another thread to fill the buffers (to go with the one which empties them). I'm also thinking about better ways to arrange this buffering.

http://www.sendspace.com/file/sv2i90

http://img77.imageshack.us/img77/6474/hddqq3.gif

Fixed:

* AES setup bug

Features:

* Graphical user interface
* Decrypting happens in its own thread and can be aborted at any time
* File and total job progress bars
* Start/stop and elapsed time output

Optimizations (which may have no effect at all):

* Removed tons of unnecessary memory copy operations
* File writing done in separate thread with double buffer
* Multiple write operations consolidated into one chunk sized write
* Chunk size (the size of the chunks of file loaded for decryption) adjustable

Needs fixing:

* IME or anything else that isn't in the java 1.0 app
* More error control
* UI needs work
* A lot of other stuff

[NghtShd]

Hey NightShd,

I was testing AES in a home-brew tftp setup a while back, and I found Gladman's implementation to be really fast (plus he seems to devote a *lot* of time to refining it). There are generic c/c++ routines that are portable, and of course his assembler routines which you could optionally link with #ifdefs.

You can also look at using the Christhophe Devine's tidy XySSL AES and SHA1 routines, they are more generic, short and probably written to run well on more than just x86/x64.

I haven't benchmarked them, but I'm sure they are quite fast. Given the drive data load delays, you may be fine using them even though they may not be as cycle efficient as Gladman's.

see: http://xyssl.org/code/source/aes/

keep on hacking! keep in mind that BackupHDDVD's decryption is broken! cwm9 seems to be working on it, anyone talk to him?

Everyone else is beating around the bush... :sly:

Thanks for that info. I'm using one of the asm versions (V2) of Gladman's code. Not sure wich version is best in this context.

@XKCorp

Yes the full source will be released. I've had to C++-ify it a bit inorder to more easily work it into my MFC frontend, but the code remains fairly generic and should be easy to adapt to any platform. I put in hooks for printing messages and reporting progress via optional callbacks, so you can just use stdio routines or whatever suits your needs.

Also, I haven't looked closely, but I think openssl contains all the necessary routines for AES, SHA1, etc... it is in c/c++ and cross platform. Optimization are also done in assembly in that library. Might be worth to take a look.

Thanks, Momotte, I'll take a look.

[LokiHD]

If anyone wants to test, I'd love to know 1) if it works, 2) if it's fast/slow, 3) if changing the chunksize has any effect on speed. The chunk sizes available are 32k, 64k, 128k, and 256k. I don't see much difference between them, but I only have my test files to work with. I also tested with 8K and 1024K, both of which were slow.

Note that there could be bugs with the double buffering code. It checks out whith my test files, but all output blocks are identical in those files (I should write something to generate test files with random data, but I'm lazy).

There is probably a lot more that can be done to speed up the code, assuming it isn't disk bound. Also, the code has to wait on reads right now. I'm thinking about another thread to fill the buffers (to go with the one which empties them). I'm also thinking about better ways to arrange this buffering.

http://www.sendspace.com/file/sv2i90

http://img77.imageshack.us/img77/6474/hddqq3.gif

Fixed:

* AES setup bug

Features:

* Graphical user interface
* Decrypting happens in its own thread and can be aborted at any time
* File and total job progress bars
* Start/stop and elapsed time output

Optimizations (which may have no effect at all):

* Removed tons of unnecessary memory copy operations
* File writing done in separate thread with double buffer
* Multiple write operations consolidated into one chunk sized write
* Chunk size (the size of the chunks of file loaded for decryption) adjustable

Needs fixing:

* IME or anything else that isn't in the java 1.0 app
* More error control
* UI needs work
* A lot of other stuff

fast! for hdd to hdd

Running time: 00:04:24 @ 128k

nice work!

edit: ill do all the buffers and post the results

[Oleg_Jdev]

If anyone wants to test, I'd love to know 1) if it works, 2) if it's fast/slow, 3) if changing the chunksize has any effect on speed. The chunk sizes available are 32k, 64k, 128k, and 256k. I don't see much difference between them, but I only have my test files to work with. I also tested with 8K and 1024K, both of which were slow.

Note that there could be bugs with the double buffering code. It checks out whith my test files, but all output blocks are identical in those files (I should write something to generate test files with random data, but I'm lazy).

There is probably a lot more that can be done to speed up the code, assuming it isn't disk bound. Also, the code has to wait on reads right now. I'm thinking about another thread to fill the buffers (to go with the one which empties them). I'm also thinking about better ways to arrange this buffering.

http://www.sendspace.com/file/sv2i90

http://img77.imageshack.us/img77/6474/hddqq3.gif

Fixed:

* AES setup bug

Features:

* Graphical user interface
* Decrypting happens in its own thread and can be aborted at any time
* File and total job progress bars
* Start/stop and elapsed time output

Optimizations (which may have no effect at all):

* Removed tons of unnecessary memory copy operations
* File writing done in separate thread with double buffer
* Multiple write operations consolidated into one chunk sized write
* Chunk size (the size of the chunks of file loaded for decryption) adjustable

Needs fixing:

* IME or anything else that isn't in the java 1.0 app
* More error control
* UI needs work
* A lot of other stuff
Nice.
What the compiler you use?

[Momotte]

Nice.
What the compiler you use?

My bet is that he is using MS Visual Studio since he is talking about MFC... that is perfect with me, I am using the same :)

[LokiHD]

Running time: 00:05:29 @ 32k
Running time: 00:04:41 @ 64k
Running time: 00:04:24 @ 128k
Running time: 00:03:50 @ 256k


if you would like to experiment with 512k, 1024k, 2048k chunks, no problem, im still not hardware limited

[Nomadic]

2 LokiHD: test it on real HDDVD disk

[LokiHD]

i dit it once more by close/open hd decrypter after each chunk,

Running time: 00:06:18 @ 32k
Running time: 00:04:43 @ 64k
Running time: 00:03:50 @ 128k
Running time: 00:03:28 @ 256k


ok Nomadic thats next but its not gonna be any faster than 1 hour

[hajj_3]

looks impressive, might give it a whirl later tonight.

[NghtShd]

Nice.
What the compiler you use?

Yeah, it's Visual Studio. The core decryption code should be easy enough to move to anything.

[NghtShd]

I just realized I disabled the threaded write in the last build, so that isn't actually even happening yet. With that on it's about 20% faster on my test files. I need to do a bit more work on some things before I release a new version, though.

[LokiHD]

2 LokiHD: test it on real HDDVD disk

Running time: 01:00:12 @ 32k

[LokiHD]

I just realized I disabled the threaded write in the last build, so that isn't actually even happening yet. With that on it's about 20% faster on my test files. I need to do a bit more work on some things before I release a new version, though.

whoa, nice..

awesome work!

[Momotte]

the "Job Progress" seems to be broken too... the whole progress is flickering instead of showing the true %

[NghtShd]

@LokiHD

Thanks for all the feedback. It's appreciated.

the "Job Progress" seems to be broken too... the whole progress is flickering instead of showing the true %

Ok, thanks. I'm not seeing that, but I'll see if I can track it down. The call to incriment it looks fine so it would probably be the initializing of the bar's max setting. Maybe I'm overlowing the signed int?

I had a race condition in the threaded writer that caused a truncation of the next to last chunk, which is fixed now. I think threading and buffering the read may be more beneficial when reading from optical media, but it might make it slower if the same disk were used for reading and writing. Perhaps logic to turn it off in that case or at least user options are in order.

[Momotte]

could be an overflow... the bar keep flickering and shows a full progress even though it is now doing the feature_2.evo of T.I.J.

[ShadowFlare]

From the comments, it seems at least the internal parts of the program work just fine; so, when are you going to release the source code? :) I was actually considering porting BackupHDDVD myself (but to C), but I don't think I'll do that since you've released a C++ version already. However, when you do release the source code, I think I may try porting your version to C and possibly combining it with Zotty's ACA program.

[kad77]

From the comments, it seems at least the internal parts of the program work just fine; so, when are you going to release the source code? :) I was actually considering porting BackupHDDVD myself (but to C), but I don't think I'll do that since you've released a C++ version already. However, when you do release the source code, I think I may try porting your version to C and possibly combining it with Zotty's ACA program.

Whoa there buddy.

Read: http://forum.doom9.org/showthread.php?p=941169#post941169

If by 'internal parts' you mean the main decryption algorithm, you are not current with the state of BackupHDDVD. It is incomplete, and muslix64 has publicly stated (I asked him) he is not going to maintain it anymore.

Read the above link to read what the only person (I can find) is doing to push the code out of beta, and fix the AACS decryption.

[ShadowFlare]

Whoa there buddy.

Read: http://forum.doom9.org/showthread.php?p=941169#post941169

If by 'internal parts' you mean the main decryption algorithm, you are not current with the state of BackupHDDVD. It is incomplete, and muslix64 has publicly stated (I asked him) he is not going to maintain it anymore.

Read the above link to read what the only person (I can find) is doing to push the code out of beta, and fix the AACS decryption.
Yeah, I've read about that already. When I said "From the comments, it seems at least the internal parts of the program work just fine" I meant in comparison to the other ports, this port sounds like it is working just as well.

Unfortunately, I can't really help with fixing that part, for a couple of reasons: 1) I do not really understand the information in that text file and 2) I do not have access to any files to look at while reading that information such as to help me understand it (I do not have an HD-DVD drive or any movies in that format).

[kad77]

It's fun to write code, and improve on open-source projetcs.

I guess, as I have stated elsewhere, I oppose the proliferation of programs that will fsck up the backup. The interface may be helpful, the speed may be faster, but if the output of all these ports is basically proof-of-concept and not truly usable backups, they should not be widely distributed.

When a few good coders get together and solve the implementation errors and oversights, all will benefit. Until then, keep developing other aspects of the program *but* disclaim them as not-ready-for-primetime, as they certainly are not.

You will just spawn more newcomers asking "why doesn't my backup WORK?"

[Isochroma]

Yes, I agree with kad77: just think how many terabytes of storage (a lot permanently committed on DVD recordables) is being wasted by storing corrupt copies of HD-DVDs.

All those recordings will have to be thrown out once the code is fixed. It will be an environmental disaster.

Each minute that ticks by, more toxic data is being saved. Thus, it is essential that the problems be fixed with all haste, to avoid unnecessary waste.

[NghtShd]

Here's the app:
http://www.sendspace.com/file/0t8izc (http://www.sendspace.com/file/0t8izc)

Here's the source:
http://www.sendspace.com/file/a720c4 (http://www.sendspace.com/file/a720c4)

Here's the mostly C compatible source code for the commandline version:
http://www.sendspace.com/file/ptwv91 (http://www.sendspace.com/file/ptwv91)

The ReadMe for the app:

HD Decrypter 0.3


History

Fixed:

+ Threaded file write was inadvertently disabled

Features:

+ Added more chunk size options

Optimizations (which may have no effect at all):

* Threaded file writes advertently disabled
* Using filemapping instead of write thread

Notes:

I have no HD-DVD drive or movie files to work with. All testing was done with
artificial EVO files I created myself. My test machine is a Conroe E6600
running at 3.3GHz with 2GB of dual channel RAM. Tests were done with lots of
stuff running in the background and on moderately fragmented drives.

I experimented with various threaded reading and writing schemes so that
reading and/or writing could take place while decryption was processing. I saw
about a 20% to 25% speed gain with the write thread using a 128k chunk size.
Using a reading thread was less helpful. Using both reading and writing threads
was less helpful still. I got my best results, about a 35% speed gain, using
filemapping. The filemapping scheme maps the destination file to memory and
then does fread() calls reading directly into the memory mapped file and doing
the decryption in place.

Your results may vary. Most of my tests involved test files of only 200MB and
HD to HD decrytion using a USB connected laptop HD as the source and a standard
IDE drive as the destination. I did do some tests from an optical drive, but
the caching of the relatively small test files made for far from real-world
results there too. I suspect that the rest of the code will spend most of its
time waiting on the optical drive when ripping retail movie.


Version 0.2

Fixed:

* AES setup bug

Features:

* Graphical user interface
* Decrypting happens in its own thread and can be aborted at any time
* File and total job progress bars
* Start/stop and elapsed time output

Optimizations (which may have no effect at all):

* Removed tons of unnecessary memory copy operations
* File writing done in separate thread with double buffer
* Multiple write operations consolidated into one chunk sized write
* Chunk size (the size of the chunks of file loaded for decrypted) adjustable

Needs fixing:

* IME or anything else that isn't in the java 1.0 app
* More error control
* UI needs work
* A lot of other stuff

[Momotte]

thanks, very appreciated.

[LokiHD]

^ @ NghtShd, you're welcome


v0.3

regardless of the chunck size runtime @ 00:00:30

output folder @ 3.42GB
feature1 @ 1,301,162KB
feature2 @ 1,768,116KB

the rest of the files are fine

[noclip]

Here is a UI mockup I think would improve user-friendliness and be more advanced:

http://img441.imageshack.us/img441/3061/bhdmockbz5.png

[NghtShd]

Here's the app:
http://www.sendspace.com/file/0t8izc (http://www.sendspace.com/file/0t8izc)

Here's the source:
http://www.sendspace.com/file/a720c4 (http://www.sendspace.com/file/a720c4)


HD Decrypter 0.3.1

Fixed:

+ Bug causing missing chunk in output may have been in the last release.

Optimizations (which probably have no effect at all):

* Using filemapping for reads as well as writes.