Error correction for XCD is made

[Elic]

Hi there!

I wrote error correction program for XCD. Strictly speaking, there are two programs in one package called RR (as "CD-R eRRoR coRRectoR"):
rrenc - reads all of files which are to be written into XCD, computes an array of redundant data (usual 16-bit Solomon-Reed code) and prepares paramfile for mode2cdmaker;
rrdec - reads all of files (including redundant data) from XCD and writes them to your hard disk, computes the contents of damaged sectors and insert them into proper places.

There is a thread about RR program package on http://forum.sharereactor.ru/showthread.php?t=59777. The thread is in Russian (and my native language is Ukrainian), but I made my programs shipped with English command-line interface. You can download entire package from http://rapidshare.de/files/1260596/rr119.zip.html but this is not persistent link; please check alternate links located on first post of ShareReactor's thread mentioned above.

Sorry, but my English is weak and this is my first message on this forum. I plan to write docs for RR in Russian and English nearly simultaneously (Russian first :) of course) and I will appreciate if Russian-speaking visitors vill verify my translations.

2006/02/01
The package is resides of its own site (http://elicit.vn.ua/) now. Last version, 1.24, is splitted into two packages - executables (http://elicit.vn.ua/files/rr/rr-1.24.main.zip) and extra files (http://elicit.vn.ua/files/rr/rr-1.24.extra.zip). All previous publicly available versions are here (http://elicit.vn.ua/files/outdated/).

2006/02/16
Version 1.26 (http://elicit.vn.ua/files/rr/rr-1.26.update.zip) is released. Full description is in this post (http://forum.doom9.org/showthread.php?p=786285#post786285).

2006/02/20
Version 1.27 (http://elicit.vn.ua/files/rr/rr-1.27.update.zip) is released; description (http://forum.doom9.org/showthread.php?p=788500#post788500).

2006/02/24
Version 1.28 (http://elicit.vn.ua/files/rr/rr-1.28.update.zip), and there will be front-end (http://home.arcor.de/kerberos002/cdrtfe/download_en.html) soon (details are here (http://forum.doom9.org/showthread.php?p=790433#post790433)).

2006/02/28
Version 1.29 (http://elicit.vn.ua/files/rr/rr-1.29.update.zip), front-end (http://home.arcor.de/kerberos002/cdrtfe/download_en.html), details (http://forum.doom9.org/showthread.php?p=791799#post791799) :)

2012/11/02
Version 1.30 (https://sourceforge.net/projects/cd-rr/files/cd-rr-1.30/?) : win32 binaries (http://sourceforge.net/projects/cd-rr/files/cd-rr-1.30/cd-rr-1.30.win32.7z/download), GPLed source code (http://sourceforge.net/projects/cd-rr/files/cd-rr-1.30/cd-rr-1.30.src.7z/download), details (http://forum.doom9.org/showpost.php?p=1598794&postcount=38)

[ukb008]

Your English is quite OK. Welcome to The Forum and thanks for the Pie. We're all waiting for the guide, if and when you have the time.

What do the XCD developers say about this? The related posts in the Doom9's Forum are quite o-l-d: Thread 1 (http://forum.doom9.org/showthread.php?s=&threadid=23743), Thread 2 (http://forum.doom9.org/showthread.php?s=&threadid=24288), Thread 3 (http://forum.doom9.org/showthread.php?s=&threadid=23384), Thread 4 (http://forum.doom9.org/showthread.php?s=&threadid=23956), and Thread 5 (http://forum.doom9.org/showthread.php?s=&threadid=8317).

Regards.

[bond]

great stuff, thanks a lot! :)

i moved this to an own thread, cause i think it deserves this

[ukb008]

I make my XCDs from XviD .avi files by making a cuesheet and a .bin file via mode2cdmaker, and writing from the cuesheet with Nero.

Now, in this scheme of things, how do I use this error corection?

I am not a programmer, and, therefore, will wait for someone who has done it to tell us all. Perhaps Mr bond has examined Elic's file? We'd very much like to hear from him or Mr avih, another XCD expert.

Regards.

[jkwarras]

Nice tool. Thanks you.

[Koepi]

Well, I wrote an app ages ago which produces the "important backup sections" which were defined by the "standard" we made up. Unfortunately we never brought this to a proper release (though I think on Dext's XCD-site it is still downloadable.)

It's not doing error-correction like Elic's tool (which should eat up so much space that XCD is unnecessary?) but only keeps the vital parts of the files on the mode1-part of the CD so the file is partly reconstructable and playable (i.e. indexes for avi, ID3-tags for mp3s,...).

Thank you for the tool Elic!

Cheers
Koepi

[Elic]

Hi there!

Sorry I don't have much time for even starting to translate documentation into English.

@Koepi
Definitely the main goal of RR is to produce much less redundancy than usual ISO-9660 CDs have. I decided to protect whole amount of data instead of protect small pieces. As a result, I think 1% of redundant data (about 3600 sectors) is more than enough for protect XCD trustworthy.

I made my first XCDs (protected by very first version of RR, slooow like a turtle :) ) on September and October 2003, and for today only two of ten RR-XCDs have more than 100 sectors with bad EDC (of course I made copies but didn't thrown out originals mainly for historical reasons :) ). I think if I will test my XCDs regularly (say once per year) then I will always manage to restore disc and make its copy before it becomes completely unreadable.

Unfortunately RR-protected XCDs cannot be restored on-the-fly during playing, because of relatively huge amount of computations (about 200,000,000,000 16-bit-polynom multiplications in case of 1% redundancy) and because recovering program must read whole disc before start reconstructing lost sectors.

[Emp3r0r]

hey cool!

I read this and had to dig up this old thread:

http://forum.doom9.org/showthread.php?s=&threadid=27568

[Elic]

@ Emp3r0r
Thanks for a thread about XCD and PAR - I didn't know whether anyone tried to protect XCD by redundancy. So I'm not first in that direction. ;)

Of course, I tried parchive as protection tool before I started my project. But IMHO parchive is not suited for XCD because
(1) we need protect every sector, but parchive operates with blocks as small as 12 sectors (max ~32767 blocks per disc); therefore 1% redundancy is 300 blocks == 3600 sectors but only 300 single damaged sectors can be recovered;
(2) parchive needs a lot of extra space for storing CRC (or maybe MD5) of every block (when I request 30000 blocks and 1% redundancy (8 MB) - parchive eats almost 12 megs) although damaged sectors can be easily detected via EDC mismatch.

My program also operates by blocks but in different way: I use ~60000 blocks of 6 sectors each but every damaged sector can be recovered independently. (Yes, 1% redundancy is actually 6 rows * 600 columns, but all 600 cells damaged in one row still allows other rows to be damaged with guarantee to be recovered.)

Apropos, I plan to implement 32-bit Solomon Reed code as soon as I shall find the way to polynom multiplication without table functions.

[Emp3r0r]

sounds interesting, keep up the good work

[defunkt]

Do you plan to release your source? I'd like to look at implementing partial protection based on padding out an image to the maximum XCD size. Video encoding for specific media typically results in undersized files & while one might not be worried enough about error correction to sacrifice space for it, using otherwise empty space is a no-brainer. Partial redundant data could be written at the center of the disk to protect as much of the more fragile outside as space allows.

[Elic]

defunktDo you plan to release your source?I didn't make a decision yet. Maybe I shall release my program under GPL or as a library/plugin for Nero or so (as LGPL), or maybe it will be freeware or shareware - it depends on many things I don't know about (software patents (especially in Europe and Ukraine), how to sell a program, and so on).I'd like to look at implementing partial protection based on padding out an imageSorry, but my English is not good enough. Do you mean by word "partial" that you can protect only part of disc contents? Definitely yes - you can select files you want to protect, and you can protect not only M2F2 files but also files of ISO-part of the disc.
Or you mean "can I protect only vital parts of single file on my disc?" - No, I didn't work in that direction (although it may be interesting).redundant data could be written at the center of the diskI cannot agree, because usable data (movie) is more valuable than redundancy. :) So I prefer to place movie towards beginning of a disc and redundant data at the edge.

[defunkt]

Originally posted by Elic
Do you mean by word "partial" that you can protect only part of disc contents?
Yes, in fact if writing an AV file to XCD probably only part of one file.

Originally posted by Elic
Or you mean "can I protect only vital parts of single file on my disc?"
Having thought a little more on this, MP4Box has a -dref option which I believe is supposed to make an MP4 reference the streams in external files. Haven't played with it and don't know to what degree it's supported by players. But this would make a relatively small container file with all of the really valuable indexes which could be thoroughly error protected while the raw stream data can probably stand a few scratches & glitches.

[Elic]

defunktthis would make a relatively small container file with all of the really valuable indexes which could be thoroughly error protected while the raw stream data can probably stand a few scratches & glitches.Yes, I understood.
But my program don't protect any headers / indexes / other vital parts of file - it protect whole file (or set of files). The goal is to make reliable, scratch-proof and long-term-storing copy even at price of some discomfort in playing of scratched disc (because no on-the-fly repair is available - my program needs to read all protected files before it reconstruct damaged sectors).
In other hand, on-the-fly repairing of file from vital-parts-copy is impossible without some support from DShow-filter, so we win nothing in this case.
In fact, IMHO Solomon-Reed code is sufficient for repairing as many damaged sectors as we have sectors with redundancy so all we need is testing every disc regularly and repair it before damage degree will grow above limit caused by redundancy volume.

[Atamido]

I think what he meant was that if you have 795 MB available for an image, but your video file takes up 770MB, then automatically use 25MB for ECC. But if your file takes up only 650MB, automatically use 145MB for ECC.

[Elic]

@ Pamel & defunkt
Sorry, I haven't noticed that subtle point about automatic size determination. Thank you both for suggestion! This feature will be very usable, and its implementation must be quite simple, so most likely I shall include --autoselect-size into next version.

[Elic]

Hi there!

Sorry for long absence - I was busy by making next version of RR and attempting to translate its docs from Russian (my English is still weak and translation from native language is damn hard work :) ). My homepage is far from being ready yet, so I uploaded fresh version onto RapidShare again (http://rapidshare.de/files/1843782/rr-1.20.zip.html, 155047 bytes).

What's new in version 1.20:
- added support of Russian letters in filenames in both ANSI and OEM codepages under M$ Windows;
- DeXT (http://webs.ono.com/usr016/de_xt/)'s mode2cdmaker (http://dext.peque.org/xcd/mode2cdmaker-1.5.1-src.zip) has undergone some changes;
- Russian documentation is finally made (not included in package - please see thread named "RR - cd-R eRRoR coRRectoR" on ShareReactor (http://board.sharereactor.ru/showthread.php?p=582295)), translation into English already started (few first pieces is included in package, and I will post few others pieces onto this thread during week-end or so);
- improved processing of directory tree during protecting;
- few errors are fixed.

Thanks to all for your answers. I hope that my program is useful to you.

Unfortunately I not decided yet how to implement idea offered by defunkt and Pamel about autodetect redundant data size, but I plan to write a calculator or some kind a text-only frontend to rrenc that allows to simplify using of my program.

And some more about XCD.
Few days ago I finished reading old threads about XCD (thanks ukb008 for URLs), and I realized that XCD CD-level specifications exist I never heard about before. Then I spend few hours searching across Internet for another mentioning about these specs... nothing found. Does this mean that nobody developed further in this (XCD CD-level) direction? What a nuisance!

Also, I searched for Koepi's app "which produces the important backup sections" and found nothing again :(

But then, I found another GUI frontend for mode2cdmaker called cdrtfe (http://home.arcor.de/kerberos002/cdrtfe/index_en.html). Unfortunately cdrtfe is written in Deutsch and in Delphi and is therefore as twice as incomprehensible for me :) Well, at least, I'm not alone in enthusiasm for XCD :)

Thanks to all!

Best regards,
Elic

EDIT: 2005/05/19 I was wrong about cdrtfe (http://home.arcor.de/kerberos002/cdrtfe/index_en.html). I didn't noticed that it has file with messages translated into English and Russian. I played with it couple of hours, and IMHO cdrtfe is the most comfortable frontend to mode2cdmaker I ever seen. First release was about two years ago, and seems strange that nobody mentioned about this program and kerberos002, its author, on doom9 forum.

[Koepi]

Hi Elic,

your post motivated me to search for myself. It's fun to read those _really old_ threads again.

Then I found that the XCDBackupCreator isn't lost, avih was so kind to upload it to sourceforge's CVS:
http://cvs.sourceforge.net/viewcvs.py/xcd/xcd/backup-creator/

There you have what I wrote 3 years ago...

Cheers
Koepi

EDIT: I took the time to d/l the files, remove a bug(!)[well - err, I had a variable declared too late) and zip up the result with the compiled .exe and the soruces combined.

[Koepi]

Here is the file:

[Elic]

@ Koepi
Very thanks for XCDBackupCreator and XCD-specs from CVS (relatively fresh :) - dated two month later than last specs published on http://xcd.sourceforge.net).
Apropos, IMHO it would be good to implant RR into "XCD CD level". But I don't know yet how to do that without breaking specs, or may I change the specs in order to implant RR easily into XCD as option?

[Koepi]

Hi Elic,

simply changing the specs is a very bad idea since XCD even made it into hardware playback devices and it would break compatibility then.

An addendum would be possible though as long as it doesn't break backwards compatibility. Like i.e. extending the XCH/Backup system accordingly, we kept that very simple.

Let's see if de_xt is still active, we could enter into a new discussion of that issue then.

Cheers
Koepi

[Elic]

@ Koepi
> XCD even made it into hardware playback devices
Wow! I didn't even heard about it! Where can I read more about them?

> extending the XCH/Backup system accordingly, we kept that very simple.
Thanks, I shall get a grasp of specs more closely :)

> Let's see if de_xt is still active,
His last message on forum was on December 2004. :( I wrote a PM to him on 5 May, and still no reply :(

[Koepi]

Originally posted by Elic
@ Koepi
> XCD even made it into hardware playback devices
Wow! I didn't even heard about it! Where can I read more about them?


A fast shot (be careful, it's a german advertisment site): http://www.fotos.de/shop/index.php?file=amazon&mode=details&number=B0002IE2L8&catalog=electronics&language=de

To my knowledge, every mpeg4-enabled player supports XCD too.

Back to de_xt, give him some time. Sometimes it can take 4 weeks until somebody realises there's something in his mailbox ;)

I'll try to contact avih as well, he is also involved into XCD.

Cheers
Koepi

[Elic]

New version (1.21) of RR is released as update package (rr-1.21.update.zip (http://www.smart.vn.ua/forum/attachment.php?attachmentid=13110), 93011 bytes, contains only rrenc.exe, rrdec.exe and whatsnew.*) which is to be combined with previous version (http://rapidshare.de/files/1843782/rr-1.20.zip.html).

No new features, just fixes:
- bug in polynom math library caused crash on CPUs without SSE (due to inattentive read help on Intel compiler);
- bug in --process-dir switch handler.

@ Koepi
I searched more info about Gericom Bellagio AD2020 DVD-Player but didn't realise if it uses CD-level XCD (CLX) or file-level XCD (FLX), i.e. seems to be it can playback just mode2-form2 files regardless of disc structure (or maybe manufacturer have detailed specs closed). In my city, I didn't find DVD-players in sale which would claim to be XCD compatible. :( Although, I don't pretend to buy standalone DVD player, I'm totally satisfied by my PC :)

DeXT is still in silence :(

[Elic]

Hi there!

Nowadays, I got the site - http://elicit.vn.ua and uploaded all of my stuff there. Site is under construction, sorry about uncomfortabilities.

The new version (RR 1.24) has many fixes of minor bugs and nearly invisible :) optimization. It is also splitted onto two packages: http://elicit.vn.ua/files/rr/rr-1.24.main.zip - main executables and minimalistic set of docs, and http://elicit.vn.ua/files/rr/rr-1.24.extra.zip - example of use and modified mode2cdmaker with sources.

All of previous versions are also available here (http://elicit.vn.ua/files/outdated/). Some hole in version numbering is caused by extensive refactoring and thus couple of semi-raw builds which remain unpublished.

[bond]

nice stuff! :)

[Elic]

Version 1.25 (2006/02/13)
Lightweight (rr-1.25.update (http://elicit.vn.ua/files/rr/rr-1.25.update.zip)) package (main files only); designed to use together with rr-1.24.extra (http://elicit.vn.ua/files/rr/rr-1.24.extra.zip) package.

Changelog:Common:
- few small bugs fixed.

rrenc:
- support for absolute paths added (in --process-dir mode only)
in source filelist (when --source-dir is empty) and in destination
files (when --target-dir is empty);
- new command-line option added: --multi-track
- new RR-list option added: -u
- output of command-line into RR-list added;
- built-in help improved and slightly reordered, info about errorlevels
added.

rrdec:
- new RR-list option added: -u
- built-in help improved and slightly reordered.The main current development vector is to simplify integration with mode2cdmaker's GUIs.

[Elic]

Version 1.26 (2006/02/15)
Lightweight (rr-1.26.update (http://elicit.vn.ua/files/rr/rr-1.26.update.zip)) package (main files only); designed to use
together with rr-1.24.extra (http://elicit.vn.ua/files/rr/rr-1.24.extra.zip) package.

Changelog:Common:
- fixed few bugs in support of absolute paths.

rrenc:
- new command-line option added: --keep-old-ext (as m2cdm -extname).

rrdec:
- fixed serious bug in 1.25 which might cause mixing of directories;
- support for XDC-discs made with -extname option.

[Elic]

Version 1.27 (2006/02/20)
Lightweight (rr-1.27.update (http://elicit.vn.ua/files/rr/rr-1.27.update.zip)) package (main files only); designed to use together with rr-1.24.extra (http://elicit.vn.ua/files/rr/rr-1.24.extra.zip) package.

Changelog:Common:
- max linesize of .rrt file increased up to ~1080 bytes (for long
comments, especially rrenc's command-line copy);
- fixed bug in comment processing module (crash if buffer overflows);
- new RR-list option added: -e <ext> - custom extension of XCD files;
- GUI front-end integration is slightly improved.

Edit: there was an erroneous link. Corrected.

[bond]

beaware you still link to version 1.26!

[Elic]

bond
> beaware
Oops... Thanks for warning. Maybe I was slightly sleepy :o :)

[Elic]

Version 1.28 (http://elicit.vn.ua/files/rr/rr-1.28.update.zip) (2006/02/24) (for use together with rr-1.24.extra (http://elicit.vn.ua/files/rr/rr-1.24.extra.zip), as usual :) ). Changelog:Common:
- fixed bug in command-line parser.And a novelty: kerberos002 (http://home.arcor.de/kerberos002/) is about to release cdrtfe (http://home.arcor.de/kerberos002/cdrtfe/index_en.html) (GUI front-end for cdrtools) with built-in support of rrenc. He said that rrenc-capable version can be released in the next one or two weeks.

[Elic]

Version 1.29 (http://elicit.vn.ua/files/rr/rr-1.29.update.zip) (2006/02/27) is an update to rr-1.24.extra (http://elicit.vn.ua/files/rr/rr-1.24.extra.zip), as usual. Changelog:Common:
- fixed bug in m2cdm paramfile composer;
- GUI front-end integration is improved (status line output).And what's important: kerberos002 (http://home.arcor.de/kerberos002/) have released version 1.2pre1 of his GUI front-end to mode2cdmaker & cdrtools called cdrtfe (http://home.arcor.de/kerberos002/cdrtfe/index_en.html) which allows RR-protection of XCD discs. So we already have a nice front-end for rrenc :) (in both binary (http://home.arcor.de/kerberos002/cdrtfe/download_en.html) and sources (http://home.arcor.de/kerberos002/cdrtfe/source_en.html)). :thanks:

[Tima]

It's very difficult to find RR packages today (official site is dead and most of other sources are also invalid), so I decided to upload the latest versions here.

[Brazil2]

It's very difficult to find RR packages today (official site is dead and most of other sources are also invalid), so I decided to upload the latest versions here.
Looks like a nice find, thanks for that :)
If you could add rr-1.24.main.zip as well so the package is complete ;)

[Tima]

If you could add rr-1.24.main.zip as well so the package is complete ;)

AFAIK, rr-1.29.update.zip is a substitution for rr-1.24.main.zip.

See also last post of Elic in this thread.

[Midzuki]

It's very difficult to find RR packages today (official site is dead and most of other sources are also invalid), so I decided to upload the latest versions here.

Thanks!

[Elic]

RR revived as open source (GNU GPL v3): cd-rr (http://sourceforge.net/projects/cd-rr/).
Version 1.30 (2012/11/02) (http://sourceforge.net/projects/cd-rr/files/cd-rr-1.30/) - changelog:Common:
- example of use (window$ only).

rrenc, rrdec:
- port to Linux added.

m2cdm:
- preserve original timestamps of files placed on disc;
- some compiler warnings fixed;
- use compilation date instead of predefined one.Win32 binaries (http://sourceforge.net/projects/cd-rr/files/cd-rr-1.30/cd-rr-1.30.win32.7z/download), Linux x86 binaries (http://sourceforge.net/projects/cd-rr/files/cd-rr-1.30/cd-rr-1.30.linux.7z/download) (made by yas_1975), GPLed source code (http://sourceforge.net/projects/cd-rr/files/cd-rr-1.30/cd-rr-1.30.src.7z/download).