Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > New and alternative a/v containers

Reply
 
Thread Tools Search this Thread Display Modes
Old 14th April 2005, 21:41   #1  |  Link
Elic
Registered User
 
Elic's Avatar
 
Join Date: Apr 2005
Location: Ukraine, Vinnitsa
Posts: 110
Error correction for XCD is made

Hi there!

I wrote error correction program for XCD. Strictly speaking, there are two programs in one package called RR (as "CD-R eRRoR coRRectoR"):
rrenc - reads all of files which are to be written into XCD, computes an array of redundant data (usual 16-bit Solomon-Reed code) and prepares paramfile for mode2cdmaker;
rrdec - reads all of files (including redundant data) from XCD and writes them to your hard disk, computes the contents of damaged sectors and insert them into proper places.

There is a thread about RR program package on http://forum.sharereactor.ru/showthread.php?t=59777. The thread is in Russian (and my native language is Ukrainian), but I made my programs shipped with English command-line interface. You can download entire package from http://rapidshare.de/files/1260596/rr119.zip.html but this is not persistent link; please check alternate links located on first post of ShareReactor's thread mentioned above.

Sorry, but my English is weak and this is my first message on this forum. I plan to write docs for RR in Russian and English nearly simultaneously (Russian first of course) and I will appreciate if Russian-speaking visitors vill verify my translations.

2006/02/01
The package is resides of its own site now. Last version, 1.24, is splitted into two packages - executables and extra files. All previous publicly available versions are here.

2006/02/16
Version 1.26 is released. Full description is in this post.

2006/02/20
Version 1.27 is released; description.

2006/02/24
Version 1.28, and there will be front-end soon (details are here).

2006/02/28
Version 1.29, front-end, details

2012/11/02
Version 1.30 : win32 binaries, GPLed source code, details

Last edited by Elic; 2nd November 2012 at 14:46.
Elic is offline   Reply With Quote
Old 15th April 2005, 03:14   #2  |  Link
ukb008
Registered User
 
Join Date: Feb 2005
Location: South of the Equator
Posts: 410
Bravo, Elic.

Your English is quite OK. Welcome to The Forum and thanks for the Pie. We're all waiting for the guide, if and when you have the time.

What do the XCD developers say about this? The related posts in the Doom9's Forum are quite o-l-d: Thread 1, Thread 2, Thread 3, Thread 4, and Thread 5.

Regards.
ukb008 is offline   Reply With Quote
Old 15th April 2005, 16:39   #3  |  Link
bond
Registered User
 
Join Date: Nov 2001
Posts: 9,779
great stuff, thanks a lot!

i moved this to an own thread, cause i think it deserves this
__________________
Between the weak and the strong one it is the freedom which oppresses and the law that liberates (Jean Jacques Rousseau)
I know, that I know nothing (Socrates)

MPEG-4 ASP FAQ | AVC/H.264 FAQ | AAC FAQ | MP4 FAQ | MP4Menu stores DVD Menus in MP4 (guide)
Ogg Theora | Ogg Vorbis
use WM9 today and get Micro$oft controlling the A/V market tomorrow for free
bond is offline   Reply With Quote
Old 26th April 2005, 02:44   #4  |  Link
ukb008
Registered User
 
Join Date: Feb 2005
Location: South of the Equator
Posts: 410
How do we use it?

I make my XCDs from XviD .avi files by making a cuesheet and a .bin file via mode2cdmaker, and writing from the cuesheet with Nero.

Now, in this scheme of things, how do I use this error corection?

I am not a programmer, and, therefore, will wait for someone who has done it to tell us all. Perhaps Mr bond has examined Elic's file? We'd very much like to hear from him or Mr avih, another XCD expert.

Regards.
ukb008 is offline   Reply With Quote
Old 28th April 2005, 09:25   #5  |  Link
jkwarras
Registered User
 
jkwarras's Avatar
 
Join Date: Feb 2003
Location: Zaragoza
Posts: 405
Nice tool. Thanks you.
__________________
Iván
Blog: http://ivancastell.org | Myspace: http://myspace.com/ivancastell | Last.fm: http://lastfm.es/user/jkwarras/
jkwarras is offline   Reply With Quote
Old 28th April 2005, 10:36   #6  |  Link
Koepi
Moderator
 
Koepi's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 4,455
Well, I wrote an app ages ago which produces the "important backup sections" which were defined by the "standard" we made up. Unfortunately we never brought this to a proper release (though I think on Dext's XCD-site it is still downloadable.)

It's not doing error-correction like Elic's tool (which should eat up so much space that XCD is unnecessary?) but only keeps the vital parts of the files on the mode1-part of the CD so the file is partly reconstructable and playable (i.e. indexes for avi, ID3-tags for mp3s,...).

Thank you for the tool Elic!

Cheers
Koepi
Koepi is offline   Reply With Quote
Old 30th April 2005, 22:55   #7  |  Link
Elic
Registered User
 
Elic's Avatar
 
Join Date: Apr 2005
Location: Ukraine, Vinnitsa
Posts: 110
Hi there!

Sorry I don't have much time for even starting to translate documentation into English.

@Koepi
Definitely the main goal of RR is to produce much less redundancy than usual ISO-9660 CDs have. I decided to protect whole amount of data instead of protect small pieces. As a result, I think 1% of redundant data (about 3600 sectors) is more than enough for protect XCD trustworthy.

I made my first XCDs (protected by very first version of RR, slooow like a turtle ) on September and October 2003, and for today only two of ten RR-XCDs have more than 100 sectors with bad EDC (of course I made copies but didn't thrown out originals mainly for historical reasons ). I think if I will test my XCDs regularly (say once per year) then I will always manage to restore disc and make its copy before it becomes completely unreadable.

Unfortunately RR-protected XCDs cannot be restored on-the-fly during playing, because of relatively huge amount of computations (about 200,000,000,000 16-bit-polynom multiplications in case of 1% redundancy) and because recovering program must read whole disc before start reconstructing lost sectors.
Elic is offline   Reply With Quote
Old 4th May 2005, 04:53   #8  |  Link
Emp3r0r
Registered User
 
Emp3r0r's Avatar
 
Join Date: Oct 2001
Location: Alabama, USA
Posts: 769
hey cool!

I read this and had to dig up this old thread:

http://forum.doom9.org/showthread.php?s=&threadid=27568
__________________
ChapterGrabber - add names to your chapters | AtomSite - open source AtomPub server
Emp3r0r is offline   Reply With Quote
Old 4th May 2005, 22:15   #9  |  Link
Elic
Registered User
 
Elic's Avatar
 
Join Date: Apr 2005
Location: Ukraine, Vinnitsa
Posts: 110
@ Emp3r0r
Thanks for a thread about XCD and PAR - I didn't know whether anyone tried to protect XCD by redundancy. So I'm not first in that direction.

Of course, I tried parchive as protection tool before I started my project. But IMHO parchive is not suited for XCD because
(1) we need protect every sector, but parchive operates with blocks as small as 12 sectors (max ~32767 blocks per disc); therefore 1% redundancy is 300 blocks == 3600 sectors but only 300 single damaged sectors can be recovered;
(2) parchive needs a lot of extra space for storing CRC (or maybe MD5) of every block (when I request 30000 blocks and 1% redundancy (8 MB) - parchive eats almost 12 megs) although damaged sectors can be easily detected via EDC mismatch.

My program also operates by blocks but in different way: I use ~60000 blocks of 6 sectors each but every damaged sector can be recovered independently. (Yes, 1% redundancy is actually 6 rows * 600 columns, but all 600 cells damaged in one row still allows other rows to be damaged with guarantee to be recovered.)

Apropos, I plan to implement 32-bit Solomon Reed code as soon as I shall find the way to polynom multiplication without table functions.
Elic is offline   Reply With Quote
Old 5th May 2005, 04:15   #10  |  Link
Emp3r0r
Registered User
 
Emp3r0r's Avatar
 
Join Date: Oct 2001
Location: Alabama, USA
Posts: 769
sounds interesting, keep up the good work
__________________
ChapterGrabber - add names to your chapters | AtomSite - open source AtomPub server
Emp3r0r is offline   Reply With Quote
Old 5th May 2005, 22:15   #11  |  Link
defunkt
defkunt
 
defunkt's Avatar
 
Join Date: Dec 2004
Location: New Zealand
Posts: 58
Do you plan to release your source? I'd like to look at implementing partial protection based on padding out an image to the maximum XCD size. Video encoding for specific media typically results in undersized files & while one might not be worried enough about error correction to sacrifice space for it, using otherwise empty space is a no-brainer. Partial redundant data could be written at the center of the disk to protect as much of the more fragile outside as space allows.
defunkt is offline   Reply With Quote
Old 6th May 2005, 21:42   #12  |  Link
Elic
Registered User
 
Elic's Avatar
 
Join Date: Apr 2005
Location: Ukraine, Vinnitsa
Posts: 110
defunkt
Quote:
Do you plan to release your source?
I didn't make a decision yet. Maybe I shall release my program under GPL or as a library/plugin for Nero or so (as LGPL), or maybe it will be freeware or shareware - it depends on many things I don't know about (software patents (especially in Europe and Ukraine), how to sell a program, and so on).
Quote:
I'd like to look at implementing partial protection based on padding out an image
Sorry, but my English is not good enough. Do you mean by word "partial" that you can protect only part of disc contents? Definitely yes - you can select files you want to protect, and you can protect not only M2F2 files but also files of ISO-part of the disc.
Or you mean "can I protect only vital parts of single file on my disc?" - No, I didn't work in that direction (although it may be interesting).
Quote:
redundant data could be written at the center of the disk
I cannot agree, because usable data (movie) is more valuable than redundancy. So I prefer to place movie towards beginning of a disc and redundant data at the edge.
__________________
-= Elicit Gingers =-
Elic is offline   Reply With Quote
Old 6th May 2005, 22:41   #13  |  Link
defunkt
defkunt
 
defunkt's Avatar
 
Join Date: Dec 2004
Location: New Zealand
Posts: 58
Quote:
Originally posted by Elic
Do you mean by word "partial" that you can protect only part of disc contents?
Yes, in fact if writing an AV file to XCD probably only part of one file.

Quote:
Originally posted by Elic
Or you mean "can I protect only vital parts of single file on my disc?"
Having thought a little more on this, MP4Box has a -dref option which I believe is supposed to make an MP4 reference the streams in external files. Haven't played with it and don't know to what degree it's supported by players. But this would make a relatively small container file with all of the really valuable indexes which could be thoroughly error protected while the raw stream data can probably stand a few scratches & glitches.
defunkt is offline   Reply With Quote
Old 7th May 2005, 17:47   #14  |  Link
Elic
Registered User
 
Elic's Avatar
 
Join Date: Apr 2005
Location: Ukraine, Vinnitsa
Posts: 110
defunkt
Quote:
this would make a relatively small container file with all of the really valuable indexes which could be thoroughly error protected while the raw stream data can probably stand a few scratches & glitches.
Yes, I understood.
But my program don't protect any headers / indexes / other vital parts of file - it protect whole file (or set of files). The goal is to make reliable, scratch-proof and long-term-storing copy even at price of some discomfort in playing of scratched disc (because no on-the-fly repair is available - my program needs to read all protected files before it reconstruct damaged sectors).
In other hand, on-the-fly repairing of file from vital-parts-copy is impossible without some support from DShow-filter, so we win nothing in this case.
In fact, IMHO Solomon-Reed code is sufficient for repairing as many damaged sectors as we have sectors with redundancy so all we need is testing every disc regularly and repair it before damage degree will grow above limit caused by redundancy volume.
__________________
-= Elicit Gingers =-
Elic is offline   Reply With Quote
Old 8th May 2005, 06:37   #15  |  Link
Atamido
Señor Member
 
Atamido's Avatar
 
Join Date: May 2002
Location: Austin, Texas
Posts: 915
I think what he meant was that if you have 795 MB available for an image, but your video file takes up 770MB, then automatically use 25MB for ECC. But if your file takes up only 650MB, automatically use 145MB for ECC.
Atamido is offline   Reply With Quote
Old 8th May 2005, 08:35   #16  |  Link
Elic
Registered User
 
Elic's Avatar
 
Join Date: Apr 2005
Location: Ukraine, Vinnitsa
Posts: 110
@ Pamel & defunkt
Sorry, I haven't noticed that subtle point about automatic size determination. Thank you both for suggestion! This feature will be very usable, and its implementation must be quite simple, so most likely I shall include --autoselect-size into next version.
__________________
-= Elicit Gingers =-
Elic is offline   Reply With Quote
Old 18th May 2005, 23:01   #17  |  Link
Elic
Registered User
 
Elic's Avatar
 
Join Date: Apr 2005
Location: Ukraine, Vinnitsa
Posts: 110
New version and first portion of dosc

Hi there!

Sorry for long absence - I was busy by making next version of RR and attempting to translate its docs from Russian (my English is still weak and translation from native language is damn hard work ). My homepage is far from being ready yet, so I uploaded fresh version onto RapidShare again (http://rapidshare.de/files/1843782/rr-1.20.zip.html, 155047 bytes).

What's new in version 1.20:
- added support of Russian letters in filenames in both ANSI and OEM codepages under M$ Windows;
- DeXT's mode2cdmaker has undergone some changes;
- Russian documentation is finally made (not included in package - please see thread named "RR - cd-R eRRoR coRRectoR" on ShareReactor), translation into English already started (few first pieces is included in package, and I will post few others pieces onto this thread during week-end or so);
- improved processing of directory tree during protecting;
- few errors are fixed.

Thanks to all for your answers. I hope that my program is useful to you.

Unfortunately I not decided yet how to implement idea offered by defunkt and Pamel about autodetect redundant data size, but I plan to write a calculator or some kind a text-only frontend to rrenc that allows to simplify using of my program.

And some more about XCD.
Few days ago I finished reading old threads about XCD (thanks ukb008 for URLs), and I realized that XCD CD-level specifications exist I never heard about before. Then I spend few hours searching across Internet for another mentioning about these specs... nothing found. Does this mean that nobody developed further in this (XCD CD-level) direction? What a nuisance!

Also, I searched for Koepi's app "which produces the important backup sections" and found nothing again

But then, I found another GUI frontend for mode2cdmaker called cdrtfe. Unfortunately cdrtfe is written in Deutsch and in Delphi and is therefore as twice as incomprehensible for me Well, at least, I'm not alone in enthusiasm for XCD

Thanks to all!

Best regards,
Elic

EDIT: 2005/05/19 I was wrong about cdrtfe. I didn't noticed that it has file with messages translated into English and Russian. I played with it couple of hours, and IMHO cdrtfe is the most comfortable frontend to mode2cdmaker I ever seen. First release was about two years ago, and seems strange that nobody mentioned about this program and kerberos002, its author, on doom9 forum.
__________________
-= Elicit Gingers =-

Last edited by Elic; 19th May 2005 at 21:16.
Elic is offline   Reply With Quote
Old 19th May 2005, 09:55   #18  |  Link
Koepi
Moderator
 
Koepi's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 4,455
Hi Elic,

your post motivated me to search for myself. It's fun to read those _really old_ threads again.

Then I found that the XCDBackupCreator isn't lost, avih was so kind to upload it to sourceforge's CVS:
http://cvs.sourceforge.net/viewcvs.p...ackup-creator/

There you have what I wrote 3 years ago...

Cheers
Koepi

EDIT: I took the time to d/l the files, remove a bug(!)[well - err, I had a variable declared too late) and zip up the result with the compiled .exe and the soruces combined.

Last edited by Koepi; 19th May 2005 at 10:20.
Koepi is offline   Reply With Quote
Old 19th May 2005, 10:21   #19  |  Link
Koepi
Moderator
 
Koepi's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 4,455
Here is the file:
Attached Files
File Type: zip xcdbackupcreator.zip (19.7 KB, 223 views)
Koepi is offline   Reply With Quote
Old 19th May 2005, 21:50   #20  |  Link
Elic
Registered User
 
Elic's Avatar
 
Join Date: Apr 2005
Location: Ukraine, Vinnitsa
Posts: 110
@ Koepi
Very thanks for XCDBackupCreator and XCD-specs from CVS (relatively fresh - dated two month later than last specs published on http://xcd.sourceforge.net).
Apropos, IMHO it would be good to implant RR into "XCD CD level". But I don't know yet how to do that without breaking specs, or may I change the specs in order to implant RR easily into XCD as option?
__________________
-= Elicit Gingers =-
Elic is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 13:19.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2017, vBulletin Solutions Inc.