self repairing disc

[Emp3r0r]

Has anybody heard of this new PAR thing they are doing on the newsgroups? I'm no expert but I believe any PAR file can take the place of any RAR file by using some sort of parity algorithm. Could this concept be applied to maybe XCD by use of maybe a couple megs to store parity information and then you could run a little proggy to repair the source disk... hence, a self repairing disc. I'd gladly set aside a few megs for this feature. Maybe there is already a much better way of doing such a thing, but in any case, I think this could help in overcoming the scratch.

[Koepi]

PAR is in NO way new, it's really really old in fact.

Those parity information (it's wrong to call it parity since with parity information it's impossible to _correct_ data).

The overhead is immense, I once got hold of something secured this way and the PAR stuff was ~1/4 of the size of the archive.

so - why using PAR? On a 700MB CDR we gain 100MB from mode2form2. BUT we have 150MB recovery info and 650MB real file left....


So your proposal is to make a 700MB CDR a 650MB CDR.

Sorry, I don't support that ;)

Regards,
Koepi

[Latexxx]

Correct me if I'm wrong. Ogg and upcoming mcf have error correctioning in fileformat.

[Koepi]

No, they don't have error correction - but they are error tolerant and just skip broken parts.

Regards,
Koepi

[Emp3r0r]

I didn't propose using 150 megs megs worth of parity information, just a small amount. Is this not possible? For example, you have a lump of data, say 725 megs big... you decide to add 25 megs of error recovery. That would make 29 25meg seperate sections of data (30 overall). so if maybe you get a scratch in the middle that corrupts just one 25 meg section of data, you could run a little proggy that would use the error correction data and spit out your original 725 meg file.

@Koepi: maybe I don't know what I'm talking about (which is highly likely) but I don't see why the concept couldn't be applied for sensitive backups. BTW, i know the concept of parity checking is really really old, but the way the release groups are using it for newsgroups is new afaik, well at least in the last 5 or 6 years new. I haven't downloaded anything from newsgroups since Windows(96) Nashville.

[pandv]

XCD has error detection at the sector level?

If the response is yes, I think it's possible to implement a PAR like protection scheme.

If one sector is 3 Kbytes (rounding up) with 3000 Kbytes in PAR information you can recover a disc with 1000 sectors damageds.

If there aren't error detection forgot it.


pandv.

[Koepi]

Head over to xcd.sourceforge.net and read the specs.
Then you know what is done - and waht is not done.

Btw., there won't be anyting like PAR. But if you read the specs you'll know why.

[pandv]

Ok, I understand a protection scheme as Par is not in the specs (the protection level is your backup aplication).

But i am thinking about implementing this scheme maybe at a high level. When I burn a CD I have always free space. Maybe only 3 Mb, maybe more.

What about to create a file with the free space, and use it to store recovery information? If your CD is damaged you can try (with a special aplication) to recreate a correct ISO for it.

Can this be useful?

It's only a sugestion for a idle programmer (exists?). I don't have the knowledge nor the time.

pandv.

[Milkman Dan]

Guys.
CRC/what you're calling 'parity' doesn't recover information.
It detects errors, rather than recovering them.

Even if it did recover information, it would take ALOT more space than 3 MB. Think more like RAID 5. Minimum # of discs for RAID 5 is 3. You lose the space of a third disc because it holds the (true)parity of the other 2. For every block written to two discs, an XOR operation is performed, and the result stored in round-robin fashion on the remaining disc.

CRC is already built into MCF. I don't know what Ogg holds in this arena. But neither does error RECOVERABILITY as far as I know.

[Koepi]

Dan,

that's exactly what I'm talking about. I think my first reply was VERY clear?!?!

Regards,
Koepi

[pandv]

I am sure my english is not good enough to make my point.

@Koepi

Your information is very clear for me, and I trust it.

@Dan

CRC doesn't recover information.
OK. The named PAR doesn't uses CRC for correction. For the detection it uses MD5 checksums. Despite the name, it uses Red Solomon Code for correction.

Spec at: http://sourceforge.net/docman/display_doc.php?docid=7273&group_id=30568

Of course, this spec is not valid to protect a CD, but the idea is here.

About RAID.
With a 50% overhead you get protection for the 50% of the data (one complete disk).
With 3 Mb you get protection for 3Mb of the data (maybe a scratch in the CD surface).
There are no miracle here.

I'm not proposing to change XCD to implement a PAR like protection scheme, I'm only saying a similar scheme maybe can be useful.

There are computacional dificults (there are thousands of sectors, not a few rar files to protect), and specification dificults (they need to be implemented at ISO level if the internal CRC protection of the CD is used).

So, I am only arguing about the usefulness of this thing, and not proposing any concrete action.

pandv

[avih]

@pandv
you are correct that error recovery facility IS required for some parts of the media file, while for others, error detection is enough.

there's also the known trade-off between the level of protection to the size of the overhead data.

now, the whole point of xcd is that most media files DON'T need error correction, since detection is enough. (as in audio-cd or svcd). if there are errors, there are temporary artefacts during playback, and after that the playback resumes normally. so the aim was to throw-away the overhead involved in error CORRECTION, and leave only the error detection. that why the media file is in mode 2 form 2 instead of mode 2 form 1 (which is the same as normal mode 1 for that matter)

since mode 2 form 2 (which is the storage for the main media file) DOES have error detection, all we need is error CORRECTION for the really important parts. and that is implemented in mode 2 form 1 file. again, read the spec.

now after you've read my explenation and the spec, can you define exactly why do we need PAR? do u think it should REPLACE the header file correction method? do u think it should be applied for the media file ON-TOP of the error detection? or maybe instead of the error detection?

pls explain youre idea again, and with as much details as u can, with references to the current xcd spec.

thx
avi

[spyder]

Why are you people so worried about error correction over the entire disc? No one ever makes such a big fuss over VCD/SVCD not having error correction. BTW something of this nature is already planned for MCF. Tronic suggested separate ECC files to correct the data lost in the original. I might also add ECC support for my MCF server. But you can hear more about my server later when it's under development.

[Milkman Dan]

Originally posted by Koepi
Dan,

that's exactly what I'm talking about. I think my first reply was VERY clear?!?!

Regards,
Koepi

Yeah, I know. They weren't getting it though. Thought I'd try a different tack.

[Milkman Dan]

About RAID.
With a 50% overhead you get protection for the 50% of the data (one complete disk).

Not quite. When a disk in a RAID 5 fails, the disk is rebuilt from the parity present on the other two(or more) disks. The array (and the data) is still safe with the failure of one(or more) disks.

Thus, the integrity of the entire array is safe. This is not so for a CD protected this way. You give up much more than 3MB. Which brings me to your comment about protecting just 3MB that might be destroyed in a scratch.

Well, how do you know which 3MB to protect? You could spread the bits out really thin, but then you lose the effectiveness of the protection.

I respect you bringing up the utility of such a system, but it doesn't and can't feasibly work. It takes too much space on the limited medium of a CD, and hard disks already have protections in place, such as RAID 1, 10, and 5.

[pandv]

Thank's for your replies.

I not propose to change XCD specs.

I think this need more thinking. I investigate on the subject when I get to hollidays.

But,

@Milkman Dan
Par protects any damaged part. Let's explain the common use. You have a 40 rar set, but you get only 39 rars and a par. With this par you can recover the missing rar (any missing rar, with any par file). You only need a quantity of par files equal to the missing rar files.

pandv

[DaveEL]

a small amount of data cannot be enough you cant reconstruct the data

3mb of data would not protect any 3mb of data just 3mb of data in a very specific pattern (in the case of par 3mb of data within a single 3mb file) in order to get an real protection you need a lot more

using your example of par i decided to give it a go

To protect 232mb of video stored in 7 files from failure of at most 3 files i need to add an extra 145mb of data.

Even to protect 1 file i needed ~50mb (size of the largest file(ish)) of course i could reduce this by making the videos smaller and using playlists to turn them into one playable file but if i split them into 3mb files its far more likely several would have errors and so i would need to generate lots of par files to be safe and so it would take up more then 3mb to protect them.

DaveEL

[frodoontop]

Now Matroska with error correction is not in sight yet and there is a new, superior version of par, I thought this could be potentially handy for backup. So I'm bringing this thread alive again. :)

On hydrogenaudio.org they already had a discussion for backing up music, however this wasn't applied with XCD. You can find that discussion here (http://www.hydrogenaudio.org/index.php?act=ST&f=1&t=9236&hl=par2&) . You can also find more information about par2 there.

Would it be possible to store this par2 file on Mode2-filemode? And use dat2file to get both files back from CD? I know this will not be as effective as Matroska's coming intelligent error correction, but on the other hand it won't hurt using par eather, I suppose.

I'm not planning to use this on every CD, but just for those that have filesizes like 705-780 MB.

[temporance]

Well, I think it's quite a good idea. As soon as an XCD disk develops an error, you run a special application that uses the reundant block to repair the dodgy data. Then you burn a fresh disk with the original data, error-free.

With 3MB, you can repair any single contiguous block of 3MB on the disk. If there are errors in more than one place on the disk, it's less likely that it's repairable.

The reason that this hasn't been done before on CDs/DVDs is that it would take a very long time to recover the lost data. We're used to waiting a long time for things (ripping, encoding, burning, etc.) so this is more acceptable for us.

[Atamido]

@DaveEL: You numbers are not quite right. First you must know that there are two versions of PAR, 1.0 and 2.0.

Wit 1.0 you can create parity files with several blocks of data, but it requires that all of the parity files be at least as large as the largest data block. So, if you have 10 file ranging between 2MB and 50MB, your parity files would be 50MB. And you would need one 50MB parity file, in addition to the rest of the set, to repair/rebuild a single file. So, 50MB to rebuild a 2MB file. This method is really only good where the data blocks are all the same size. IE, you have 30 files that are 10MB each. You have 5x10MB parity files. You can use these to replace up to 10 corrupt files.

With 2.0, you pick a set parity size, and only need at least as much data as you need to rebuild. So, this is what you would use with a single large media file as it could be used to replace part of the file.

In your example of 232MB that you want to protect (doesn't matter if they are in a single file, or multiple files). You just create a 200 100KB files for parity which would take up 20MB. That would allow you to replace up to 200 non-contiguous damaged segments in the file(s). Adn/or you could replace a single 20MB missing data chunk.

The idea behind 2.0 is what we were planning for Matroska. You have several data chunks of random sizes. You have a user-selected parity block size, and number for each cluster. Then you should be able to rebuild clusters on the fly...

[DaveEL]

Originally posted by Pamel
@DaveEL: You numbers are not quite right. First you must know that there are two versions of PAR, 1.0 and 2.0.


Check the post date its before par 2.0 this is a very old thread.

DaveEL

[avih]

sounds like PAR2 could be usefull as an extra protection. if one wants, one can put the PAR2 file as a mode2form1 (protected) file on the cd as well. however, i don't think it will be usable in real time playback, since it sounds to me as if it would take quote a long time to correct an error.

however, if one copiees the movie file to the HD (dat2file or xcd2file when the backup protection is incorporated), then the PAR2 file could be used to correct the data.

nice.

[Atamido]

Originally posted by DaveEL
Check the post date its before par 2.0 this is a very old thread. Though the draft was just finalized, it has been mostly completed for a very long time. In fact, the first drafts were being drawn up shortly after 1.0 was released. :eek: Ironically I didn't even realize that the final draft was released until writing this post. Thanks DaveEL

But really, Parchives are just an implementation of the same old Reed-Solomon encoding that have been used on optical discs forever. So, the idea is to use reed-solomon encoding to increase error protection at a non-hardware level to make it more flexible for the amount you want to use each time.

[Atamido]

Originally posted by avih
however, i don't think it will be usable in real time playback, since it sounds to me as if it would take quote a long time to correct an error. In the case of a whole file, this is very true. But, if you are dealing with much smaller data chunks, as would be used within Matroska, then it should be easy to do on the fly.

[avih]

pamel, i guess you could be right, however, you'd need to implement it at the matroska level, which will make it hard to 'fix' a scratched cd to recreate the original clip without specialized matroska tools.

my idea was of an optional supplemental ECC facility for the file as a whole, to retain the original clip as much as possible, in an off-line type of execution, and with standard tools.

i think it's important that users will be able to correct scratched CDs and recreate the original. although a JIT ECC would be nice as well of course ;)

[DaveEL]

Originally posted by avih
pamel, i guess you could be right, however, you'd need to implement it at the matroska level, which will make it hard to 'fix' a scratched cd to recreate the original clip without specialized matroska tools.


Wouldn't a simple remux do as it would read it and correct the errors and then when you write it creates the ECC again.

DaveEL

[Atamido]

Originally posted by DaveEL
Wouldn't a simple remux do as it would read it and correct the errors and then when you write it creates the ECC again. Thats the idea. Or rather, you shouldn't even have to worry about it because you SHOULD be able to have the blocks repaired on the fly. This would mean that a damaged CD would still be watchable with a little more CPU power.

[Alestrix]

Kinda "overread" this interesting thread and thought to share my thoughts:

Originally posted by Pamel
Or rather, you shouldn't even have to worry about it because you SHOULD be able to have the blocks repaired on the fly.
Information about the file having (correctable) errors should still be given to the user, though. This way the user can recreate a fixed CD before the number of errors increases to a unrepairable amount.

- Alex

PS: looks like I'm gonna add par2 information to my XCDs from now on :)

[temporance]

Originally posted by Pamel
Thats the idea. Or rather, you shouldn't even have to worry about it because you SHOULD be able to have the blocks repaired on the fly. This would mean that a damaged CD would still be watchable with a little more CPU power. Depending on the scheme, it could take a lot more CPU power and disk access to recreate the damaged blocks. If you use a 3MB parity file to protect any contiguous 3MB data segment, then you need to scan in the whole disk in order to reconstruct any of the missing data. Not practical on the fly.

If you use a error correcting code with finer granularity (so that it can be corrected on the fly), you need more overhead to achieve a comparible level of protection - that's what's done on a conventional 650MB CDROM.