Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > New and alternative a/v containers

Reply
 
Thread Tools Search this Thread Display Modes
Old 19th June 2002, 03:37   #1  |  Link
Emp3r0r
Registered User
 
Emp3r0r's Avatar
 
Join Date: Oct 2001
Location: Alabama, USA
Posts: 769
self repairing disc

Has anybody heard of this new PAR thing they are doing on the newsgroups? I'm no expert but I believe any PAR file can take the place of any RAR file by using some sort of parity algorithm. Could this concept be applied to maybe XCD by use of maybe a couple megs to store parity information and then you could run a little proggy to repair the source disk... hence, a self repairing disc. I'd gladly set aside a few megs for this feature. Maybe there is already a much better way of doing such a thing, but in any case, I think this could help in overcoming the scratch.
__________________
ChapterGrabber - add names to your chapters | AtomSite - open source AtomPub server
Emp3r0r is offline   Reply With Quote
Old 19th June 2002, 10:53   #2  |  Link
Koepi
Moderator
 
Koepi's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 4,454
PAR is in NO way new, it's really really old in fact.

Those parity information (it's wrong to call it parity since with parity information it's impossible to _correct_ data).

The overhead is immense, I once got hold of something secured this way and the PAR stuff was ~1/4 of the size of the archive.

so - why using PAR? On a 700MB CDR we gain 100MB from mode2form2. BUT we have 150MB recovery info and 650MB real file left....


So your proposal is to make a 700MB CDR a 650MB CDR.

Sorry, I don't support that

Regards,
Koepi
Koepi is offline   Reply With Quote
Old 20th June 2002, 13:15   #3  |  Link
Latexxx
Registered User
 
Join Date: Nov 2001
Location: Tampere, Finland
Posts: 618
Correct me if I'm wrong. Ogg and upcoming mcf have error correctioning in fileformat.
__________________
A/V moderator @ hydrogenaudio.org
My weird old sh*t: http://www.nic.fi/~lhahne/
http://last.fm/user/Latexxx/
Latexxx is offline   Reply With Quote
Old 20th June 2002, 13:44   #4  |  Link
Koepi
Moderator
 
Koepi's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 4,454
No, they don't have error correction - but they are error tolerant and just skip broken parts.

Regards,
Koepi
Koepi is offline   Reply With Quote
Old 21st June 2002, 03:59   #5  |  Link
Emp3r0r
Registered User
 
Emp3r0r's Avatar
 
Join Date: Oct 2001
Location: Alabama, USA
Posts: 769
I didn't propose using 150 megs megs worth of parity information, just a small amount. Is this not possible? For example, you have a lump of data, say 725 megs big... you decide to add 25 megs of error recovery. That would make 29 25meg seperate sections of data (30 overall). so if maybe you get a scratch in the middle that corrupts just one 25 meg section of data, you could run a little proggy that would use the error correction data and spit out your original 725 meg file.

@Koepi: maybe I don't know what I'm talking about (which is highly likely) but I don't see why the concept couldn't be applied for sensitive backups. BTW, i know the concept of parity checking is really really old, but the way the release groups are using it for newsgroups is new afaik, well at least in the last 5 or 6 years new. I haven't downloaded anything from newsgroups since Windows(96) Nashville.
__________________
ChapterGrabber - add names to your chapters | AtomSite - open source AtomPub server

Last edited by Emp3r0r; 21st June 2002 at 04:01.
Emp3r0r is offline   Reply With Quote
Old 22nd June 2002, 03:27   #6  |  Link
pandv
Registered User
 
Join Date: Oct 2001
Posts: 62
XCD and PAR

XCD has error detection at the sector level?

If the response is yes, I think it's possible to implement a PAR like protection scheme.

If one sector is 3 Kbytes (rounding up) with 3000 Kbytes in PAR information you can recover a disc with 1000 sectors damageds.

If there aren't error detection forgot it.


pandv.
pandv is offline   Reply With Quote
Old 22nd June 2002, 11:55   #7  |  Link
Koepi
Moderator
 
Koepi's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 4,454
Head over to xcd.sourceforge.net and read the specs.
Then you know what is done - and waht is not done.

Btw., there won't be anyting like PAR. But if you read the specs you'll know why.
Koepi is offline   Reply With Quote
Old 22nd June 2002, 15:44   #8  |  Link
pandv
Registered User
 
Join Date: Oct 2001
Posts: 62
Ok, I understand a protection scheme as Par is not in the specs (the protection level is your backup aplication).

But i am thinking about implementing this scheme maybe at a high level. When I burn a CD I have always free space. Maybe only 3 Mb, maybe more.

What about to create a file with the free space, and use it to store recovery information? If your CD is damaged you can try (with a special aplication) to recreate a correct ISO for it.

Can this be useful?

It's only a sugestion for a idle programmer (exists?). I don't have the knowledge nor the time.

pandv.
pandv is offline   Reply With Quote
Old 23rd June 2002, 10:38   #9  |  Link
Milkman Dan
Registered User
 
Join Date: Oct 2001
Posts: 243
Guys.
CRC/what you're calling 'parity' doesn't recover information.
It detects errors, rather than recovering them.

Even if it did recover information, it would take ALOT more space than 3 MB. Think more like RAID 5. Minimum # of discs for RAID 5 is 3. You lose the space of a third disc because it holds the (true)parity of the other 2. For every block written to two discs, an XOR operation is performed, and the result stored in round-robin fashion on the remaining disc.

CRC is already built into MCF. I don't know what Ogg holds in this arena. But neither does error RECOVERABILITY as far as I know.
Milkman Dan is offline   Reply With Quote
Old 23rd June 2002, 11:12   #10  |  Link
Koepi
Moderator
 
Koepi's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 4,454
Dan,

that's exactly what I'm talking about. I think my first reply was VERY clear?!?!

Regards,
Koepi
Koepi is offline   Reply With Quote
Old 23rd June 2002, 22:37   #11  |  Link
pandv
Registered User
 
Join Date: Oct 2001
Posts: 62
I am sure my english is not good enough to make my point.

@Koepi

Your information is very clear for me, and I trust it.

@Dan

CRC doesn't recover information.
OK. The named PAR doesn't uses CRC for correction. For the detection it uses MD5 checksums. Despite the name, it uses Red Solomon Code for correction.

Spec at: http://sourceforge.net/docman/displa...group_id=30568

Of course, this spec is not valid to protect a CD, but the idea is here.

About RAID.
With a 50% overhead you get protection for the 50% of the data (one complete disk).
With 3 Mb you get protection for 3Mb of the data (maybe a scratch in the CD surface).
There are no miracle here.

I'm not proposing to change XCD to implement a PAR like protection scheme, I'm only saying a similar scheme maybe can be useful.

There are computacional dificults (there are thousands of sectors, not a few rar files to protect), and specification dificults (they need to be implemented at ISO level if the internal CRC protection of the CD is used).

So, I am only arguing about the usefulness of this thing, and not proposing any concrete action.

pandv

Last edited by pandv; 23rd June 2002 at 22:39.
pandv is offline   Reply With Quote
Old 23rd June 2002, 23:10   #12  |  Link
avih
Capture, Deinterlace
 
avih's Avatar
 
Join Date: Feb 2002
Location: Right there
Posts: 1,971
@pandv
you are correct that error recovery facility IS required for some parts of the media file, while for others, error detection is enough.

there's also the known trade-off between the level of protection to the size of the overhead data.

now, the whole point of xcd is that most media files DON'T need error correction, since detection is enough. (as in audio-cd or svcd). if there are errors, there are temporary artefacts during playback, and after that the playback resumes normally. so the aim was to throw-away the overhead involved in error CORRECTION, and leave only the error detection. that why the media file is in mode 2 form 2 instead of mode 2 form 1 (which is the same as normal mode 1 for that matter)

since mode 2 form 2 (which is the storage for the main media file) DOES have error detection, all we need is error CORRECTION for the really important parts. and that is implemented in mode 2 form 1 file. again, read the spec.

now after you've read my explenation and the spec, can you define exactly why do we need PAR? do u think it should REPLACE the header file correction method? do u think it should be applied for the media file ON-TOP of the error detection? or maybe instead of the error detection?

pls explain youre idea again, and with as much details as u can, with references to the current xcd spec.

thx
avi

Last edited by avih; 23rd June 2002 at 23:16.
avih is offline   Reply With Quote
Old 24th June 2002, 17:42   #13  |  Link
spyder
Matroska Developer
 
spyder's Avatar
 
Join Date: Nov 2001
Posts: 315
Why are you people so worried about error correction over the entire disc? No one ever makes such a big fuss over VCD/SVCD not having error correction. BTW something of this nature is already planned for MCF. Tronic suggested separate ECC files to correct the data lost in the original. I might also add ECC support for my MCF server. But you can hear more about my server later when it's under development.
spyder is offline   Reply With Quote
Old 25th June 2002, 04:07   #14  |  Link
Milkman Dan
Registered User
 
Join Date: Oct 2001
Posts: 243
Quote:
Originally posted by Koepi
Dan,

that's exactly what I'm talking about. I think my first reply was VERY clear?!?!

Regards,
Koepi
Yeah, I know. They weren't getting it though. Thought I'd try a different tack.
Milkman Dan is offline   Reply With Quote
Old 25th June 2002, 04:16   #15  |  Link
Milkman Dan
Registered User
 
Join Date: Oct 2001
Posts: 243
Quote:
About RAID.
With a 50% overhead you get protection for the 50% of the data (one complete disk).
Not quite. When a disk in a RAID 5 fails, the disk is rebuilt from the parity present on the other two(or more) disks. The array (and the data) is still safe with the failure of one(or more) disks.

Thus, the integrity of the entire array is safe. This is not so for a CD protected this way. You give up much more than 3MB. Which brings me to your comment about protecting just 3MB that might be destroyed in a scratch.

Well, how do you know which 3MB to protect? You could spread the bits out really thin, but then you lose the effectiveness of the protection.

I respect you bringing up the utility of such a system, but it doesn't and can't feasibly work. It takes too much space on the limited medium of a CD, and hard disks already have protections in place, such as RAID 1, 10, and 5.
Milkman Dan is offline   Reply With Quote
Old 27th June 2002, 22:24   #16  |  Link
pandv
Registered User
 
Join Date: Oct 2001
Posts: 62
Thank's for your replies.

I not propose to change XCD specs.

I think this need more thinking. I investigate on the subject when I get to hollidays.

But,

@Milkman Dan
Par protects any damaged part. Let's explain the common use. You have a 40 rar set, but you get only 39 rars and a par. With this par you can recover the missing rar (any missing rar, with any par file). You only need a quantity of par files equal to the missing rar files.

pandv
pandv is offline   Reply With Quote
Old 27th June 2002, 23:49   #17  |  Link
DaveEL
Moderator
 
Join Date: Nov 2001
Posts: 581
a small amount of data cannot be enough you cant reconstruct the data

3mb of data would not protect any 3mb of data just 3mb of data in a very specific pattern (in the case of par 3mb of data within a single 3mb file) in order to get an real protection you need a lot more

using your example of par i decided to give it a go

To protect 232mb of video stored in 7 files from failure of at most 3 files i need to add an extra 145mb of data.

Even to protect 1 file i needed ~50mb (size of the largest file(ish)) of course i could reduce this by making the videos smaller and using playlists to turn them into one playable file but if i split them into 3mb files its far more likely several would have errors and so i would need to generate lots of par files to be safe and so it would take up more then 3mb to protect them.

DaveEL
DaveEL is offline   Reply With Quote
Old 6th June 2003, 15:07   #18  |  Link
frodoontop
Registered User
 
frodoontop's Avatar
 
Join Date: Jan 2003
Location: Netherlands
Posts: 126
Now Matroska with error correction is not in sight yet and there is a new, superior version of par, I thought this could be potentially handy for backup. So I'm bringing this thread alive again.

On hydrogenaudio.org they already had a discussion for backing up music, however this wasn't applied with XCD. You can find that discussion here . You can also find more information about par2 there.

Would it be possible to store this par2 file on Mode2-filemode? And use dat2file to get both files back from CD? I know this will not be as effective as Matroska's coming intelligent error correction, but on the other hand it won't hurt using par eather, I suppose.

I'm not planning to use this on every CD, but just for those that have filesizes like 705-780 MB.
frodoontop is offline   Reply With Quote
Old 6th June 2003, 19:55   #19  |  Link
temporance
Registered User
 
Join Date: Mar 2002
Posts: 486
Well, I think it's quite a good idea. As soon as an XCD disk develops an error, you run a special application that uses the reundant block to repair the dodgy data. Then you burn a fresh disk with the original data, error-free.

With 3MB, you can repair any single contiguous block of 3MB on the disk. If there are errors in more than one place on the disk, it's less likely that it's repairable.

The reason that this hasn't been done before on CDs/DVDs is that it would take a very long time to recover the lost data. We're used to waiting a long time for things (ripping, encoding, burning, etc.) so this is more acceptable for us.
temporance is offline   Reply With Quote
Old 7th June 2003, 01:04   #20  |  Link
Atamido
Seņor Member
 
Atamido's Avatar
 
Join Date: May 2002
Location: Austin, Texas
Posts: 915
@DaveEL: You numbers are not quite right. First you must know that there are two versions of PAR, 1.0 and 2.0.

Wit 1.0 you can create parity files with several blocks of data, but it requires that all of the parity files be at least as large as the largest data block. So, if you have 10 file ranging between 2MB and 50MB, your parity files would be 50MB. And you would need one 50MB parity file, in addition to the rest of the set, to repair/rebuild a single file. So, 50MB to rebuild a 2MB file. This method is really only good where the data blocks are all the same size. IE, you have 30 files that are 10MB each. You have 5x10MB parity files. You can use these to replace up to 10 corrupt files.

With 2.0, you pick a set parity size, and only need at least as much data as you need to rebuild. So, this is what you would use with a single large media file as it could be used to replace part of the file.

In your example of 232MB that you want to protect (doesn't matter if they are in a single file, or multiple files). You just create a 200 100KB files for parity which would take up 20MB. That would allow you to replace up to 200 non-contiguous damaged segments in the file(s). Adn/or you could replace a single 20MB missing data chunk.

The idea behind 2.0 is what we were planning for Matroska. You have several data chunks of random sizes. You have a user-selected parity block size, and number for each cluster. Then you should be able to rebuild clusters on the fly...
Atamido is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 15:02.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.