could a video get "loss" over time?

[kenny1999]

I had two quite advanced questions about video file to ask.

I once accidentally encountered a situation quite some time ago. I had a video file and I had a copy of it on another drive. The "copy" was directly copied and pasted from the original, but not taken from any other sources.

Last week I check the file with MD5 checksum tool. I tested the files but the file didn't match. I felt strange and then I checked the properties of the files and I found that the two big files (of several hundreds mega bytes) have 3 bytes (Yes, just 3 little bytes) different.

I never do any video editing. By the way, I just don't know how editing or changing a video file could result in such a tiny difference (3 bytes).

My question is
- Does anyone have any idea why it could happen.
- Somebody said that as time passes by the size of a stable file (even though they are never edited) could get deteriorated by some little bytes like iron corrosion or its MD5 checksum could change. Transferring (Copying) of a file from one physical storage media to another could also lead to such a small change. (I think it is a joke, isn't it?)

Please let me know any ideas. Thanks

[sneaker_ger]

- Somebody said that as time passes by the size of a stable file (even though they are never edited) could get deteriorated by some little bytes like iron corrosion or its MD5 checksum could change. Transferring (Copying) of a file from one physical storage media to another could also lead to such a small change. (I think it is a joke, isn't it?)

If files get corrupted over time or by copying then you have a hardware failure at hand. Reasons could be manifold, like RAM errors, HDD errors etc., so you should check these things. CD/DVDs and other mediums (any storage, really) have a limited lifetime, so you have to check them regularly and create new copies before any errors appear, as they are usually not recoverable.

[von_Runkel]

Copying a file from one hard drive to another hard drive can not change the file size or the checksum. There are built-in checksums in every known file systems that controls this - either the whole file is readable or - if the file is corrupt - the whole file is unreadable.

One exception is FTP transfers. FTP transfers take place at the application level, not the file system level. A part of a file can be transferred successfully, and the file system is happy with that. In this case you will have a valid file regarding the underlying file system - but the file would be "corrupt" regarding the reading application.

So, hardware failures is not the case here. Otherwise we could not trust any hard drive to deliver the correct data.

[Ghitulescu]

Well, it was in the mid '90ies a documented HW error on the IDE HW-SW-BIOS interface with some Intel chipsets.

On the other hand, CRC is not infallible.

Secondly, small errors do not automatically render a file useless, in particular with A/V-media - I remember reading people often had issues with downloaded AVIs (in the golden age of sharing), yet they could see the movie, even repair it under circumstances.

To the OP's original question, this could only happen at HW level. The operating system sets the CRC of the file after what it considers to be a successful copy - for speed reasons no READ-after-WRITE is performed (like in MOD or RAM discs) and the OK given by the HW is taken as COPY OK irrespective of its significance (it simply says that the HW successfully wrote the content, ie no errors occurred during writing).

[nevcairiel]

Any harddrive can have bit rot. Just by sitting there and doing nothing. There are some file systems (like ZFS) which are designed to fight bit rot by constantly checking your data against their internal checksums, and with the help of parity data try to fix it. But if your file system does not do this, it is possible for a file to flip a bit randomly over time, sometimes triggered by external factors like cosmic rays.

Luckily video compression is in many cases fault-tolerant so that a single bit error does not destroy the whole file, but maybe only one macroblock, or one frame.

[von_Runkel]

Secondly, small errors do not automatically render a file useless, in particular with A/V-media - I remember reading people often had issues with downloaded AVIs (in the golden age of sharing), yet they could see the movie, even repair it under circumstances.

Yes, but this happens at the application level and have nothing to do with the hardware. It's just an exercise in manipulating a correct file (according to the hardware and file system) to be readable by the application.


The operating system sets the CRC of the file after what it considers to be a successful copy

I am not sure about this. In my opinion it is the hard drive and the controller that is responsible for this.

[von_Runkel]

Any harddrive can have bit rot. Just by sitting there and doing nothing. There are some file systems (like ZFS) which are designed to fight bit rot by constantly checking your data against their internal checksums, and with the help of parity data try to fix it. But if your file system does not do this, it is possible for a file to flip a bit randomly over time, sometimes triggered by external factors like cosmic rays.

Luckily video compression is in many cases fault-tolerant so that a single bit error does not destroy the whole file, but maybe only one macroblock, or one frame.

You are correct but bit rot occurs extremely seldom and ZFS (and ReFS) fights this. Bit rot can destroy a gigantic database but is normally harmless in A/V applications.

I don't think the OP's problem is a HW failure.

[Ghitulescu]

I am not sure about this. In my opinion it is the hard drive and the controller that is responsible for this.

The way I knew was that the OS gives the command to the HDD to store data, then the HDD signals the data has been successfully copied to the HDD cache so the OS can go further (this speeds the things us quite remarkably) then the HDD copies the data from the cache using the algorithms it has to the real magnetic medium, with no read-after-write procedure, as this will cost at least 2 extra rotations (performance degrading). If the sector is found defective, the HDD remaps it somewhere else (there was a time when they had the luxury of a spare side for such things) - anyway that will be too late for the OS to be signalised.

[von_Runkel]

The way I knew was that the OS gives the command to the HDD to store data, then the HDD signals the data has been successfully copied to the HDD cache so the OS can go further (this speeds the things us quite remarkably) then the HDD copies the data from the cache using the algorithms it has to the real magnetic medium, with no read-after-write procedure, as this will cost at least 2 extra rotations (performance degrading). If the sector is found defective, the HDD remaps it somewhere else (there was a time when they had the luxury of a spare side for such things) - anyway that will be too late for the OS to be signalised.

We know this, but it is not relevant to the OP's question about 3 bytes missing from a file copy. You discussion above is about optimization of hard drive read and writes.

NTFS and other traditional file systems do NOT store CRC data in its file system; it is of the underlying hardware that is responsible for this. All data that is successfully read or written from the hardware is correct from the OS's point of view. Otherwise we wouldn't have any reliable data systems at all!

So the problem of missing 3 bytes of data after a file copy must be at the application level, for example a failed FTP transfer, or a transcode 1:1 that went wrong.

[wonkey_monkey]

I found that the two big files (of several hundreds mega bytes) have 3 bytes (Yes, just 3 little bytes) different.

I'm curious to know how many of those 24 bits were different :)

[filler56789]

I had a video file and I had a copy of it on another drive. The "copy" was directly copied and pasted from the original, but not taken from any other sources.

Last week I check the file with MD5 checksum tool. I tested the files but the file didn't match. I felt strange and then I checked the properties of the files and I found that the two big files (of several hundreds mega bytes) have 3 bytes (Yes, just 3 little bytes) different.

IF the unit that has the "copy" is a USB-drive, then the fault may be of the USB-adapter itself:

http://bigacid.wordpress.com/2008/12/08/jm20337-read-data-corruption-solution/

[Ghitulescu]

We know this, but it is not relevant to the OP's question about 3 bytes missing from a file copy. You discussion above is about optimization of hard drive read and writes.
Those bytes are not missing, they are different.
NTFS and other traditional file systems do NOT store CRC data in its file system; it is of the underlying hardware that is responsible for this. All data that is successfully read or written from the hardware is correct from the OS's point of view.
That is exactly what I said. Should have been the OS aware thereof, then the user would have been noticed.

[kenny1999]

Copying a file from one hard drive to another hard drive can not change the file size or the checksum. There are built-in checksums in every known file systems that controls this - either the whole file is readable or - if the file is corrupt - the whole file is unreadable.

One exception is FTP transfers. FTP transfers take place at the application level, not the file system level. A part of a file can be transferred successfully, and the file system is happy with that. In this case you will have a valid file regarding the underlying file system - but the file would be "corrupt" regarding the reading application.

So, hardware failures is not the case here. Otherwise we could not trust any hard drive to deliver the correct data.


FTP Transfer? Is it the same as HTTP transfer? Does it occur to HTTP Transfer as well? A file transferred by HTTP, in other words, a HTTP link containing the video

[kenny1999]

Any harddrive can have bit rot. Just by sitting there and doing nothing. There are some file systems (like ZFS) which are designed to fight bit rot by constantly checking your data against their internal checksums, and with the help of parity data try to fix it. But if your file system does not do this, it is possible for a file to flip a bit randomly over time, sometimes triggered by external factors like cosmic rays.

Luckily video compression is in many cases fault-tolerant so that a single bit error does not destroy the whole file, but maybe only one macroblock, or one frame.

Can bit rot that flips a bit be detected with MD5 or SHD-1 checksum?

triggered by external factors like cosmic rays.??？？ does it often happen or only happen under extreme conditions?

[nevcairiel]

Checksums can detect this yes, - and no, it does not happen very often.

[StainlessS]

Just curious, where was difference located, right at the beginning, end, in header ?

EDIT: What file container and codec ?

[foxyshadis]

I'm thinking antivirus, tagger, thumbnailer, indexer, or some other overly aggressive piece of software. Hardware problems are rare and tend to make themselves very well known in a very short time, but software that screws with media can be subtle and hide for quite some time.

[marquis]

I've lost numerous video files over time due to inexperience and poor equipment. I started making movies as a teenager, so just about everything I did back then has been erased because burned CDs get damaged and copied from one place to another can sometimes mess up. Now, I always make sure to keep things in more than on place and to make sure they stored properly. Sorry about your loss in files.