Good codec for professional compression/archiving?

[syc0pathic]

I work at a data research archive. Basically, some1 does an experiment and gives their data to us. We standardize the data and make it available to other researchers. Up until now, this has primarily been survey and interview data. But now we're exploring the idea of archiving and distributing video data. Since I do video encoding/back-ups/editing for a hobby, I volunteered to work on the project. Nobody else has much experience in this area.

We expect to get all different types of video formats -- VHS, DVD, DV tape, WMV, MPEG, etc. All of it will be converted if necessary and stored as computer files. So the 1st thing we have to decide on is a standard format. Of course, my 1st choice would be Xvid. We could potentially end up storing tens of thousands of hours of this data, so good compression (w/ minimal loss of quality) would be very helpful. On the other hand, good archiving practices demand that whatever format we use must be ezily converted as newer/better formats come along in the future.

Right now, there's a lot of resistance to using Xvid, becuz it's not a "legitimate" codec and there are concerns about its longevity and future compatibility. They do like the fact that it's open source though. I also suggested some type of MPEG format, as that seems like a more universal "generic" format. Of course, it doesn't offer nearly the compression of Xvid. Someone else suggested WMV, but that might even be worse than Xvid in terms of longevity. Sure, it's a Microsoft format, but so was ASF and when was the last time u saw 1 of those?? Microsoft is a huge company, but anything that isn't open-source will be more difficult to deal w/ in the long run. Besides, the idea of working w/ Microsoft codecs makes my skin crawl:eek: DVD/VOB files was also suggested, but the file size makes that impractical.

So... any thoughts or suggestions? What do professional video studios use?

[Manao]

Does encoding speed matter ? Decoding speed ? If you were using XviD, what quantizer would you try to target ( that allows us to what what compression/quality tradeoff you're actually seeking ).

For the container, I would use mp4, but it depends on the codec.

Do you plan to compress the audio too ?

Without your answers to those question, I would advise x264 / aac / mp4.

[Gornot]

I'd reccomend x264 in an MP4 container. Although XviD/DivX is a global standard for video compression, with x264 you can squeeze out the same quality and have a smaller filesize. For example, if an 1h 30mins XviD movie "costs" me 700MB of space, the same file in MP4 is only 350MB and the quality is the same. Of course, that depends on which software you're using and what settings you prefer...

[syc0pathic]

Does encoding speed matter ? Decoding speed ? If you were using XviD, what quantizer would you try to target ( that allows us to what what compression/quality tradeoff you're actually seeking ).

For the container, I would use mp4, but it depends on the codec.

Do you plan to compress the audio too ?

Without your answers to those question, I would advise x264 / aac / mp4.Encoding speed isn't critical, but if it takes 2-3 times long than Xvid/avi, then it probably isn't practical.

We've been talking about a 2 hour video clip fitting on a 700MB CD. So that translates into an Xvid bit rate of about 685 w/ 128kb MP3 audio. Some of our source material will probably be low quality (like captured VHS:( ) so that should allow for even lower bitrates.

Keep in mind that while compression is a concern, future compatibility is essential. Xvid/MP3/avi is considered too "cutting edge" for most of the people on the video committee, so I doubt they're gonna go for an even newer format/codec/container.

[Manao]

We've been talking about a 2 hour video clip fitting on a 700MB CD. So that translates into an Xvid bit rate of about 685 w/ 128kb MP3 audio.Then, without a shadow of a doubt, x264 + aac + mp4. Even XviD's afficionados will tell you so. Future compatibility is ensured - everything will be able to decode h264, since most content ( HDDVD/Blueray/HDTV ) will be encoded in that format soon enough.

Aac is more efficient than mp3, especially since you target rather low bitrates. You'll be able to use 64kbit/s for audio, which will raise the video bitrate a little.

[Gornot]

Well, actually, ogg vorbis is more efficient than aac, not to mention mp3, but that's not the point here :) ...
I'm glad that people are finally starting to see just how much x264 is a good solution. I used to rip my DVDs into RV10 (rmvb) and OMG it looked so bad! I gave up on DivX and XviD a long time ago (at least for personal rips) and, after one of my friends recommended x264 I finally found what I was looking for. That's actually how I found doom9...

[Kid-A]

I downloaded x264 and I want to use it in VirtualDub-MPEG2. I double-clicked x264.exe but nothing happened, I think. And I can't see it in VIrtualDub's compression list. Am I doing something wrong?

[check]

@kid-a: yes, posting in the wrong place.

I can understand why the commitee won't jump at formats they haven't even heard of, so I'll offer some arguments you can make against the other formats:
o MPEG1 or MPEG2: just... far too low quality for your target size. You can encode some tests to see, even with VHS it will look pretty bad.
o WMV: while the latest version (VC-1) has been standardised and is an open codec, it has far less open source support than MPEG* codecs (xvid is MPEG-4 ASP and h264 is also MPEG-4 AVC), and future compatibility is nowhere near as well guaranteed.

Now, as for why you should use h264 + aac + mp4, the answer is simple: compatibility. These are all the latest official MPEG standards, and thus you can be assured they will be well supported. There are free open source tools for all of the above and you will find plenty of commercial support as well, them being MPEG standards.

[Mug Funky]

@ Gornot:

proof on the vorbis v aac thing? last i checked they were around the same bitrate for perceptual transparency, and for decent sound at low bitrates AAC was the winner.

vorbis is still good, and interesting because it's patent free and still more efficient than mp3 (and my little flash player wont play aac but will play vorbis).

[edit]

@ syc0pathic:

you might want to edit the title of this thread as per the rule "don't ask what's best"... it's a technicality, but there for a good reason because (as i'm sure you're finding out) the ideal solution is different for every situation.

[syc0pathic]

This is good info guys -- thanx! We have another meeting in a couple days, so I'll discuss it w/ them then.

There's also another issue I didn't ask about... what software is required to play these files? I would guess that most of the researchers who use these files won't have admin rights on their work computers. This is also 1 of the objections against using Xvid, but at least the Xvid installer is small. And once it's installed, people can watch the vids on WMP -- something virtually every copy of Windows has. I searched around this site and while I didn't find any specific info on x264/MP4 playback, it does look like it's more complicated than Xvid playback.

[Blue_MiSfit]

There are several options for decoding MPEG-4 ASP (XviD/DivX) and MPEG-4 AVC / H.264 (x264).

Possibly the easiest method is to use MPlayer, which is a simple executible (shouldn't require admin accounts), and has all the necessary decoders without depending on DirectShow (like WMP).

If you want to make x264+AAC playable in Windows Media Player, you need 3 things, an MP4 splitter, an H.264 decoder, and an AAC decoder. ffdshow can handle H.264 and AAC, and Haali's splitter can handle splitting the MP4.

So thats 2 things, that you need admin account to install. Mplayer would surely be simpler :) There's also some way to wrap videos and Mplayer into a single executible.

~MiSfit

[Manao]

You can use :
- mplayer (http://www.mplayerhq.hu) : doesn't need to be installed - but no GUI. Doesn't need any third party program/filter.
- VLC (http://www.videolan.org) : it is installed, it has a GUI, it doesn't need any third party program/filter
- DirectShow-based solutions : FFDShow for decoding h264 and aac, Haali's splitter (http://haali.cs.msu.ru/mkv/MatroskaSplitter.exe) for demuxing mp4. I can't give you a single URL for FFDShow, there are too much of them. You may want to look at x264.nl, or at the ffdshow-tryout project, or even CCCP.

Actually, FFDShow can also handle XviD, WMV, mp3, AAC, and an impressive number of other codecs. It's as simple as XviD to install. Alone, it can understand h264 in avi, but it would really be better to put the whole thing into an mp4, hence the need for Haali's splitter.

Amongs the three solutions, I would advise VLC. It's cross platform, it works on almost every kind of video, and it has a GUI. Once installed, it does take up a little bit of space ( 35 MB on my computer ), but it's nothing with nowaday's computers.

[Gornot]

On my part I'd recommend the simple Media Player Classic 6.4.9.0
It doesn't have an installer, it also requires ffdshow and haali's media splitter, and it supports a large number of different formats.

The best way is to try all of the mentioned yourself and just go for the one that suites you best...

[smok3]

x264/aac (or he-aac) in mp4 would be my vote as well, for playback you can use MPUI on windows (or any mplayer based distro on other systems). You wont be able to convert to newer/better formats thought, since that would be retarded, so you dont even have to consider that.

[Revgen]

MPEG-2:

Advantages:

-Excellent Compatibility.
-Quick Encoding and Decoding.
-Easy to Edit. Can be edited without re-encoding.
-Lot's of Editing Apps support it.

Disadvantages:

- Compression isn't as good as other codecs. Certainly not good for 1 CD encodes.


Xvid:

Advantages:

1) Good Compression
2) Easy to Edit with Virtual Dub. Can be editited without re-encoding.
3) Doesn't require a lot of CPU power to playback, but not quite as easy as MPEG-2.
4) Speed isn't as good compared to MPEG-2, but not nearly as slow as H264


Disadvantages:

Not a standard codec. May have issues as far as compatibility in the future.


H264

Advatages:

1) Excellent Compression
2) It's a standard codec. Should be future-proof.

Disadvantages:

1) Currently it is slow to encode and may be too slow to decode for some older computers depending upon the content.
2) Not easy to edit. Not too many (if any) current solutions that allow you to edit without reencoding.

[zambelli]

Someone else suggested WMV, but that might even be worse than Xvid in terms of longevity. Sure, it's a Microsoft format, but so was ASF and when was the last time u saw 1 of those??
ASF? I see it every day and most likely you do too. ASF is still the Microsoft container of choice for WMV and WMA codecs. Any .wmv or .wma file is an ASF file. They're all over the web in download form or streaming form... so yeah, I'd say ASF is very common, probably more than ever.
Microsoft is a huge company, but anything that isn't open-source will be more difficult to deal w/ in the long run.
Since your question was actually about a codec... Then WMV9/VC-1 is as good of an option as any MPEG-4. WMV9 has been standardized by SMPTE as VC-1, much like the MPEG codecs, so there's nothing proprietary any longer about a WMV9 video bitstream - any VC-1 decoder can decode it. Considering VC-1 is present in both HD-DVD and BluRay standards, it's unlikely it will go away as long as those formats are alive.

While not the most efficient, MPEG-2 is probably your best bet in terms of support and longevity, given its dominance in the industry of the last 20 years.

[syc0pathic]

You can use :
mplayer : doesn't need to be installed - but no GUI. A lot of our users are not very computer-literate, so giving them a program w/ no GUI would discourage most of them from using our video data.

After reading the info in this post, doing my own research, and considering what our users would be able to handle, I've come to the conclusion that we're gonna need 2 different formats -- 1 for preservation and 1 for distribution. WMV is the obvious choice for distribution -- every1 can play it w/o installing anything or trying to learn a new program. We currently use 4 different formats for our text data, to insure universal compatibility and preservation. Of course, encoding video takes orders of magnitude more time... but I digress. Anyway, for preservation, it sounds like x264 is the way to go, given its excellent file size/quality compromise and potential for future compatibility.

On the other hand, I came across some guidelines for video preservation, and they recommended that archive copies should use lossless compression. They didn't give an explanation, but it got me to thinking about the "stack-up" of compression artifacts. For example (and I don't know if this is a realistic example, but it should get the pt across) let's say that the 1st lossy codec we use tends to discard color information to reduce file size. Then a few yrs down the line, another lossy codec comes along that tends to discard shading information. Either 1 of these codecs would be fine if used w/ an original source. But since the video has been converted twice, both color and shading info have been lost -- and the quality of the video has been noticeably downgraded. Of course, if we stick w/ lossless compression, the file sizes may make archiving impratical.

ASF? I see it every day and most likely you do too. ASF is still the Microsoft container of choice for WMV and WMA codecs. Any .wmv or .wma file is an ASF file. They're all over the web in download form or streaming form... so yeah, I'd say ASF is very common, probably more than ever.I thought ASF was a codec... my bad. It's just that I used to see a lot of files w/ a .ASF file extension (and the quality was usually pretty bad) but now u don't. But u do see WMV, so I thought that was the replacement for ASF.

[Gornot]

Of course, if we stick w/ lossless compression, the file sizes may make archiving impratical.

Exactly... That's why, no matter how many format's you're gonna use, you have to be sure that you do both encodings from the best quality source file. I mean, encoding to wmv and then using that file for encoding to mp4/mkv (x264) would be just :eek: :scared:

[Manao]

The best lossless way to encode is not to encode at all ( at least not what is already numerized and compressed with a video codec ).

You really have to weight your different options. Uniformization of your videos format may not be needed after all. As it has already been said, lots of players are able to play almost anything. Of course, WMP can't, but between installing VLC on every computer and compressing all my videos, losing both time and quality, my choice is easily made.WMV is the obvious choice for distributionI disagree. It highly depends on the OSs you're using.

[syc0pathic]

Exactly... That's why, no matter how many format's you're gonna use, you have to be sure that you do both encodings from the best quality source file. I mean, encoding to wmv and then using that file for encoding to mp4/mkv (x264) would be just :eek: :scared:Of course, both copies would be encoded from the source.

The best lossless way to encode is not to encode at all ( at least not what is already numerized and compressed with a video codec ).

You really have to weight your different options. Uniformization of your videos format may not be needed after all. As it has already been said, lots of players are able to play almost anything. Of course, WMP can't, but between installing VLC on every computer and compressing all my videos, losing both time and quality, my choice is easily made.
Quote:
WMV is the obvious choice for distribution
I disagree. It highly depends on the OSs you're using.Becuz of the sheer volume of files, everything has to be standardized. It would be too difficult to keep track of 20 different formats. And as new codecs/formats come along, everything will be batch-encoded, and that will be much ezr if they're all the same format.

I'm not sure if u guys are understanding the difficulties involved w/ making these files available to our users. We're not talking about a few people around the office. We're talking about thousands of users, all over the world, w/ greatly varying levels of computer knowledge. Therefore, the process of getting the files and playing them must be extremely simple. We can't ask them to install a bunch of new software (remember that many of them don't have admin rights) and learn how to use it. Even installing the Xvid codec would be a major complication. Since the vast majority of people use Windows OSs, WMP is the way to go for distribution -- and those that don't use Windows would probably be computer-savvy enough to figure out how to play the files (well, maybe except for Mac users:p )

[setarip_old]

@syc0pathic

Hi!Becuz of the sheer volume of files, everything has to be standardized.Therefore, the process of getting the files and playing them must be extremely simple.These two statements cry out for MPEG-1 format.

Although it won't give you the degree of compression of other formats mentioned throughout this string, an MPEG-1 file only requires a doubleclick of a PC mouse to be played ;>}

As far as storage, the cost per Mb just keeps going lower and lower for hard drives and burnable media, so I don't think this should be either a concern or a deciding factor regarding format...

[shakey]

I'd also suggest against wmv. You say most researchers use windows, but sadly this isn't true. The entire department at my uni uses linux, and I'm pretty certain some of the others do to. I also know that the change only happened within the past couple of years, so you may find that other institutions start moving away from windows too, at least in part. A video format that can be easily decoded on as many different O/S as possible suck as Xvid, h264 or MPEG1/2 seems to be the way to go.

[LoRd_MuldeR]

I'd recommend to use H.264 (x264) in MP4 container.
It's the most efficient format available today, it's an official standard, it's platform independent and it's future-proof...
Furthermore I really think you shouldn't expect to majority of users to be on Windows!

You say that you don't want people to install additional software.
Okay, but a player with support of H.264/MP4 should be a minimum standard today, no matter which OS.
The best way would be to use MPlayer. It can be distributed with the video, it can run form the disc (no install process needed) and it's available for all important platforms (Windows, Linux, MacOS, etc)

[*.mp4 guy]

Libavcodec Mpeg1. Its about the same quality as most commercial mpeg4-SP compressors, and almost as good as mpeg4-ASP, the quality is quite incredible for an mpeg1 codec at low bitrates. Really there isn't a good reason to use mpeg4-asp, its only a little better then libavcodec mpeg1, but is much less compatible and future proof. Its either Mpeg1, or Mpeg4-AVC (H.264), mpeg1 is currently the most widely suported format, and AVC is going to be widely supported in the future. I would suggest archiving in AVC+LC-AAC(@96kbps-160kbps) in the .mp4 container, and distributing in Mpeg1 for the time being, you can switch over to distributing your archived AVC in Mp4 files when there is more widespread support for the format.

[syc0pathic]

I'd recommend to use H.264 (x264) in MP4 container.
It's the most efficient format available today, it's an official standard, it's platform independent and it's future-proof...
Furthermore I really think you shouldn't expect to majority of users to be on Windows!

You say that you don't want people to install additional software.
Okay, but a player with support of H.264/MP4 should be a minimum standard today, no matter which OS.
The best way would be to use MPlayer. It can be distributed with the video, it can run form the disc (no install process needed) and it's available for all important platforms (Windows, Linux, MacOS, etc)So MPlayer is all they need to play h264/MP4 files -- there's no other additional codecs or anything? But even if that is the case, we're back to the problem of not having a GUI. I just downloaded it and used it to play an Xvid file... it wasn't as hard as I thought it would be.

We just had another meeting, and we decided to talk to our users about what formats they would prefer. I could include questions about whether or not they would feel comfortable using MPlayer.

As for archiving, they are really pushing the idea of using lossless compression. They claim that space won't be a problem... another 5TB of space was just allocated to us, and we can partner w/ other archives that have a lot of space. I've never used lossless compression, so I don't know a lot about it. I searched on this site and I found some info... can the h264/MP4 format be used to make lossless copies? I read this post: http://forum.doom9.org/showthread.php?t=123984&highlight=lossless

It says something about changing the profile for lossless compression. Or is there another codec/container/format I should look at? Can u give me a rough estimate of processing times and file sizes for a 2hr clip using lossless compression?

[Manao]

There aren't any major lossless standard as of now.

Anyway, be aware that good lossless codecs compress between a fifth and a third of the original _uncompressed_ size.

Let's say you've got a DVD, 2 hours long. Uncompressed, it will be 110 GB. Losslessly compressed, it will be between 22 to 36 GB. The DVD it self is less than 9 GB, with audios.

So imho lossless it definitely not the way to go. No real standard, loss of space... Better not to compress at all.

[LoRd_MuldeR]

So MPlayer is all they need to play h264/MP4 files -- there's no other additional codecs or anything? But even if that is the case, we're back to the problem of not having a GUI. I just downloaded it and used it to play an Xvid file... it wasn't as hard as I thought it would be.

MPlayer is self-contained. That's the good thing about it :)
You won't need any codecs, filters, plugins or whatever it is called. All required decoders/splitters are built-in.
Only for some proprietary formats MPlayer needs to load external DLLs, like nasty Real or WMV stuff.

MPlayer itself has an official GUI. It's called "gmplayer".
AFAIK you can simply build MPlayer with GUI enabled, think it's just some compiler switch.
Nevertheless there are dozens of other GUI's for MPlayer available, like "MPUI", "smplayer" and many more.
I even made a small tool to bundle a video file with MPlayer to one self-running EXE file (it's Win32 only though)
Shouldn't be too hard to find a GUI that suits your needs ;)

I also think lossless is not the way to go for distributing a video. If lossless at all, then H.264 in lossless mode...

[SealTooGreat]

I would recommend gmplayer - mplayer built-in GUI . It's very clean and neat, has all basic options you need and above all it's less buggy than other mplayer GUIs. Don't need installation, can run without admin rights and supports various containers and codecs - so your clients don't get only one purpose player.
VLC also comes without installation.
Considering codec/container issue, does standalone compatibility have any part of this project?!

[syc0pathic]

MPlayer is self-contained. That's the good thing about it :)
You won't need any codecs, filters, plugins or whatever it is called. All required decoders/splitters are built-in.
Only for some proprietary formats MPlayer needs to load external DLLs, like nasty Real or WMV stuff.

MPlayer itself has an official GUI. It's called "gmplayer".

I also think lossless is not the way to go for distributing a video. If lossless at all, then H.264 in lossless mode...Yeah it looks like MPlayer will work ok, and I included questions about it in the survey of users that we sent out.

As for lossless compression, that is under consideration ONLY for archiving. It would be ridiculous to distribute anything like that. Of course, it's pretty ridiculous for archiving too:rolleyes: Don't DVDs use MPEG-2 as their native format? I think digital camcorders do too, and any video capture card (which we'll be using for VHS sources) that I've seen uses MPEG format as well. So basically, everything that we receive from those sources will be in MPEG-2, right? It has already undergone some lossy compression, and decompressing it won't replace what was lost. The best we can do is keep it in MPEG-2 and not compress the original any more than that. Therefore, MPEG-2 should be our archival format. Make sense?

Hmm... I just thought of something else. At some pt in the future, we'll probably start getting HD-DVD sources. Won't we lose quality by converting them to MPEG-2?

[Manao]

Camcorders use DV. Capture cards allow you to capture to mostly any format.

I have trouble to understand why you want a unique format. I get the standardization idea, but what for ? Mplayer / VLC will be able to decode anything anyway.

Furthermore, you really have to state clearly your constraints, and the order in which they prevail :
- encoding time
- compression efficiency
- target quality
- decoding complexity
- player constraints ( size, OS, ... )
- ...

Do also remember that, whatever the codec, you will always lose either quality or increase bitrate. Only conversion from the less efficient codecs to the most efficient ones would reduce the quality loss.

[syc0pathic]

I have trouble to understand why you want a unique format. I get the standardization idea, but what for ? Mplayer / VLC will be able to decode anything anyway.

Furthermore, you really have to state clearly your constraints, and the order in which they prevail :
- encoding time
- compression efficiency
- target quality
- decoding complexity
- player constraints ( size, OS, ... )
- ...

Do also remember that, whatever the codec, you will always lose either quality or increase bitrate. Only conversion from the less efficient codecs to the most efficient ones would reduce the quality loss.Standardization is essential for archiving, becuz then only 1 solution is needed when updating to newer formats (something that should be done every decade or so to ensure that the data doesn't get stuck in an obsolete, unusable format). MPlayer may be able to play anything right now, but will it be around in 100yrs? Our video data probably will be.

As for the constraints, keep in mind that we're talking about 2 separate tasks -- archiving and distribution to users. For archiving, future compatibility is essential. The 2nd priority is minimize, or if possible, eliminate loss of quality at each format conversion. Based on what what we've discussed at the last meeting, processing time and filesizes don't really matter for archiving.

For distribution, ease of playback is probably the biggest concern. The next priority is small filesize, at least to the extent that quality isn't severely affected.

I came across some interesting information while I was researching this... it's a sales proposal from Sun to Hollywood movie studios about preserving movies in a digital format. If u'r curious about the topic, it's an interesting read and explains more about the goals and process of archiving: http://blogs.sun.com/kevinr/resource/Archiving%20Movies%20in%20a%20Digital%20World.pdf

[foxyshadis]

I like that sun paper, I'll keep a copy here.

It'd probably be best to create a duopoly of formats: Anything that comes to you in mpeg-2, leave in mpeg-2, though perhaps in a standard container, like mpeg-ps or mp4. Likewise, AVC is being embraced as the current mpeg standard, so it makes sense that anything you can't afford to store uncompressed should be AVC. The lossless you could create now won't be compatible with the lossless you create in a year, whereas any q1 encode will be. If you need 4:2:2 and 4:4:4 you'll probably have to wait until the revised standard comes out this year anyway. Audio can be uncompressed without a serious burden.

The 100 years factor is separate. Nothing you create today will be readable in 100 years, neither the hardware nor the file formats. Accept that as one of the priorities that you have to plan around; plan to refresh the entire archive on new hardware once every 10-12 years, before it becomes obselete enough that nothing can read from it. Since you'll have 50-100 times as much storage space for the same cost by then, what once needed to be compressed can probably be cheaply moved to an uncompressed format that can be readable even if all software and all specs are lost. At the same time newer, higher definition content will be feeding into your archive that will require compression in its place, and you'll need to again carefully consider current standards. However, with steady hardware migration you should also always have an x86 emulation option available to you, and by keeping installation media & isos you will always being able to create a windows/linux media OS that can read what you create now, even when no one remembers them. If you want to archive things now and never transfer them again, that's the way you'll need to go. You'll need to be able to add new virtual systems and new formats for things available down the line not even concievable today, whether it's holographic movies or what.

[smok3]

basically a 100 year method would be celuloid system, film with additional data track, and on that data track you would keep some sort of digital copy as well (that sun paper got me thinking on how wrong they really are...).

[Revgen]

Since you say that filesize isn't as important as future compatibility, retaining quality, and having a single standard, then I'd suggest that you just store them as uncompressed video. I can't see how uncompressed YUV video could be incompatible in 100 years.

[Blue_MiSfit]

And I don't see how anything we have now will be inaccessible in 100 years. As foxyshadis mentions, x86 emulation will surely be possible. I mean look - we can emulate the most OBSCURE computer systems from the 1970s and 1980s today at speeds many times faster than those computers originally operated. As long as we keep the software around (any modern x86 OS and media player - Linux and MPlayer for example) we can do SOMETHING with old legacy video, like pipe out uncompressed to something newer :)

~MiSfit

[foxyshadis]

It's not the software, it's always the hardware. Disk drives fail a lot faster than standards change, but tapes can stick around for decades (they will demagnetize eventually, like some film will slowly disintegrate), long after the tape-loader has broken down and replacements are gone. And even hardware stored pristinely and unused has a high probably of breaking down after a century. Although current tape hardware standards are very backward compatible, major changes in the shape and nature of the technologies are bound to happen to eliminate current hardware standards.

How would you get something off of a 5 1/4" disk today, let alone an 8" or a data cassette (http://stinkygoat.livejournal.com/2006/10/18/) tape? Or one of the many proprietary interface SCSI hard drives from long-gone companies?

So keeping everything on two formats might be a good idea; if one set ends up totally unrecoverable due to unforeseen circumstances, you can hastily make a modern update with the other. But expense is always a factor. (And celluloid prints are very expensive to make and store.)

[zambelli]

How would you get something off of a 5 1/4" disk today, let alone an 8" or a data cassette (http://stinkygoat.livejournal.com/2006/10/18/) tape? Or one of the many proprietary interface SCSI hard drives from long-gone companies?
I agree, and I even think that's a mild example. In 100 years the interfaces and technologies will be so different that reading today's disks will be more like trying to read an 8-inch floppy (http://en.wikipedia.org/wiki/Table_of_8-inch_floppy_formats) or a UNIVAC punch card (http://en.wikipedia.org/wiki/Punch_cards).

[Blue_MiSfit]

Thats true, but the market for popular, standardized data storage formats is SO much bigger these days. I think today's technology will last much longer than we think...

What about just burning archival quality DVD+Rs with whatever format you want (maybe even Video DVD), and re-copying them every 10 years or so? Even old 1970s era VTRs and tape stock is still around (U-Mattic :D), and DVD stuff is so widespread I can't imagine it disappearing anytime soon. And when it does, it couldn't be that hard to convert to a newer format..

Very good points though, Zambelli and Foxyshadis.

~Misfit

[syc0pathic]

What about just burning archival quality DVD+Rs with whatever format you want (maybe even Video DVD), and re-copying them every 10 years or so? Storage space -- a few thousand DVDs take up a lot of room. Not to mention burning time. Updating every 10yrs is a standard part of the archiving process.

[Mug Funky]

hmm. maybe uncompressed is the way to go for long time storage, or at least a lossless compression simple enough to be able to provide on paper along with stable data storage.

unfortunately, though demand is high for data storage technologies, they are extremely short lived - try find a new zip drive, or even a new motherboard with PATA.

however, tape backups are a good idea. you can store large amounts in a smallish amount of space, they're pretty stable, and backup technologies are usually backward compatible (look at DLT for instance - the capacity keeps going up, but each new superDLT drive will read it's lower capacity ancestors). you can keep things uncompressed because you can keep 400GB per tape (currently - it'll increase). also, you can back up offsite thus preventing data loss in case of disaster (there was an observatory in my country that burnt down in bushfires - the main loss was data, though that part didn't get into the news like the loss of a telescope did).

[syc0pathic]

hmm. maybe uncompressed is the way to go for long time storage, or at least a lossless compression simple enough to be able to provide on paper along with stable data storage.

however, tape backups are a good idea. also, you can back up offsite thus preventing data loss in case of disaster.Yeah... remember that I work at 1 of the largest archives in the world. These types of procedures are already in place.

Anyway, we had another meeting this morning. The other people in the group have been doing their own research and they're starting to come around to the idea not only that MPEG-2 is currently the most widely used, but also that MP4 is emerging as the next standard. The thinking right now is that we should use MP4 as our archive standard and MPEG-2 for distribution (becuz everyone will be able to play MPEG-2). As MP4 becomes more standardized and more widely used over the next few yrs, then we can use it both for archiving and for distribution.

So the next step is to write up a report and send it up the chain of command. I'll keep u guys updated... and thanx a lot for all your help. This has been a very informative and interesting discussion:thanks::goodpost:

[masken]

Before plunging into technical orgies, one very important thing to remember here is the archiving demands. For how long should this data be available and how?

If it's for longer periods, I'd recommend that you contact official national authorities first, to see if there's some recommendations or even rules/standards for approved long-time archival video media. Here in .se "The National Archives of Sweden" (Riksarkivet) sets the rules for how archival data should be stored and the allowed formats.

If the archival data should meet such parameters, then you've got no choice but to go with the standards provided, even though there might be more "sexy" solutions.

If there are no national standards, there's most likely ISO and EN/EU-standards available. My recommendation would most definetly be to follow these. Even if it's just for internal use/archiving. If it's for paying customers, you should most definetly use a public standard.