Project Gutenberg seeking some video archiving advice

[jhutch]

Greetings!

Project Gutenberg (http://www.gutenberg.net/) is tentatively looking into expanding their collection into old public domain films and shorts. As you can imagine, this is a major shift from the text based archives we are used to dealing with.

My reason for posting here is to get some "expert" opinions of "best practices" to look into and what things should be avoided.

One of the biggest discussions right now is what should be the "master" format and what (if any) other formats should be offered.

Now, to give my thinking on this as a starting point.

1) Most of these will be captured from analog sources, most likely (VHS tape quality). Plus, these are OLD films (pre-1923) so the quality isn't great to begin with. I'm thinking a capture with a lossless codec (perhaps Huffy?), then slight preprocessing (deinterlacing, denoise?). This will result in a HUGE file, obviously, and completely unsuitable for Internet archiving.

2) My personal experience lies with XVID mostly, so that is the format I'm most comfortable with. I'd imagine that creating a version in MPEG2 is a good idea. What all versions should we archive in? DIVX? Windows Media? Real? Quicktime? Which formats (if any) should be avoided?

The different versions will be created, most likely, from the original lossless compressed video, but after all the other formats are created, the lossless will probably be discard, just because of space/bandwidth reasons. ie, I can't afford to store terabytes of raw video and my poor DSL would rise up in revolt if I tried to upload that somewhere else, assuming we could find that kind of storage space somewhere. But if anyone has any good ideas for addressing this, I'd greatly appreciate the advice.

Please, rip apart my assumptions above, and give us a good solid footing to embark on. I think this could be a valuable/exciting addition to PG and the Internet as a whole, but I want to see it done well, not haphazardly.

Thanks in advance!

JHutch

[fccHandler]

Your questions are similar to asking "what's best," which is forbidden here. But even if the thread doesn't get closed, every response you get is bound to be highly opinionated (including mine). ;)

Over the course of several years I've tried just about every popular codec around. Ultimately your choice for long-term backup is going to depend on which of two things is more important to you: video quality, or storage capacity.

100% quality is easily achieved using uncompressed video, or lossless Huffyuv compression, but the needed storage is immense. However, the moment you descend to a lossy compression method, you enter a realm in which the results can only be judged subjectively. Every codec has its own peculiar advantages and faults. All lossy codecs produce video artifacts. Some artifacts may appear pleasing to one person, while being intolerable to another person.

My personal advice is to stick with MPEG-1 (for progressive content) or MPEG-2 (for interlaced content). IMHO the quality can't be beat by any other lossy compression technique. Furthermore, they are international standards which are still used today in many, many applications (VCD, SVCD, DVD, DVB, HDTV, and so on). They are old standards, but they have withstood the test of time. The specs are firmly written in stone. I suspect the MPEG-1 and -2 formats will remain viable even after we have become senile old men.

Anyway, there's my 2 cents. YMMV.


EDIT: One more thought...

Consider that Doom9's site, his codec tests, and forums content are overwhelmingly dedicated to converting MPEG-2 into some other (more lossy) format. In other words, I believe it's fair to say here that MPEG-2 is the true "original" against which all other "replicas" are judged. Given that perspective, why settle for less than the best?

[Arachnotron]

I agree with FCChandlers comments. You might also want to consider the codecs being considered for the new DVD successor standard (don't know which they are), but only if this standard has been finalized by now.

I do have another point though:

1) Most of these will be captured from analog sources, most likely (VHS tape quality). Plus, these are OLD films (pre-1923) so the quality isn't great to begin with. I'm thinking a capture with a lossless codec (perhaps Huffy?), then slight preprocessing (deinterlacing, denoise?). This will result in a HUGE file, obviously, and completely unsuitable for Internet archiving.

If I understand you correctly you are dealing with old movies that have been transferred to tape. So the original material was progresssive in nature. I would be carefull in deinterlacing that, since doing this may introduce extra artefacts and will destroy any chance of recovering the original frames.

You migh want to consider IVTC-ing before archiving instead to reconstruct the original frames. However, old silent movies can have all sorts of framerates, so what you have on tape will vary for each film (even apart from what was done to create the extra frames during the tape transfer). You would have to use different procedures for each film, making it very labour intensive and even then it may be impossible for some recordings.

As a compromise, you could also archive the captures as high resolution interlaced MPEG-2's, and only deinterlace or IVTC the versions you use for streaming. Basically, you stay as close as possible to what is on the tape. This leaves you the option to try to IVTC later.

[jhutch]

Does anyone see a downside to using MPEG2 as the master format? In my experience, a sufficiently high rate MEPG2 file is virtually indistinguishable from the lossless video, but I've also got a notoriously "forgiving" eye for artifacts compared to some of the folks around here.

As far as MPEG1, that one surprises me. The few times I've played with it, even at relatively high bit rates, the quality is noticably below just about everything else. Granted, it would be a good idea to have a copy in that format for VCD players, but as a master format? I don't know.

Getting away from codec advocacy, though, how about capture methods? Is there a method out there for capturing directly from film to digital that could be employed by a dedicated volunteer (relatively low cost is the biggest factor). I've seen home-made projection boxes that people have used to transfer their old home movie tapes to their camcorder, but that quality is usually far below what I'd expect a archive like this to contain. Ultimately, my thought process is that capturing from studio made VHS tapes is going to provide better quality than any film to digital method I could put together in my basement, but I may be making false assumptions here.

Thanks for the replies,
JHutch

[fccHandler]

Originally posted by jhutch
Does anyone see a downside to using MPEG2 as the master format?
In general, it costs money to use it. Maybe this wouldn't apply in your case, since you are not seeking profit and you're doing it for posterity. But just to be safe you should check with the MPEG LA (http://www.mpegla.com/) before you begin your project.

I don't think MPEG-1 is licensed the way MPEG-2 is, or at least I've never heard of any legal issues with the use of MPEG-1. Myself, I haven't seen much quality difference between 1 and 2, other than the fact that MPEG-1 is unsuitable for interlaced video (like VHS).

As far as capturing goes, I know the professional companies use big expensive rigs ("telecine") which scan each frame line by line. The scans are then mastered onto fat digital tapes (DigiBeta, for instance) before being authored to DVD. I know of at least one company here in Georgia called CinePost (http://www.posthouse.com/) which transfers films to digital tape. Their facilities are excellent (I've seen them). I don't know what their prices are like. Of course, all this assumes that you have the actual films in your possession.

If you must capture from VHS, any modern PC with a capture card and a big hard drive can do that job.

[Arachnotron]

Ultimately, my thought process is that capturing from studio made VHS tapes is going to provide better quality than any film to digital method I could put together in my basement, but I may be making false assumptions here.

I don't know anything about film transfers, but do keep in mind that a modern digital camera will deliver more quality and detail than VHS tape can. So while professional telecine equipment undoubtedly will give better quality compared to an home-made setup, the transfer to VHS tape will loose part of that advantage again. Which step is the more important I cannot say. VHS is a very poor carrier for video.

[Soulhunter]

Originally posted by fccHandler
My personal advice is to stick with MPEG-1 (for progressive content) or MPEG-2 (for interlaced content). IMHO the quality can't be beat by any other lossy compression technique.Very boldy statement I think !!!

So let me add my own one... ;)

IMHO XviD is able to reach exact the same quality as MPEG2 but @ 25% smaller filesize !!!


Bye

[fccHandler]

Originally posted by Soulhunter
IMHO XviD is able to reach exact the same quality as MPEG2 but @ 25% smaller filesize !!!
Only in your dreams !!! :devil:

Anyway, I did say my response was highly opinionated. :D