archiving DV

[gvm]

May I have your advice please.

I routinely digitise analog video as an AVI file before editing it and burning to DVD. However, because video editing software is continually improving (and DVD formats etc too I suppose) I want to archive my source video in a format that I can re-use sometime in the future.

What is the best format to do this please? A 2hr AVI file is 27GB so there are serious practical problems storing such a large file. A DVD quality MPEG-2 file takes about 5GB which I can almost squeeze onto a DVD but is there any point ... maybe there's not advantage in doing this because I've burned a DVD already anyway? Are there other options?

thanks for any suggestions!

[mustardman]

Purely from a personal perspective, I am moving my material from VHS, digitising and repairing glitches caused by bad sync signals (from the original VHS). Encoding to an AVI type 2 DV file. I split these into convinient sizes (usually scene or sequence boundaries) and burn to DVD-R.

I chose DV over MPEG-2 because it gives better quality and is intra-frame based. I would have preferred MJPEG as it has more options and you can get even better quality out of it, but also seems to be much less of a standard format. At least DV is natively supported in nearly all consumer digital video cameras.

I would not use DivX or XviD, or any other similar codecs. Don't be too concerned about the size, a DVD-R is less than $1 per Gig!

[bb]

Originally posted by mustardman
[...]I would have preferred MJPEG as it has more options and you can get even better quality out of it[...]
You can use many more resolutions, yes. But better quality? DV is very similar to MJPEG, but makes use of some advanced techniques. Thus DV should be superior over MJPEG in terms of quality, if you compare both formats at the same bitrate. Of course you can use higher bitrates for MJPEG than for DV to obtain better quality, but then you'll get huge files, too.

Read this: http://www.adamwilt.com/DV-FAQ-editing.html#transcoding

It seems to be generally accepted that M-JPEG compression at 3:1 is roughly equivalent in quality to DV's 5:1 compression.

bb

[mustardman]

@bb
Yes, sorry - that is what I was getting at. I used to use MJPEG at 6MBytes/sec, full PAL res, with excellent results. However, the bitrate is higher than DV, so I suppose better quality is to be expected!

From some research into DV, the claim is that it is able to 'detect too much interlacing, and switch the coder from frame based to field based'. From things I have seen, some hardware coders do a terrible job of this. From what I saw, I wonder if this feature was in fact even implemented!

[ie: testing by separating the fields and examining them]

The MJPEG codec I was using certainly could do this, as even very large inter-field differences were correctly coded, and looked quite good. Perhaps it was coding on a field basis all the time... I don't know enough about the internals of MJPEG.

[gvm]

Mustardman,

thanks for your reply. I really apppreciate it. :)

May I ask two followup questions please:
1) why did you choose AVI type 2? Compatability reasons?
2) which codec do you recommend for producing type 2? Panasonic DV?

[mustardman]

AVI type 2:

Basically an AVI is a container for what is supposed to be Audio and Video. An AVI can contain only audio, or only video, or multiple streams of each (although I have never seen one, I beleive it is possible).

Type 1 DV is video that is multiplexed with audio. However, an AVI file requires separate video and audio, hence type 2 DV is simply the video (with multiplexed audio still in it) with that audio extracted and put into a separate stream.

I suppose it would be possible to have the audio that is multiplexed with the video being completely different from the "broken out" audio stream - but why? And would anything actually make use of it?

DV codec:

I use the Panasonic DV codec (VFW). Much reading suggested that it had the best multi-generation capability. It also handled the image dynamic range properly (which not all codecs did).

I use WinDV for IEE1394 capture straight to type 2. I use VirtualDub for most of my editing, which creates type 2 files - AFAIK the panasonic DV codec will create a DV compressed frame with no audio. Whether that field in the data stream is missing, or padded, I don't know.

[hendrix]

type 1 avi has a 4gig file size limit regardless of FAT32 or NTFS, type 2 is unlimited

[theReal]

IMO, the best way to store DV video is a MiniDV tape. After editing play it back on MiniDV tape without any conversion or quality loss, you'll get 13GB on one small tape and you can convert it to anything you want in the future.

You could of course split the DV avi and burn it to DVD-R, but I'm not sure what will last longer: a tape, or a DVD-R... (not to speak of that the tape is most probably less expensive than 3-4 quality DVD-R's)

[mustardman]

Conversion & quality : There is of course no quality loss - it is a data stream just recorded straight to the tape. I am concerned about dropouts though. With a new tape and new camcorder, I generally get about 1 dropout per tape, lasting about quarter to half a second (even with expensive tapes, and only ever using them once). The sound was OK, but the picture became blocky and staggered before resuming. The tape has no way of re-reading quickly (like a disc could). Even though it may have error detection and error correction, the lack of ability to retry real time is an issue.
Only half a second out of an hour, that's pretty good. But - so far, touch wood - my data disks have given me zero faults.

Splitting DV files : Yeah, this is a real pain in the proverbial. And when a sequence is bigger than a disk, then it requires disks in multiples of 4.5GB (say a file was 5GB = 4GB wasted space!). I don't like doing it, but it is easy access for later NL editing!

Cost : I buy a tape for about $10 to $15. A DVD-R costs about $3. The price is fairly equivilant, but it is a big drag on the time, and the wasted disk space (eg above).

Longetivity : I personally think that tapes will last longer, and degrade progressively. I think disks will not last as long, and will suddenly "crash" when the error correction threshold gets exceeded. Certainly mis-care of either media will result in early and untimely death, so I'm pretty careful about temperature & humidity. I also think that analog tape will degrade more gracefully than digital tape - ie: a small 'analog' error will cause a slightly blurry/noisy picture. A small 'digital' error may cause complete loss of the picture.

Space : Disks do take up more space than DV tapes. Disks are certainly more robust to magnetic fields, but not as good for temperature.

All in all, I would agree that tapes are better.

I decided to go the DVD-R route, mostly because of my finnicky desires for no dropouts and easy access for later NLE.

[theReal]

You're right about the dropouts - you get them more often with small camcorders and MiniDV tapes. A (rather expensive) workaround would be to buy a seperate DVCAM recorder (~2000$) and record your stuff to DVCAM tapes (184min tape for roughly 40$, I think). On these tapes and recorders, you'll get about one dropout in ten years :)

[gvm]

This discussion has been really useful, thank you.

I use Nero ROM burner. When I try to burn the AVI file to a Data DVD, I get message saying file is limited to 2GB and to compile using a different format in order to achieve larger file sizes (presumably up to 4.7GB?). Can you tell me how I do this using Nero please?

[theReal]

Choose DVD-ROM (UDF) instead of DVD-ROM (ISO) and it should work (haven't tried it myself, but theoretically UDF should be able to do it)

[gvm]

I don't have access to a DV tape recorder or camera so I archived DV as a 4GB type 2 AVI file using UDF compilation (Mode1) successfully tonight. This amounted to 18 minutes of video. Like you said Mustardman, the process is laborious but it worked.

Fyi, the following URL describes how to use Nero to create a UDF compilation. http://www.nero.com/en/631939469946227.html

[mustardman]

Yeah - very time consuming, but excellent results! I am in the process of converting old VHS camcorder material to DV, then archiving that. Once that is done, I am then going to pick out the 'good bits' and NLE them. A good thing about going through them all first is that I will know exactly what I've got.

To save the stuff to DVD-R (as DV-type2 data) I split the files manually where there are scene/subject/sequence changes. Hence, many small but related clips all go into the same file. A long sequence I try to pick a good place to cut - generally where there is little movement.

I try to stick to a maximum of 1.5GB (10,000 frames) but it does go up to about 1.8GB, and the smallest I've needed to do so far is about 200MB. I then combine these onto several DVD-R, working with a total size of 4.5GB per disk. A three hour tape breaks down to about 7 DVD-Rs.

It had the advantage that I would never have to exceed 2GB, which may be problematic. If I stick to 1.5GB per file, then 3 files would fit nicely onto a DVD-R with some room left over for indexes and other stuff (codecs/players/still images/etc).

I use VirtualDub to split the files and WinDV for both capture and playback. (I found a problem with WinDV and multi-segment file playback - message me if you are interested).

[theReal]

I am in the process of converting old VHS camcorder material to DV, then archiving that. Once that is done, I am then going to pick out the 'good bits' and NLE them. A good thing about going through them all first is that I will know exactly what I've got.

If you don't want to keep all the sources, but only the good parts, why don't you watch it on VHS, make some notes where you find what, then play some of it in, NLE it, then delete the sources and go on likewise. That way you wouldn't have to archive everything on DVDs first (unless you want to archive everything on DVD, of course)

[theReal]

Fyi, the following URL describes how to use Nero to create a UDF compilation. http://www.nero.com/en/631939469946227.html
On reading that I thought there was something completely different in this process :D :D

For those of you who know how to burn ISO CDs/DVDs with Nero - you don't need to read this 6-page-long pdf - all that's different is that you choose UDF instead of ISO at the beginning ;)

[mustardman]

You have a good point there, although I must say that a good deal of my VHS camcorder stuff is interesting, even if only slightly!

But I am cutting out segments that are just tripod-ed shots with nothing happening, or blank (as in horribly underexposed - essentially black) bits.

I also have a list of what is on each VHS tape, although it is a good refresher to review how the one line summary sums up a scene.

I would like to say more, but I notice you have PM disabled?

[holdar]

Hi, this may be overkill but here goes.

I archive DV captures to DV tape. This seemed the simplest way because I capture using either a Sony DV camera, or Canopus ADVC-100. Of course to archive you must have a camera or a DV deck of some sort. Good DVcams are now cheap ($500-$700).

My Setup:
Hardware: JVC HR-9800u (wTBC), Canopus ADVC-100, Sony TRV10, Panasonic miniDV LinearPlus 80/120,
Software: Scenalyzer (capture program), VirtualdubMod (preview), Avisynth (DV cleanup and telecine), Premiere (titles, slideshow(Ken Burns),sound), Tmpgenc (encode Mpeg2)

Suggestions:
Use only one brand of miniDV tape, apparently brands use different lubricants, mixing them can clog heads and reduce camera life (Sony Tech Support). Same number of bits are written to tape per frame in LP and SP mode. That means no reduction in quality in LP. However a tape dropout means more data loss (Sony Tech Suuport).
I use LP for VHS capture, and SP for live camcorder. If you start a tape in LP mode always use it in LP mode, same with SP. Only problem I’ve had was using a tape for stills, then trying to record DV over it, did not work. Buy the best tape you can afford, I use Panasonic LinearPlus 80/120 ($6.00 ea. amazon.com).
I first write timecode to the entire tape before recording on it. This means an extra pass over the tape. Some sites recommend writing Black to Tape (premiere timeline), I usually hit VTR record with no input, I cant tell the difference but don’t know how to measure it anyway. Have not had noticeable dropouts, if you mean scrambled frames with color blocks. My max_tape_overwrite=3. This allows for timecode, initial capture, and archive w filename.
Scenalyzer allows DV.avi’s to be written with filename and will generate a tape index, this is invaluable for archival. Also you can use Scenealyzer to create “thumbnails” of DV frames as an id aid. Right now my tape catalogue is a word table with thumbnail, file name, and tape name. What I need is a “mass store frontend” to automate everything and retrieve files on demand. Any ideas?