Archive format for DV material ?

[lupus11]

What do you think would be a good compromise between quality and storage space for archiving private uncut "raw" DV video recordings ? I do not want to just store the originals, because they need way too much space on e.g. a DVD. I tried to recode them to MPEG2 at a relatively high bitrate, which gives my a compression ration of about 1:5, would it be better to use MPEG4 (XVID) instead ?
What are you using for your achives ?

Thank You
lupus

[JohnnyMalaria]

What are you using for your achives ?


Simple - I keep the original tapes.

If the material is *that* important - NEVER, NEVER get rid of the originals!

Recordable DVDs seem to be the darling of archiving at the moment. But, to compress DV to MPEG2 and throw away the original DV is foolhardy. To then commit that MPEG2 material to cheap DVD-Rs that will likely degrade within 5 years is even more so. Especially if you are impatient and burn at x16 or whatever.

Keeping the original tapes is actually more cost effective, too, if you want to keep the original format. Plus, in ten years time, it will likely be easier to find a DV player than a computer capable of playing what, by then, will be an old - and likely defunct - storage format (remember floppy discs, Zip drives etc?)

[Blue_MiSfit]

Definitely. Keep two copies if it's important! :) DV tapes if properly stored are pretty good at keeping, but be prepared to accept the occasional dropout after a few years!

If you're not too concerned about keeping pristine original DV for editing, but just want to hold on to it, then encode to Q2 interlaced XviD or something, with some chroma denoising. DV is usually very noisy in the chroma department. It will encode pretty quickly, and should look pretty close to the original.

The only real downside to archiving in an MPEG format is the colorspace conversion. MPEG is 4:2:0 (YV12), and NTSC DV is 4:1:1, so there is some loss inherent. Still, it's not too noticeable unless you're in a professional / broadcast environment.

Keep the tape or a high quality MPEG-4 version :)

~MiSfit

[JoeShrubbery]

And the colourspace conversion issue is only when dealing with NTSC footage, if he's using PAL (he looks to be in Germany, so that's probably the case) then there's no colourspace conversion to worry about as he's already working in 4:2:0.

[swiego]

I'm considering using Windows Media Advanced Profile in interlaced mode. At very high bitrates (around 15mbit/sec) it very nearly seems to encode even noise in the original footage, and it maintains the interlacing. Downsides are editing but I don't think I'd be doing much editing beyond simple splicing so I suspect WMV will work just fine.

[communist]

But whats the point in going from 25MBit/s (that is easily edited without conversion hassles) down to 15MBit/s?
You're not really saving anything I think. I totally agree that the tapes are the best storage - though you're free to do what you like with your footage think about it:
- You capture the tapes (not faster than real time unless you have a lot of money to buy super high end gear)
- then convert them, put them on CD/DVD or keep em on the HDD
- later you decide you need to edit it again, you either edit it in the format you converted or convert it back to DV (because the format is not so easily editable)

Is that little bit of saving really worth all that trouble (for lower quality) and a lot less reliable media?

Last but not least this has been discussed many times before ;):
http://forum.doom9.org/showthread.php?t=92590
http://forum.doom9.org/showthread.php?t=101514
http://forum.doom9.org/showthread.php?t=78853

[juhok]

I'm mad as a cow and I keep the original tapes plus I burn the DV files on DVD's as is (about 2-3 discs per tape). 250 disc box keeps aprox 75 hours of video on DVD5 (Yuden media + PARs). I don't find it too difficult or expensive, but I guess some or most might :)

[aabxx]

I archive the tapes as xvid. The tapes are then reused.

I don't do any processing and use a bitrate of 6 mbit. They turn out excellent. I use twopass encoding and I/P-frames tend to be a good mixture of q2 and q3.

Yes of course they are a little less sharp but I can't really notice that when played back regularly. Going from 6 mbit to 25 mbit offers A LOT of advantages.

- I don't have to buy a lot of tapes. They're not that expensive of course but I just don't like collecting tapes, I've also sold my DVD collection because now I want everything to be more "central" ie. stored on hard drive clusters with backups and everything.
- I can afford to keep THREE backups. One I keep for regular usage, one I hide in storage outside (general backup and in case of fire) and one is encrypted and given to a friend for storage (in case police should bust up the place and confiscate a lot of stuff or something unlikely like that).
- I have four computers in my house, and only one has the videos. But 6 mbit/s is easy to stream over WLAN to the other computers. 25 mbit/s does not work.
- One-click easy access to everything.

I realize and fully accept that I lose a little quality FOREVER. That sounds very dramatic but if you do a comparison you'll realize you're not really losing a lot. Under normal use no one will notice a difference. The reason many people here don't do this is that they tend to be obsessive compulsive about video quality ;) If the loss of quality is so difficult to bear, why not spend an extra $200 on the cam to offset the loss of quality you will get by reencoding? :P

Or if you're so compulsive about quality you don't want to reencode, how can you live with yourself that your camera cost $700 instead of $1300? Imagine how much quality you're losing! :D

Anyway, when I watch these tapes again in 10 years time, the last thing I will worry about is how I lost a small amount of quality. I have tapes 20 years old of bad quality [converted to digital format]... when watching them, the bad quality does not ruin the viewing experience AT ALL. That's the thing with family videos... the content is FAR more important than the technical quality. So a little loss of quality is quite insignificant.

But this is just my opinion of course.

Of course, for editing, DV is great. That doesn't mean xvid or mpeg-2 do not function for editing though... they do work well enough, at least for amateur editing.

[aabxx]

Since I'm bored I'll write some more.

I asked the very same question you ask, you see. The thing is, I could easily have afforded say 8 mbit (to keep in spec) instead of 6. But I wanted to be effective. There exists a point of diminishing returns when it comes to quality. I found out that for my dv's, something above 5 mbit would be enough for great quality. So I went with 6. I tested with 8 as max, but it's really difficult to tell the difference between 6 and 8 mbit on my material. So I really did not see much point in going for 8. Unless I intended to reencode lossy again in the future, but that was out of the question.

My thinking was also that instead of trying to save and backup 25mbit/s videos, which would quicky fill up hard drives, it would be much better in term of content to go to something 6-8 mbit/s. Why? Because then I would allow myself to film A LOT more. Because space becomes less of a issue. 13 gb for one hour is a lot if you intend to film a lot and keep 2 backups. That's 40 gb for an hour, or 120 gb for 3 hours! Now instead, I can have 3 copies of 3 hours in 30 gb! Quite a dramatic difference isn't it!

So what would you prefer, to film in perfect quality (compared to what your cam can deliver) or slightly less quality but have space to film 4 times more in the same space? That was an easy choice for me.

On to codec choice. I really wanted to go with h.264 but I decided against that. They did not look better than xvid at the same bitrate (maybe because of interlaced and noise characteristics of material) and none of them were as mature (obviously), well supported and tested as xvid. Also, they were slower (but that wasn't really a factor in my personal decision but may be for others).

MPEG-2 was a better contender than h.264 because of compatibility but xvid is more advanced and basically compatibility is not a big issue with xvid because of the widespread usage it enjoys and because the source code is available ;)

Hope some of this helps.

[zambelli]

I don't do any processing and use a bitrate of 6 mbit. They turn out excellent. I use twopass encoding and I/P-frames tend to be a good mixture of q2 and q3.
2pass encoding seems like a waste of time when you're using 6Mbps for standard definition encoding. Have you tried just encoding at fixed Q=2? Even 1pass CBR will probably give you the same quality.

Yes of course they are a little less sharp but I can't really notice that when played back regularly. Going from 6 mbit to 25 mbit offers A LOT of advantages.
Have you tried measuring PSNR or SSIM to quantify the quality? I'd expect SSIM to be in very high 90s in order to deem the quality as "archival".

My main concern about XviD archiving of DV would be interlaced encoding and chroma conversion/subsampling. NTSC DV uses 4:1:1, while XviD uses 4:2:0. This means that an intermediate conversion (probably to 4:2:2) needs to happen. If executed incorrectly on interlaced material, this could have a negative effect on final chroma encoding.

Also, if you encode interlaced DV as progressive XviD, the motion estimation turns out very inefficient because it's evaluating field pairs as progressive frames. I'm not sure if this can have a damaging effect on the final encode, but it certainly isn't the most efficient way to encode interlaced content. For this reason I'd rather go with MPEG-2, H.264 or VC-1 - a codec that's more interlace friendly.

[2Bdecided]

Anyway, when I watch these tapes again in 10 years time, the last thing I will worry about is how I lost a small amount of quality. I have tapes 20 years old of bad quality [converted to digital format]... when watching them, the bad quality does not ruin the viewing experience AT ALL. That's the thing with family videos... the content is FAR more important than the technical quality. So a little loss of quality is quite insignificant.


You think? I'm already watching the "home movies" I made last year and wishing that (a) my DV camcorder was better, and (b) wishing it was HD!

It's just like the video footage I shot of my son on a digital stills camera at 320x240 - it's really cute on the PC, but it's such a shame that it looks so rough when converted to 720x576 to include on a DVD.

Today's SD stuff is going to be similarly disappointing in the future.

FWIW, when I look at old photos or films, all my parents stuff is great because it's good, while some other family member's stuff is disappointing because it was so badly shot in the first place, and has been so badly preserved. I don't think Xvid at ~6Mbps comes into that category, but quality does matter. It's sad when all that's left is very bad quality.

You're right though - I should buy a better camcorder and keep the DV files!

Cheers,
David.

[aabxx]

Yes, this is a matter of opinion. But as you also acknowledged, if your clip looks bad with xvid at 6-8 mbit/s, it's not because of xvid (unless you're using very noise material or very high resolution, in which case more bitrate is needed), it's because the source sucked to begin with (most likely due to low level of light). I can visibly see that xvid degrades quality a small amount when comparing picture by picture, however there is no way I am able to see the difference when the footage is in motion. So we're speaking about neglible differences, not the sort of differences that will make you say "wow, this sucks after I encoded it with xvid".

And I've blind tested this on three different types of screens through side by side (half original/half xvid) in-motion comparison. Both me and the wife could not discern.

So I'd say xvid is a great compromise. But it IS a compromise of course.

[lupus11]

@aabxx: Thank you for your comment

You are very close to what I am thinking, 25 MB/sec for just storing the interlaced picture of a mid priced today's handycam seems like an overkill to me, the quality is just not worth it.

I tend to cut and process my videos immediately after shooting (in DV of course). The problem there is that only 20%-40% of the original footage will find its way into the final DVD. However the rest may also be of interest somewhen in the future, but not for sure. So I am not willing to spend GBs and GBs of storage to data I may never use again. This was the inital sense of my question. XVID in the way suggested seems to do quite a good compromise. It will keep a large part of quality at drastically lower storage requirements. I will give it a try.

Thank You
lupus11

[Blue_MiSfit]

Good deal. Just make sure you preserve the interlaced structure!

[swiego]

I'm starting to lean towards just keeping everything in DV. There seems to be a lot of uncertainty and change regarding even the most basic steps away from DV. There are posts in this forum about allegedly big differences in DV decoder quality which is a bit of a surprise to me considering how old DV is. Then the quality of the encoders keep improving. Then there is this massive open question about colorspaces that has me completely lost... I have a fair degree of colorblindness and rely on the computer to preserve color from one medium to another but it seems that all the mismatching between colorspaces and capture formats mucks everything up. It's left me quite confused and uncertain that anything I do from this point on will be done "right." In the time it would take me to develop a confidence level that all the tools (both commercial and open-source) has been debugged enough for someone like me to be assured that quality isn't lost, I could just buy another HD or two and keep everything in DV. Which sadly is what I think I'm going to do....

[PatchWorKs]

The problem is not where to store - best way is original tape (for safety) + external HDDs (for usability) - but what format can be suitable for video editing.

MPEG4 (divx/xvid) are not mutch usable in editing, nor MPEG2.
The most usable in editing seems to be MPEG1, but with its limits.

[lupus11]

Tape is horribly unflexible if you want to copy or access it quickly or as a whole. I personally prefer "double buffered" DVD (2 copies). I am quite convinced that their lifetime depends a lot on the environmental conditions, so if you store them dry, dark and cool and the media is not the cheapest one they will be readable for several years, and if there is another technology leap like we had in the past, you can copy them easily to the new media (like I did with a lot of CDs-> DVDs, of course I kept the CDs additionally)

For editing , if it is only the content of a standard DV cam, I would not hesitate to convert back to DV. If you have high quality content like progressive footage from a professional cam or other source things might look different.

lupus

[DDogg]

I could never disagree with keeping all original source tapes intact for future editing purposes. Saying that, I got to thinking about whether I would even have a working NTSC DV cam in 10 or 20 years and that really worried me. It started me thinking about a 'belt and suspenders' method.

Just for the heck of it, I thought I would experiment with the feasibility of using x264, and then attempt to "re-constitute" the archived files back into DV to see the level of damage. Instead, I first found the archived x264 files seemed to actually look better to my eye than the DV source. Just off the cuff I would think this is something to do with the inherent noise filtering of x264 (?) and perhaps the playback filters.

There was even a nicer surprise when I converted the x264 files back to DV using Celocida and VDubMod. Long story short, I would not hesitate to use the re-constituted source as my editable source.

I've tried this with the latest vfw264 builds with Direct=none, no fast P Skip and Interlaced checked. Encoded at CQuant18. Got about a 6 to 1 reduction in size.

Also, with x264.exe using direct=none, no fast P Skip enabled, interlaced and the M4G Smooth V1 matrix. CQuality 18. Hard pressed to see much difference between the 2 methods.

I know I sleep better after this, because I have been through the mental turmoil of losing, and/or not taking enough footage of my oldest child - she is now in her mid 30's and about to make me a grandpa. I am never going to allow that loss of precious memories again. So, you younger guys and gals take heed. :)

I think this method is worth checking out at the detail level and seeing what you think about the usefulness of the recreated DV source. For me, this belt and suspenders method is the way to go.

[zambelli]

There was even a nicer surprise when I converted the x264 files back to DV using Celocida and VDubMod. Long story short, I would not hesitate to use the re-constituted source as my editable source.
Interesting. Would you mind publishing some PSNR and SSIM scores (using a tool like the MSU Video Quality tool) that compare the re-constituted DV with the source DV?

[DDogg]

Interesting. Would you mind publishing some PSNR and SSIM scores (using a tool like the MSU Video Quality tool) that compare the re-constituted DV with the source DV? Z, I don't do much of that stuff anymore and I also don't much trust the figures from past experiences. Hell, we are talking about funky, shaky home DV shots. I'm thinking the eyeball method is the best for me (never suggested any blanket statement or prognostication whether it would be acceptable to any but me). Saying that, I can't tell the difference (casually) from the two sources in a quick blind test)

Anyway, I would much more trust your results :)

[Chainmax]

I'm mad as a cow and I keep the original tapes plus I burn the DV files on DVD's as is (about 2-3 discs per tape). 250 disc box keeps aprox 75 hours of video on DVD5 (Yuden media + PARs). I don't find it too difficult or expensive, but I guess some or most might :)

I completely agree, that seems to be the best way to go, only I would use HDD storage (300+GB with 16MB cache, or a 10000rpm WD Raptor).