which format to use for storage..

[QQ]

hello,

i haven't been much involved with personal recordings... till now. now after marriage and honey moon, i have 8 hours worth of MiniDV recordings. So far I've ripped them to avi-dv format, which got me close to 150gb. That's about as lossless as it can get so far, but it takes up way too much space of course.

So now I have a dilema - which format and codec to use for archival of these and future recordings? Considering that I will want to watch them, and most likely will want to make some editing which should result in DVDs to hand out for family members.

Am I correct to think about some kind of h.264 incarnation?

I'm open to all the suggestions.

Thanks a lot!

[smok3]

leave them as DV (easy to edit and it is lossy anyway) and get another HD for backup.

[QQ]

not so easy, i would like to store them on home nas for instant playback..

[smok3]

then you would

a. leave originals as DV
and
b. transcode to whatever your LAN clients can read

[QQ]

well i'm planning to have much more recordings, so if each hour takes 15gb i'll run out of space quite soon anyway. one option is just leaving them on minidv tapes..

[fibbingbear]

If you do use H.264, you could try to use a CRF that is really low, like CRF2. The file size will be big, though. For some reason CRF2-CRF10 are about the same size when I encoded some files with those settings...

I have no idea how efficient DV is, but you might want to consider MPEG2. At high bitrates, H264 won't be terribly better than other formats (although it's still a great algorithm), and MPEG2 has the nice side-effect that you can burn DVDs for friends/relatives without having to go through another conversion step.

If you're not going to keep it as lossless (or near lossless, I think DV is a lossy format), just remember that each time you convert you're going to lose some quality.

If you do want to store it in an intermediary, lossy format, the best advice I can give is "pick as big [in file size] as possible."

I did some experiments a while back on constantly re-encoded mp3/aac files, trying to quantify how many times you could re-encode them under a lossy format and still have the material be indistinguishable from the original. If you set the bit rate medium (~190kbps, which is at the threshold of transparency/indistinguishability), you can re-encode it a few times before the artifacts start to stand out. If you set it insanely high (~320-400kbps) it can survive 10+ re-encodings before the artifacts start to come out.

[J_Darnley]

If you do use H.264, you could try to use a CRF that is really low, like CRF2. The file size will be big, though. For some reason CRF2-CRF10 are about the same size when I encoded some files with those settings...
Using crf modes with values from 1-10 without changing the default value of qpmin will get you an encode where most qps are 10. This means that the encode will be the same size. At 10 and slightly lower, B-frames can be above 10 due to pbratio.

[smok3]

well i'm planning to have much more recordings, so if each hour takes 15gb i'll run out of space quite soon anyway. one option is just leaving them on minidv tapes..
tapes are good plan, so when you edit a part of the material, export that as DV or huff, depending on what color corrections and stuff like that was made, backup project and backup the edit, nuke the material that is no longer needed, then you can transcode the edit to the format that is popular at that time (and your lan clients friendy). In case edit would need some reshapening you can either;

a. go back to your original project and recapture the tapes needed (yes, have some sort of tape numbering in place)
b. reedit the edit

[*.mp4 guy]

I would definately say keeping the original, unrecompressed footage, is a must for proper archiving, but baring that you have a few options.

1- Recompress to a high efficiency lossless format, lagarith, and ffv1 can often outcompress dv, and both are true lossless formats.

2- recompress to an absurdly high quality lossy format, with no vbv or bitrate restrictions (true vbr), an example of this would be Xvid with the fox home entertainment matrix, at cq2, no chroma optimizer, msp6, vhq4, chroma motion enabled, no trellis, no b-frames, no qpel. That will survive several genrations without typical mpeg artifacts, but as you increase the number of reencodes, you will notice low level noise being introduced to the video, that looks similar to film grain, after about 5-10 generations low level blocking will beging to be noticible, though even after 10 generations it would still be relatively easy to remove the artifatcs withought damaging the video too badly, since the artifatcs remain almost completely temporally decorelated (even the blocking).

3- recompress to a high quality lossy format, but realise you will be limited to 1 or maybe 2 reencodes before artifacts start to become a serious issue. An example of this would be X264 with aq, and crf 16-20. To increase the longevity of the video, it is helpful to disable options that can cause systemic artifacts; enable no fast p-skip, enable no dct decimate, diable trellis, don't use the prestige cqm, set b-frame mode to none, even though you loose efficiency by doing this, you also avoid the vast majority of the "unfixable" artifacts, such as temporal smearing, and anomolous blocking, at low q values, the remaining artifacts should be distributed consistently over the video, and be relativly easily removed via high quality denoising.

Option 1 provides aproximately 1 to 2 times more compression then dv, while maintaining frame accurate seaking and perfect quality.

option 2 provides between aproximately 1 and 4 times more compression then option 1, while providing "good enough for all practical purposes" quality, frame accurate seaking is possible, but slow, and perhaps, difficult to set up.

option 3 provides aproximately 2 to 8 times more compression then option 2, with "good enough most of the time" quality, frame accurate seaking is possible, but difficult to setup, and very slow.

[fibbingbear]

2- recompress to an absurdly high quality lossy format, with no vbv or bitrate restrictions (true vbr), an example of this would be Xvid with the fox home entertainment matrix, at cq2, no chroma optimizer, msp6, vhq4, chroma motion enabled, no trellis, no b-frames, no qpel.

Could you provide an example with h264?

[*.mp4 guy]

It would be a bad idea to use h.264 for option 2, there are still ocaisonal artifacts that are hard to avoid (flickering, mostly) and there is overall almost no gain in efficiency over a good mpeg4-asp implementation when such high bitrates are used, mpeg4-avc is getting better and better and, I have no doubt that at some point, it will be worth using in these situations, but that time isn't now.

There are some othe issues at play here aswell, Xvid is probably the most consistent codec at high bitrates, I wouldn't recomend just any mpeg4-asp implementation. Xvid just doesn't ever screw up at high data rates, it never runs into anything that trips it up, fades, gradient fades, flat noisy areas, nothing will ever cause it to introduce a situation specific artifact; its more reliable then even dv, and miles ahead of all but the absolute best mpeg2 encoders, and even they screw up, while I havent yet seen Xvid make a mistake I could see.

[Dark Shikari]

It would be a bad idea to use h.264 for option 2, there are still ocaisonal artifacts that are hard to avoid (flickering, mostly)So its just FUD, right? :rolleyes:

Xvid has loads of "occasional artifacts" (smudging, blurring, blocking, ringing) that are hard to avoid, but they're not a problem are they?

[*.mp4 guy]

So its just FUD, right? :rolleyes:
It would only be fud if I was trying to instill Fear Uncertainty and Doubt. Which I am not. I would also have to be lying, which I also am not.

Xvid has loads of "occasional artifacts" (smudging, blurring, blocking, ringing) that are hard to avoid, but they're not a problem are they?

You've got it completely backwards, smudging, blurring blocking and ringing are a natural result of mpeg style compression and are present roughly in proportion to the quantizer used, they are not "occasional artifacts", because they are always present in every encode by every encoder always. X264 has them aswell, because it is a lossy mpeg style encoder. They are a problem, a problem that is solved by a very high bitrate.

The artifacts I am talking about happen only sometimes, hence occasional. There have been many complaints about X264 producing flickering blocks. I discussed it a while ago with someone, but I can't post it here, as it was done through pm's.

They provided some links to two different papers that explained some of the issue, I think its fine if I repost the links.
link1 (http://www.ece.ucdavis.edu/PCS2004/pdf/ID20New.pdf)
link2 (http://ftp3.itu.ch/av-arch/jvt-site/2002_10_Geneva/JVT-E070.doc)

To the best of my knowledge, this and certain other issues related to motion compensation that lead X264 to cause flickering have not yet been completely addressed, though aq certainly helps.

[Dark Shikari]

To the best of my knowledge, this and certain other issues related to motion compensation that lead X264 to cause flickering have not yet been completely addressed, though aq certainly helps.Yes, the flickering is a problem caused by a lack of DC coefficient precision; I've seen such papers before.

However, at any quantizer lower than 16 for 4x4 DCT and (I think?) 22 for 8x8dct, the DC coefficient is more precise than 1 "step" of luminance level. As such, by inherent definition, any DC luminance value can be coded exactly at such a quantizer. Therefore, flickering cannot occur at low quantizers.

[*.mp4 guy]

Yes, the flickering is a problem caused by a lack of DC coefficient precision; I've seen such papers before.

However, at any quantizer lower than 16 for 4x4 DCT and (I think?) 22 for 8x8dct, the DC coefficient is more precise than 1 "step" of luminance level. As such, by inherent definition, any DC luminance value can be coded exactly at such a quantizer. Therefore, flickering cannot occur at low quantizers.

I have personally seen the issue crop up in encodes with average q values as low as 12 (with the flat matrix). To be entirely honest, I have not done such high bitrate encodes with X264 in quite a long time, but I have not seen any real change in the amount of flickering at lower bitrates either, it has been reduced a bit, true, but it is still a problem, and has never been directly confronted. I have assumed that because things have remained for the most part the same at lower bitrates, they are, for the most part the same at higher bit rates aswell.

[Blue_MiSfit]

I would just buy a couple hard drives, and store the DV files. Make a separate encode to MPEG-2 or another lossy format if you need to stream them (and for some reason your media player won't handle DV).

Isn't it worth the cost? Hard drives are so cheap now - why delete the originals?

I've never seen a lossless codec outperform DV, but I haven't ever tried :)

Seems a little fishy to me - espeically if it's consumer grade DV we're talking about. Hostile swarms of chroma noise, camera shake, chromatic aberration from cheap lenses, low dynamic range with blown highlights and crushed, extra-noisy shadows don't compress well :D

But I'll give it a try sometime if I ever have the (mis)fortune of working with DV in the future. I think a RED Scarlet will be my next baby...

~MiSfit

[benwaggoner]

well i'm planning to have much more recordings, so if each hour takes 15gb i'll run out of space quite soon anyway. one option is just leaving them on minidv tapes..
Really? What does a 1 TB drive cost now?

How much cheaper will your second TB be by the time you fill up the first one?

Transcoding and dealing with that transcode is almost certainly more expensive than getting a minimum wage job and just buying more storage :). Maybe I'm just bitter as I spent $5000 (plus $1000 for the RAID controller card) for a 4 GB video RAID once upon a time :).

And remember, these are your honeymoon videos. Any data that gets lost in recompression is gone forever.

[fibbingbear]

benwaggoner, you raise many good points, but I think it is worth pointing out that data storage is cheap, reliable data storage is not. Even if you go with redundant drives, you still have problems, such as the fact that failures are not independent in practice (USENIX paper about this, don't recall which one), and you also have the problem that all the data is still centralized (a single disaster like a fire or theft and it's all gone). Granted, these are low probability events, so it'll depend on where you draw the cutoff and what risk is acceptable to you.

If you want very reliable data, the cost/megabyte starts to get pretty expensive. For example, if you want high reliability, one should have a backup off-site. Assuming you don't want the hassle of maintaining your own (which will involve off-site space, time investment, and the need to detect when older copies of the material have become invalidated and should be replaced), you will probably go with a pay service. Assuming you use Amazon's S3 (which is decently cheap as far as these services go), a 150gb video will cost $180 a year to store, whereas a high quality encode may be 15gb and only $18 a year to store. And this is per video :-)

[Blue_MiSfit]

@benwaggoner

Well said.

http://www.newegg.com/Product/Product.aspx?Item=N82E16822152102

Samsung Spinpoint F1 - 1TB capacity - $175

1TB = ~60+ hours of DV (or HDV for that matter)

Seems like the only way to go...

@fibbingbear
Sure, reliable data storage isn't cheap. That's not the issue at hand though... data reliability applies to any type of data. You can get as paranoid as you want - but the issue at hand is "what should I do with these giant files?"

My suggestion is to just transfer them to the terabyte drive, keep it around, and toss the master tapes in a fireproof box along with your other super-important stuff :) Heck - put the hard drive in there too, along with your other super-important data. 1TB is a lot of space. Get two, and keep two copies if you want.

Also, your argument about CDs, MP3s, and FLACs is correct, but not really applicable... Sure, lots of people ripped their CDs to MP3, but how many of those people threw their CD libraries away? ;) It's nice to have, just in case.

~MiSfit

[smok3]

Transcoding and dealing with that transcode is almost certainly more expensive than getting a minimum wage job and just buying more storage

right on the spot, (and i have learned that the ugly way, from my own mistakes).

[fibbingbear]

@Blue_MiSfit:

Fair enough, and I agree that what I suggested is overkill for most people. Just trying to point out that the argument "hard drives are cheap" doesn't apply to every situation. I used to work for a data backup company and there were horror stories of our customers (before they became customers) who would run their own tape backup at a fraction of the price... only to find out their backup had become corrupted when they needed it (hard drive reliability isn't perfect either). One of the services we provided was constant checking and checksumming, so we could detect corruption and deal with it before an emergency.

What's the probability something like this will happen? It's low. There's a reason why our customers were businesses and not individuals --- they needed extremely high reliability and were willing to pay a high cost for it.

And the CD/FLAC argument wasn't 100% great, so I edited it out :-p

[QQ]

you guys are amazing!

i've completely forgot the issue and got burried in different stuff, but wow, do i have a lot of suggestions!

i'll try to read through them all very carefully, since I still need to decide what to use :)

thanks again!

[hoboX10]

You can burn them to DVD's then convert them to real media for storage on your computer. That should keep the file size low.

[zambelli]

1- Recompress to a high efficiency lossless format, lagarith, and ffv1 can often outcompress dv, and both are true lossless formats.
Huh? There's no way Lagarith or FFV1 could possibly compress non-synthetic video better than DV25 (MiniDV). DV25, as the name suggests, is always recorded at 25 Mbps. Getting 4:2:0 video compressed losslessly at a consistent 25 Mbps is not possible with FFV1 or Lagarith.

[*.mp4 guy]

Huh? There's no way Lagarith or FFV1 could possibly compress non-synthetic video better than DV25 (MiniDV). DV25, as the name suggests, is always recorded at 25 Mbps. Getting 4:2:0 video compressed losslessly at a consistent 25 Mbps is not possible with FFV1 or Lagarith.

Surprise surprise, vbr encoding doesn't always end up at the same bitrate as cbr. However, let me assure you, lagarith and ffv1 often outcompress ntsc dv25. Though, to be a little more clear, I am assuming that you have tv levels video, and that you compress to lossless in yv12 (yv12 is the only possible destination format colorspace anyway). Though the tests I did might have been on 24P material, which could account for a slight advantage to dv if my results were aplied to 29.97 fps material. The point is, purely lossless codecs have high enough compression ratios as to make dv compretely pointless.

[zambelli]

Though the tests I did might have been on 24P material, which could account for a slight advantage to dv if my results were aplied to 29.97 fps material.
That would absolutely make a difference. Progressive video is easier to compress with a lossless codec than interlaced video is. Also, note that DV is encoded interlaced even when it's shot in 24P mode - the telecine is a hard pulldown because DV doesn't support TFF/RFF flags. What were you comparing then anyway? 30i DV with 24p FFV1? That doesn't seem right.

The point is, purely lossless codecs have high enough compression ratios as to make dv compretely pointless.
And I'll start believing you the moment you prove you're right and I'm wrong. :)

I ran tests on a number of different DV sources. I intentionally picked PAL DV so that 4:2:0 would be preserved throughout the conversion. What I found was that FFV1 was in average 1.5x-2.0x the size of its DV source. If you're trying to archive your DV footage, what exactly would be the point of transcoding DV25 to a format twice its size?

[*.mp4 guy]

That would absolutely make a difference. Progressive video is easier to compress with a lossless codec than interlaced video is. Also, note that DV is encoded interlaced even when it's shot in 24P mode - the telecine is a hard pulldown because DV doesn't support TFF/RFF flags. What were you comparing then anyway? 30i DV with 24p FFV1? That doesn't seem right.


And I'll start believing you the moment you prove you're right and I'm wrong. :)

I ran tests on a number of different DV sources. I intentionally picked PAL DV so that 4:2:0 would be preserved throughout the conversion. What I found was that FFV1 was in average 1.5x-2.0x the size of its DV source. If you're trying to archive your DV footage, what exactly would be the point of transcoding DV25 to a format twice its size?

I suggested keeping the original tape if it was in any way possible. What I tested was 30P from an old jvc dv camera, aswell as 24p from a panasonic dvx100 more recently (I didn't remove the 3:2 before reencoding, IIRC) vs. the same encoded as lossless. Same number of pixels/s to dv vs lossless, imo thats fair.

The JVC was ages ago (I forgot it used 30p instead of 24p like the dvx, in my last post), and it had really abysmal video quality, lots of visible dv compression artifacts aswell. The panasonic of course had amazing video quality, nearly perfect looking really. I could see how the panasonic might do a lot better then is realistic when compressing to lossless, but I don't see how that could have happened with the jvc.

It was just a suggestion, obviously if you don't need to reencode the video file and lossless increases its size (which can always happen) You should just leave it alone; that said, reencoding dv to lossless has worked consistently well for me which is the only reason I recomended it. If you process and edit your video at all the advantages of lossless both for compression ratio, and for quality start to increase. Especially if you do anything with the color channels.