Encoding DV for backup

[mattsmith321]

Hi all!

I've avoided writing this post for a while, but I've kind of reached a point where I need a push in the right direction. This post is actually very similar to the recent Max Quality & Archival Purposes (http://forum.doom9.org/showthread.php?t=134072) thread in the DivX forum but I wanted to give my specific requirements and not hijack that thread.

I've got ~25 mini DV tapes worth of family home video (I know, boring!) that I've recorded over the past ~10 years that I would like to:

Get off of tape and onto my computer in a format that

Works with my backup strategy - I have my primary hard drive in the computer that I sync with two external hard drives that I rotate on/off site every couple of weeks.
Is easily shareable - I'd like to be able to burn the files to a DVD (not necessarily make a DVD) or be able to take one of my external backup drives to my extended family and copy locally.
Can be encoded to other formats - Every now and then I'll have a clip that I either want to upload to YouTube or encode for my iPod.
Works in consumer video editing software - Every now and then I'll go the extra mile to actually edit the clips into a video with a soundtrack, etc.



So, the first part is easy. I use WinDV to get the video from the tape and onto my computer into individual timestamped clips based on scene change/discontinuity threshholds. What I end up with is ~12GB of data and ~30 - 60 files (per hour / tape). Doing the math for ~25 tapes, we end up with ~300GB of data and 750 - 1500 files. While I do agree with Ron Spencer that keeping the data in DV format is ideal and hard drives are fairly cheap, it just isn't practical to replicate 300GB of video + 10GB of images + 50GB of music. Yes, I know that 500GB drives are common now, but I'm fine with losing some quality at the expense of convenience. We do that with images (JPG vs RAW) and music (MP3 vs FLAC), etc.

What I need is a process or tool that can batch encode a list or directory of files to about 20% of the size (2.4GB/hour) or rate (6000kbps) of the original format. I came up with the 20% number through a little trial and error and it seems reasonable: 12GB * 0.20 = 2.4GB per hour which would allow about 2 hours (4.5GB) on a DVD.

Every few months for the past couple of years I've taken a stab at this only to give up in frustration. I've learned a lot in the past two years. I went from just pulling everything with MovieMaker into one semi-decent sized WMV file that looked like crap to finding out about all of these great open source tools. Unfortunately, I'm only able to grok the GUI tools (AutoGK, AutoMKV, RipBot, MediaCoder) but can't quite make the leap to CLI (or even MeGUI).

At this point, after having experimented with H.264 (or x264), I don't think that I want to go that route primarily because it is slightly less shareable (I know, they all require a codec, but most of my family would not be on H.264 yet) but mainly because the editor support was pretty bad with MovieMaker and Premiere Elements. Sidenote: Sure editors will get there and I should probably be using better editors but we're not there and I'd like to let me kids use the easy tools also.

At this point, I'm down to the following:

MediaCoder - I like the fact that it supports batch and I could use one tool to encode to this archive/backup format and then also use it to go to iPode format, etc. However, I am having serious de-interlacing issues that I haven't been able to get past.
AutoMKV - I really like this tool but I'm having trouble getting some of the profiles to come out like I expect.
AutoGK - Consistently easy but doesn't do batch.


Sorry for the long post. It's kind of a rant or outlet for my frustrations for not being able to move past this. As a programmer, I tend to get stuck and can't move past certain things. It seems like there should be something that is easy like Windows Media/Movie Maker/Premiere Elements/Quick Time but with better quality; but not as complicated as some of the existing options.

I think what I want is the equivalent of WinDV (minimal options) that can encode a directory of DV home video to a near-lossless quality (that the experts here agree is ideal for quality vs. space) that is easier to backup and use and share. Alas, I know that as soon as you hide the options, someone will want the ability to tweak things which gets us right to where we already are.

Anyone have any profiles for MediaCoder or AutoMKV that they use to backup their DV?

[scharfis_brain]

I'd suggest you to use MPEG2 for archiving your DV-Videos.
Procoder has very nice quality and also a built in batch processing feature.

[Video Dude]

I would do all of the below to preserve home movies that can't be recreated:

Encode high bitrate MPEG-2 (DVD). Burn two copies: Verbatim 16x MCC-004 (PAPA) and TY Premium 8x DVD+R. Give one copy to a family member or friend offsite.

Store DV files on external WD or Seagate 400 or 500 GB. Cost less than $100 and would take an only an hour to two to copy. (A lot easier than all that encoding and lets you access the the raw files later on to convert to future formats.)

Keep all the DV tapes.

[mattsmith321]

Thanks guys. MPEG-2 hasn't really even been on my radar until today. I think I was caught up in the hype around DivX, Xvid and x264. Obviously those are the better formats if I'm looking for smaller file sizes, but since I've got some flexibility...

Off to do some more research and testing!

[mattsmith321]

Great. The more I read, the less I know. The thread, Is it harder to encode DV into MPEG2?? (http://forum.videohelp.com/topic264730.html) was a very enlightening read. It seems I have another learning curve in front of me.

I have yet to look at ProCoder since I would have to purchase it. It feels odd going from all open-source for the Xvid and x264 tools to having to pay to do something in an older format.

[Southstorm]

ProCoder is good, but so is the free HCenc. My brother and I have been making amature movies from a DV camcorder with an Mpeg2 output for years. With the proper bitrate and settings, you can achieve quality results. We didn't get it right the first time, it took some experimenting and tweaking to say the least.

[smok3]

there is a difference between pal DV and ntsc DV in terms of how chroma is sampled, afaik both matrixes are incompatible with how mpeg2 stores chroma (but pal fits in a bit better).

[scharfis_brain]

There is no other format than DV AFAIK, that stores the weird chroma sampling natively.

[ron spencer]

hard drives are cheap....cheap...just store DV...simple easy


autogk will batch

[JohnnyMalaria]

there is a difference between pal DV and ntsc DV in terms of how chroma is sampled, afaik both matrixes are incompatible with how mpeg2 stores chroma (but pal fits in a bit better).

Yes - PAL uses 4:2:0 and NTSC uses 4:1:1.

[mattsmith321]

autogk will batch

Can you provide a little more detail on that. I haven't run across that so far.

[mattsmith321]

hard drives are cheap....cheap...just store DV...simple easy

I don't disagree with you about that. I do have an issue about maintaining that much data when it probably wouldn't be used that much.

I think I'm going to back off my requirement about needing to work with the encoded videos in editing software. I think at this point in time, it is more important to me to have encoded/compressed versions of the files so that I can a) keep a copy of everything on my main hard drive so that my wife and kids have easy access, b) import into Adobe Photoshop Elements so that the clips can be organized/tagged/categorized with my pictures,and c) easily keep backup copies on my current 250GB backup hard drives.

If and when I ever have a need to put a project together and I find that the encoded versions aren't working or are too lossy, then I can always go back to tape. Yes, it will be a little bit of a pain to re-import, but that is a trade-off I can live with.

So, I will probably end up with the following:
- Import with WinDV from tape to date-stamped clips
- Encode/compress clips to ~6-8Mbps (still need to decide MPEG-2/XviD/DivX)
- Delete original clips
- Import encoded clips into Adobe Photoshop Elements
- For iPod or lo-fi (youtube, online, etc.) needs, use the encoded clips.
- For projects where quality is a concern, re-import the original clip.
- At some point, I will probably copy the DV tapes to new tapes to keep them fresh.

I'm not a videophile so I have no interest in obsessing over the differences between chroma, quants, mux, etc. My family did lose everything when I was growing up so I am especially cognizant of keeping separate physical backups at different locations. That, and being able to have better access to the clips is important.

Thanks for everyone's help so far. I really do appreciate it and admire those of you that understand and have to deal with all of the nuances associated with video and audio.

[Southstorm]

You already know my vote is for Mpeg2, and as far as editing goes, Cuttermaran is awesome for Mpeg2.

[scharfis_brain]

yeah, MPEG2 should be your choice, if you need to compress your videos.

it is compatible and standardised.
not interlacing issues etc.
for divx/h.264 there is no good throughout support over all sorts of applications.
also mpeg2 is easily being 'reformatted' for DVD-Player compatible DVD-Video-Discs.
divx and h.264 will need re-encoding.

[ron spencer]

there is one BIG problem with MPEG-2...suppose in 5 years you want to convert them to HD? You will need to go back to your tapes if you have them....MPEG-2 at 740x480 to HD just does not work....you compress twice.

for autogk....just add job, then go back and load another video file...click add job...over and over....there you batch

[JohnnyMalaria]

So, I will probably end up with the following:
- Import with WinDV from tape to date-stamped clips
- Encode/compress clips to ~6-8Mbps (still need to decide MPEG-2/XviD/DivX)
- Delete original clips
- Import encoded clips into Adobe Photoshop Elements
- For iPod or lo-fi (youtube, online, etc.) needs, use the encoded clips.
- For projects where quality is a concern, re-import the original clip.
- At some point, I will probably copy the DV tapes to new tapes to keep them fresh.

About the two things I have emboldened above:

What do you mean by date-stamped? If you mean the date information is in the file but not burned onto the video itself, *any* DV capture application will do that. If you want the date on the video permanently, WinDV won't do that. You have to use it in conjunction with another program to add the date afterwards or use a separate program that does both simultaneously. Which is better depends on exactly what you are after.

Copying tapes to new DV tapes - everything on the tape is duplicated EXCEPT for the timecode. So, if your original captured information is in any way dependent on the timecode, you may have a problem down the road. e.g., if your editing program has created clips and noted the timecode so you can recapture them at a later date then the copy of the tape won't work properly.

[mattsmith321]

yeah, MPEG2 should be your choice, if you need to compress your videos.

it is compatible and standardised.
not interlacing issues etc.
for divx/h.264 there is no good throughout support over all sorts of applications.
also mpeg2 is easily being 'reformatted' for DVD-Player compatible DVD-Video-Discs.
divx and h.264 will need re-encoding.

After do some more reading and thinking about it, I definitely do agree that going to MPEG2 is probably the best approach. Now, I'm in a bit of a quandary on how to go about that. The MPEG2 Encoder Test (http://forum.videohelp.com/topic235665.html) thread was very informative regarding my options. Everybody seems to really like ProCoder. However, $200 is a little more than I was wanting to spend. I have no doubt that it is a quality tool but much of it's functionality would be lost on me. I thought I had found the perfect solution at $60 in ProCoder Express (this review (http://www.videomaker.com/article/10551/) was great) but alas, that isn't sold anymore and the functionality has been integrated into ProCoder 3.

I will be taking a look at the tools listed here (http://www.videohelp.com/tools?convert=DV%20to%20MPG) and elsewhere to see what I can come up with.

[mattsmith321]

there is one BIG problem with MPEG-2...suppose in 5 years you want to convert them to HD? You will need to go back to your tapes if you have them....MPEG-2 at 740x480 to HD just does not work....you compress twice.


I'll take your word on that. However, that would still be an issue if I went to the other formats (Xvid, x264, etc.), correct? By going MPEG-2 I would at least get the benefits that scharfis_brain laid out. Regardless, hopefully I am still out five years or so before I get on the HD bandwagon.

for autogk....just add job, then go back and load another video file...click add job...over and over....there you batch

Yes, I guess that does quality as batch but was looking for something batch-ier. :) But I guess I can't use AutoGK now that everyone has convinced me to go MPEG2. :p

[mattsmith321]

What do you mean by date-stamped? If you mean the date information is in the file but not burned onto the video itself, *any* DV capture application will do that. If you want the date on the video permanently, WinDV won't do that. You have to use it in conjunction with another program to add the date afterwards or use a separate program that does both simultaneously. Which is better depends on exactly what you are after.

I just meant that I let WinDV name each clip according to the tape date info. I use the following filename formatting: %Y-%m-%d. Most of the stuff in my head is organized chronologically, so I like having all of my clips named according to the date of the clip. The fact that WinDV could do that and split the clips based on a discontinuity threshold were exactly what I was looking for.

Copying tapes to new DV tapes - everything on the tape is duplicated EXCEPT for the timecode. So, if your original captured information is in any way dependent on the timecode, you may have a problem down the road. e.g., if your editing program has created clips and noted the timecode so you can recapture them at a later date then the copy of the tape won't work properly.

That is interesting (and will probably be annoying). Thanks for the heads-up. If it's not one thing, it's another. You just gave Ron another reason for why I should keep the original clips. :) He must work for one of the hard drive manufacturers. Just kidding.

[Southstorm]

Have you checked out HCenc? Its a very good Mpeg2 encoder, and there are a few gui's that work with it for batch converting.

[mattsmith321]

Have you checked out HCenc? Its a very good Mpeg2 encoder, and there are a few gui's that work with it for batch converting.

The only things I've tried so far are:

TMPGEnc 2.5 ($37) - The first two times I tried it, it didn't work. I just tried it (to get the error message again) and it worked just fine. However, the extension was .m2v and there was no sound. Not too good for my first attempt.
TMPGEnc 4.0 XPRESS ($100) - Very nice! It does exactly what I want. I can drag and drop a group of files onto it and walk through about 4 screens and leave the defaults and click Start and I'm good to go. The only issue I'm having at the moment is that Windows Media Player and QuickTime don't like the output. But I'm sure that is a codec thing.

The price for XPRESS is a little more than I want to pay but at this point, I'm willing to pay it just to put myself out of my misery. If the differences between the encoders are as minimal as this example (http://forum.videohelp.com/topic235665-210.html#1367483), then paying the extra $100 for ProCoder is not worth it to me (not too mention the no trial and awkward purchasing process).

I'm almost out of your hair. There maybe a couple follow-up questions.

[mattsmith321]

OK, I'm done. I can move forward with the rest of my life now :)

As mentioned above, TMPGEnc 4.0 XPress worked out great for encoding my clips to MPEG2. It two attempts to get a good DVD out of it.

On my first attempt, I took the encoded clips and created the DVD in Adobe Premiere Elements (APE) 2.0. I got kind of worried because it seemed to actually be re-encoding the video. When I played the DVD on my TV, it was very choppy/stuttered/odd/not right.

Figuring that APE had something to do with it, I decided to try to create the DVD in TMPGEnc DVD Author 3. When I got to the Simulation screen, I received a message that MPEG-1 Audio Layer II was not the best format and that the source audio should be converted to either PCM or Dolby.

So, back to TMPGEnc 4.0 XPress. While changing the audio to Dolby, I thought I saw a setting that indicated the interlacing on the clips was being read as Top Field First. From everything I've read, the DV clips should be BFF. So before re-encoding, I made sure that they were all set to BFF.*

I took the newly encoded clips and ran them through DVD Author and created another quick test DVD. When viewing the new DVD, two things were immediately apparent**:

Quality - One of my clips was shot indoors and had some frames of the ceiling texture (with motion). In the first DVD, the paused image contained lots of pixellation/blocky artifacts. The second DVD, while obviously not perfect given the conditions, was much better.
Smoothness - The choppy/stuttering motion from the previous DVD was gone.


So, my birthday is coming up this weekend and I think I'm going to buy myself a copy of TMPGEnc 4.0 XPress***. I may have to go back for DVD Author or and editor, but I'll be busy for a little while encoding what I've got.

Thanks everyone! :thanks:


*In comparing the first and second round of encoded clips, I can't tell any difference when watching them or looking at stills, so I think the Field Order was set to BFF originally. This potentially means that APE introduced the artifacts and the choppyness. But I haven't gone (and probably won't go) back to definitively prove it.

** To give you an idea how much better, I had my wife watch both DVDs to see if she could tell a difference. While not trying to discredit my wife, she is normally oblivious to the nuances that drive me up the wall. However, the differences were significant enough between these two tests that she could easily tell the difference.

*** While ProCoder came out in some of the tests in the threads I linked to, it would be lost on me (I think there was a wine analogy in one thread). TMPGEnc may not have been the best, but it wasn't as criticized as some of the others. I'm happy with my decision so please don't burst my bubble.

Thanks again!

[mattsmith321]

Two last little requests:


Move this thread - I initially posted in Xvid because that is the format that I thought I was going. However, since it became a DV to MPEG2 thread, this thread should probably go in DV (http://forum.doom9.org/forumdisplay.php?f=59) or MPEG-2 Encoding (http://forum.doom9.org/forumdisplay.php?f=62)
How do I get the .mpg clips to play in Windows Media Player? It says that it is acquiring the codec but then only plays the audio. Admittedly I haven't searched yet, but if anyone has any pointers ...

[Guest]

Move this thread Moved to the DV Forum.

How do I get the .mpg clips to play in Windows Media Player? You need a DirectShow codec that supports MPEG2. The easiest thing is to install ffdshow.

[mattsmith321]

You need a DirectShow codec that supports MPEG2. The easiest thing is to install ffdshow.

Thanks! I had ffdshow tryouts [rev 1723][2007-12-24] installed. I uninstalled, rebooted, re-downloaded, re-installed (still same version) and selected the appropriate checkboxes to get it to work with MPEG-2. It works now.

You have no idea how happy I am right now :p

[mattsmith321]

Hey Matt!

This post is from your future self. Because our main computer crapped out and we don't have access to the video we converted way back then, I decided to revisit again. Check out this thread for more details about how I did it his time and got much better results: https://forum.doom9.org/showthread.php?p=1908957#post1908957

Hope that helps. Be sure to get a Synology NAS box sooner than later. Also, Plex is great to access the home video content but comes with it's own issues as well.

Take care! Come see me some time.

[videoh]

This post is from your future self. Love it! I'm not the only schizoid on this board.

BTW, DV sucks for backup.

Y'all come back now!

[DJATOM]

FFV1 is good for backup or intermediate purposes, as well as HEVC lossless (if you have an RTX card).
Former is limited to 8 and 10 bit formats, while FFV1 can store even 16 bit clips.