Lossless Backup to RAID5 for Streaming

[Finite Reign]

First, I must apologize for how simple what is to follow may be for the vast majority on this site. I am aware many have been doing this for years, and have written a great deal on the topic, but I am quite new and am having some issues finding the right guide and/or direction to get this project on the road. So, I will begin by explaining what I would like to do, then I will ask my questions and for advice on the project as a whole. As a bonus, since I write tech documents for the lowest common denominator, I will document, step by step, everything done in an easy to follow pictorial of the entire process for those who want it.

Project:

5 Terabyte Actual Storage RAID5 Array for use as a Library of my entire DVD collection. My collection currently consists of 326 individual movies and a further 35 or so Seasons of various TV shows. The library is meant to store and stream in a lossless format, the entire collection over my home network to players in various rooms. All but one player can accept multiple video and audio formats. The player that cannot, is an Xbox360. The 360 can only accept Windows Media formats at present. (but hopefully this will change in the future, or someone will develop an on the fly encoder to acceptable formats for use with the 360)

The Library will only contain the movie file; no extras, special features, menus, subtitles or commentary. I would like to maintain the audio in it's (best) original format as well since more than one room is set-up with a 5.1 system.

The hardware is set, minus the new 10k RPM system drive that will replace the current 7200 RPM drive. (the system will end up handling the ripping of DVDs in the future, after the initial library has been ripped down by a performance box)

Questions:

Which format/standard is best to rip to for this lossless need? Xvid, DivX:-), AVI, MPG, WMV, etc?

Is there a tradeoff in audio, and if so, can it be avoided to maintain its original quality in its original format? (ie, if it is 5.1, maintain 5.1)

Specifically for the 360, is WMV an option for lossless? Will it maintain the 5.1 tracks if it is an option? (if it is not an option for lossless, what encoder would you recommend anyway to get as close as possible. reason being that some dvds will likely be encoded in this manner as well as lossless)

I've put the total data storage to be around 3.75 - 4 terabyte, giving a beefy 7 gig per movie(better to assume more than less), assuming an average of 6 discs per season of a TV show across 35 seasons to start, leaving ~1 Terabyte of actual storage reamining. Do these figures sound appropriate for a lossless data library of this magnitude? I quite frankly know I over estimated size in order to allow for library growth without near future purchases.

If there is anything you think I missed, or have any advise in regards to what direction I should be headed, hardware or software, please do tell.

I am sure I will have some more questions shortly so stay tuned...

[Awatef]

I don't understand what you want! :confused:
On the one side you're talking about "lossless", on the other about very lossy formats like DivX and WMV!

If you really want "lossless", you have to copy your DVDs 1:1 to your hard disk, without converting anything. Tools like DVDShrink can help you removing menus and extras.
In this case, the only thing you have to worry about (beside the storage capacity) is wether your network is able to handle bitrates up to 10Mbit/s. If all your systems are connected in a 100Mbit/s wired network, no problem, but if you're going to install a Wi-Fi network, you better check your bandwidth on every system before going any further, since 10Mbit/s are pretty much for a Wi-Fi network. You may even have to change your whole equipment to go beyond the 802.11g standard to be able to have guaranteed 10Mbit/s even in far rooms.

If you're going to convert to DivX or whatever codec you prefer, Bitrates around 2Mbit/s will be sufficient to deliver near-DVD quality. Dolby Digital sound can be left as is, no need to convert it, since it doesn't use bitrates higher than 448kbps.

[Finite Reign]

It was a question. I didn't "talk" about it as something to be done. As I said, I am new to this, and as such, I posted what I was looking to do, and asked if those formats were options or not. Sorry I didn't clarify the questions portion further.

I did stress testing on the network with an AVIA rip I managed some time ago. Although it was strictly laptop to performance box with the rip over the network, it was able to play fine with no hiccups as a network share in that manner, so there should be no difference otherwise. So, I should clarify here the network portion.

I have created lopro network drops in the walls key points in each room, so there will be no need to use wireless. As it stands the "G" revision is not worthwhile for this application. For me, wireless will only be an option with the "n" revission of the 802.11 standard. MIMO will be the way to go... unless of course they hurry up the HDWireless standard... Though that is a good 10 years off.

I also understand that "lossless" is not something that can be achieved in regards to compression, by nature of the word alone. So, in a 1:1 rip, I am left with .vob extensions that are not recognized across the network by the media steamers. So, I guess the question becomes, how do I make it recognizable without sacrificing quality.

I will check DVDShrink and the guides for it and give it a test run when I am off work. My understanding though is that it seperates out the audio track. So, I must then combine that again with the video yes? How is this resolved to create a single file solution?

[Awatef]

DVDShrink doesn't separate audio and video, it just let you choose which elements from the DVD you want to keep and which not. So it's pretty straight forward.

The thing about the "media streamers" is not my speciality, so I would gladly let other people comment on it, I have no idea, sorry.
I thought, you were going to share the videos in the network, so everybody can play them off your server. I do that on my home network, it works like a charm (everything works as if the file was on the client, not on a distant server, it's pretty cool :))

PS: you could even put ISO images on the server (DVD Shrink can output ISO files if you want), and they can be mounted on the client using Daemon Tools. That way, it is as if the DVD was inserted in the client. The amazing thing is: IT WORKS!

[Finite Reign]

My first thought on the project as regards the files was to do that exact same thing (ISO > mount with daemon tools. This is how I stress tested that portion of my network; by mounting the ISO located on the performance box from another point on the network with a laptop).

However, as opposed to have an established computer in every media viewable location in the home, I wanted to go the media streaming route via options such as D-Link's media player (cheaper route). Though I have not decided on a single standard player that I will purchase for all the appropriate rooms, they all have the same general features and compliance with a majority of video and audio formats.

I guess, in the end, I want the highest quality possible a/v file that is of the standard 2x click to play (assuming codecs are present). I hope I am not muddling my thought process on what I want to do, so if it seems that I am making conflicting requests, please call me on them so I can clarify.

The ideal setup, as soon as I can figure the best method for ripping my collection, will be to have all of my audio and video discs available on demand from any room in the house without having to actually touch a disc. I have the CD portion covered and will be migrating them from my performance box to my RAID array as soon as my new system drive arrives.

[Finite Reign]

Alright, I ripped down the first movie in the collection with DVD Decrypter. I selected that I wanted movie only and a "none" option somewhere to make it a single file. After 10 mins, I was left with a 7 Gig Vob file, and IFO and a playlist.

Knowing that Power DVD is able to play VOB files, I dropped the vob in which promptly started to play, but generated a typical "this app must close" windows error. The movie was playing behind the error, but, never-the-less, there was an error. Also, the movie skipped around a bit during that playback.

I was told that I could just re-name the .vob to .mpg and it would play normally as well. So, after having renamed it, it opened by default in WMP. However, it to played choppy and seemed to jump around.

Suggestions? The DVD is "The Abyss". The disc specifically is from the 2 disc special edition in white packaging.

[Awatef]

I don't remember well how DVD Decrypter outputs files, but if it doesn't create a VIDEO TS directory, use DVD Shrink, it allows you to output a VIDEO TS directory. That way, you don't load individual VOBs into PowerDVD, you just open the folder.

[Finite Reign]

Decrypter created a heirarchical structure as such:

THEABYSS > Video_TS > .vob .pls .ifo

Again, I will have to wait till after work to play around with the program.

[ilovejedd]

Aren't Xbox360's supposed to be able to open DVD files?

Anyway, does your DVD have any scratches? That's the only thing I can think of that would explain the skipping. Try opening the VOB file with VLC and see if you still experience errors playing the file. If that fails, maybe can you try ripping your DVD with RipIt4Me?

P.S. I know you're still thinking about it, but do you have a semi-concrete idea on what player you will use to watch the files? Are you planning on maybe installing a Media Center PC on every room? I'm told that a modified Xbox with XBMC (http://xboxmediacenter.com/) installed would be quite good for playing streaming media. Plus, it supports a wide variety of formats (almost everything you can play on your PC). Unfortunately, I think the processor is only PIII 733MHz so it probably won't be able to handle HD content and would have trouble playing MPEG4 AVC encoded content.

[jel]

@ Finite Reign

from what i have read, the abyss (special edition) contains seamless multi-storey (ILVU) which is why you may be getting choppy playback which jumps around (as it is playing the 'extended/different' scenes one after the other.

iirc, you can choose to use dvd shrink to do a main movie back-up and it should work (although i dont use shrink, so i am only guessing here)

good luck
j

[Finite Reign]

@ilovejedd

The 360 did not see the .vob or the .mpg in the shared folder. To qualify this statement though: The 360 has always seen my .mp3s, pics, and now the random "funny clip" WMVs I've collected. Since I have decided to build the new box, I have decided to re-rip all my music in FLAC to stream to the media players I am still scoping out. I will keep the .mp3s on the performance box just for the 360, and will look into an on the fly encoder for videos when I finally get my library built, ripped, and finalized in a persistant fashion.

I do not intend to put an HTPC in every room, that is simply not cost effective, and having them sync up with each other would be just short of creating an Advanced Server network that replicates much like a BDC would... So instead, I am making a single mass storage solution that I will share via any one of many forms of network sharing.


@jel
DVDdecrypter had the option for main movie only, which is what I selected originally. I will have to check again, possibly with a different disc?

I will download and attempt the rip tonight with DVDShrink and post back with my results

@ all

I want to thank everyone thus far for their help. Sorry if my progress doesn't seem as fast as it *should* be, I am currently trying to balance many things and fit this much wanted/needed project in.

[Awatef]

You better discard the Xbox then. WMV is not a format I would like to use (how many people transcode to WMV in this forum? 1%? 2%?!). You better get another SAP to replace the console, and which will let you read VOBs, it's that simple.

[Finite Reign]

Replacing the console is a non option. (Gears of War was just realeased... come on now...)

As I said in the previous post, I will continue with the full DVD library as "lossless" rips, and will wait (in so far as the 360 is concerned) for an on the fly transcoder. There were several developed for use with MCE, and now that MS has allowed streaming from XP SP2, I'm more than sure an on the fly transcoder will be developed for XP SP2.

At this point, my main concern is getting a successful, single file rip of the main movie in a lossless audio / video format. From reading these forums and guides, it seems that a very simple ripper should manage this just fine into a .vob.

My new ANTEC Black P180 has arrived today so, I will be finishing up the hardware portion of the entire project soon. That said, I will need to spend much more time trying to get a good process for ripping my discs down.

Again, thanks for the help thus far. More questions and comments to come after work.

[ilovejedd]

I do not intend to put an HTPC in every room, that is simply not cost effective, and having them sync up with each other would be just short of creating an Advanced Server network that replicates much like a BDC would... So instead, I am making a single mass storage solution that I will share via any one of many forms of network sharing.

True, building an HTPC in every room would be quite expensive. However, you needn't have a copy of your entire collection on each of the HTPC's HDD. You could always stream them from your 5TB file server to the HTPC.

My suggestion with the modified Xbox still stands, if you're adept at tinkering. An Xbox is pretty cheap nowadays ($99 brand-new last I saw) and the modchip will cost you about $50. Of course, there is the hassle of having to solder the modchip and compile and install XBMC, but the Xbox would be more versatile in playing various formats than a networked SAP would.

You better discard the Xbox then. WMV is not a format I would like to use (how many people transcode to WMV in this forum? 1%? 2%?!). You better get another SAP to replace the console, and which will let you read VOBs, it's that simple.

I see no reason to replace the Xbox 360 with an SAP. I'm pretty sure the media features of the Xbox 360 are just a bonus and he uses the console mainly to play games with...

[Awatef]

Ho Ho! I didn't say you should get rid of the Xbox, I just said, don't include it in your network. I don't think it's a problem to have a SAP and an Xbox in the same room, is it? Just use the Xbox for games only. The on-the-fly transcoding option is no good, you have to run all the time between your PC and your Xbox, and believe me, you really don't wanna have a WMV collection. Microsoft has tried since ages to push WMV, they never made it, and they never will, if you ask me! Wait a few months, some clever heads will surely come up with something that allows you to stream everything from your server, even VOBs, trust me on this! :)


By the way, an HTPC doesn't have to be expensive. Even a 300$ low-end PC will do it as long as it has TV-Out.

[ilovejedd]

By the way, an HTPC doesn't have to be expensive. Even a 300$ low-end PC will do it as long as it has TV-Out.

Quite true. Still, the cheapest networkable D-Link media player costs half that and looks prettier (unless you buy a case that costs $200 upwards). Although the HTPC is much much more flexible, but then again, if all you need to play are backups of your DVD collection, flexibility isn't a requirement.

[Finite Reign]

@ Awatef

I think you are sort of missing the point of the media server I am building. ilovejedd has it though. In the end, a person only needs a SINGLE pc solution in regards to hometheater. All sharing, ripping, and even copying of TV sources, both HD and SD can and actually *should* be handled by a single machine. Not only is it the most cost effective solution, it is also easier from a support stand point.

Not only would I be able to share out my entire library of DVDs, CDs, and recorded broadcast television across 3 rooms at the cost of a single HTPC, but the ambient noise of the single PC solution is far less than that of 3 machines. Although my performance box hums along at a very acceptable noise level due to careful air management within the case, it would be unacceptable to have that same hum across multiple rooms without need.

My current design for this server is built inside of a black ANTEC P180 case, using a Seasonic M12 80+ power supply, a Scythe SCNJ-1100 Heatsync and Nexus 120mm fans. Those components were chosen not just for their quality and reliability but also for their cooling capabilities without sacrificing overall noise level. The SPCR website was a great help with their reviews on these products.

As regards the Xbox 360; There will eventually be an on-the-fly transcoder for it. It, when installed, will allow the 360 to see everything, and would encode the file on the fly during playback for the purpose of streaming. So, at this point, I just need to make sure that the rips I am making now are as high a quality as possible (lossless - no compression) so that I can build and maintain a proper library...

@ all

DVDShrink Came across as a much more intuitive ripper. As regards that Abyss DVD specifically, it showed me the differentversion of the film present on the disc and allowed me to choose the most inclusive of them. The rip itself took 15 mins or so, give or take. I am, however, now presented with several files that I do not know the purpose of. I have 2x .bup 2x .ifo a .vob and a .pls.

When using Power DVD to open the Video_TS folder, the movie plays properly. No hiccups or glitches. The error of the previous issue was that DVDDecrypter tried to combine the other versions of the film into a single file. So, what is the purpose of the VIDEO_TS.BUP, VIDEO_TS.IFO, VTS_01_0.BUP, VTS_01_0.IFO, and the Default.PLS? I'm up to speed on the VTS_01_1.VOB in so far as what it is and what it contains, but the naming structure makes no sense to me. All of these are contained in aVIDEO_TS folder. The AUDIO_TS folder is empty. No idea why it was created if it is empty.

The .vob ripped down to 5.68 G with no compression. Which seems odd, since the file was listed at over 6 G in the initial DVDShrink screen. Suggestions?

[Awatef]

@ Finite Reign
I think I got your point, but may be we passed at each. A low-end HTPC doesn't have to be loud, you invest 10 or 20 more bucks and you get a silent cooler. Here I'm not talking about your server! I'm talking about the clients in the other rooms, what you called "media streamers" I think. Instead of the Xbox, you get a low-end HTPC (with player functionality only), or a VOB capable SAP. Of course you keep your Xbox for gaming, but it doesn't have to be a part of your network as of now. As said, the on-the-fly transcoders available right now require that you setup everything on the server, then go to the other room where your Xbox is to begin watching: totally betraying your idea of accessing everything from any room in a transparent way.
Please do not understand something big behind the term "HTPC" ;)

As for DVD Shrink, it probably removed the redundant data. As long as you specify "No Compression", you shouldn't worry about anything ;)

[Cyberace]

XBMC can play DVD ISO (even from RAR/ZIP archive) with the DVD menu and everything