Raid Controller

[Gabe Logan]

Hey guys i wanted to post this 5 days ago but you know how it goes :angry:.Anyways i will be capturing HD through my Blackmagic card through 3 750gb Seagates SATAII in RAID0 but was unsure what controller to get so i ordered this one today.

http://www.newegg.com/Product/Product.aspx?Item=N82E16816103058

I only want to spend as much as $200 so is a good choice or something better?In a review someone mentioned that3ware makes really good controllers but they are more than triple of what i am paying for this one.

[Seraphic-]

Hey guys i wanted to post this 5 days ago but you know how it goes :angry:.Anyways i will be capturing HD through my Blackmagic card through 3 750gb Seagates SATAII in RAID0 but was unsure what controller to get so i ordered this one today.

http://www.newegg.com/Product/Product.aspx?Item=N82E16816103058

I only want to spend as much as $200 so is a good choice or something better?In a review someone mentioned that3ware makes really good controllers but they are more than triple of what i am paying for this one.

Need more information before I can help.

1st. What Windows OS are you using?
2nd. Is it 32-bit or 64-bit?
3rd. How much space is usable on each hard drive after format?

[Blue_MiSfit]

Doesn't look too bad to me :) apparently it supports 32 and 64 bit windows, so that's good news.

There's a lot of SATA RAID controllers out there these days, and I've yet to find a definitive source of reviews and comparisons - so it's hard to tell. This seems to be a good fit for this application (maximum sustained transfer speed for HD capture). No RAID5 or battery backup / large cache, but that's for high end controllers with different requirements.

Heck, you would probably be okay with software RAID, but why be masochistic? :D

I would be interested to see how you do on the blackmagic benchmarks. Stripe size and NTFS cluster size will likely have a large impact. I would set them as large as possible since you're going to be creating ginormous uncompressed HD files :) Who knows though, maybe this controller has a sweet spot.

Let us know how it goes!!

~MiSfit

[Gabe Logan]

Need more information before I can help.

1st. What Windows OS are you using?
2nd. Is it 32-bit or 64-bit?
3rd. How much space is usable on each hard drive after format?

1) Windows XP Pro
2) 32 bit
3) I don't know but i will post that soon.

I will be using it in a quad core with 2gb ram.

[Gabe Logan]

I would be interested to see how you do on the blackmagic benchmarks. Stripe size and NTFS cluster size will likely have a large impact. I would set them as large as possible since you're going to be creating ginormous uncompressed HD files :) Who knows though, maybe this controller has a sweet spot.

Let us know how it goes!!

~MiSfit

Ya i am excited about experimenting with the best possible configuration to get as much performance as possible so it's good to see the card will fit the application.I will post all info if all goes well by this weekend.

[Seraphic-]

1) Windows XP Pro
2) 32 bit
3) I don't know but i will post that soon.

I will be using it in a quad core with 2gb ram.

Yeah, I'm using that as well. As for the size after format, each of your drives should work to 698GB each. That should bring your total to 2,094GB. Something you have to consider (as I did), is that the max partition Windows XP Pro 32 bit OS can address is 2TB (2,048GB). So you are over that and might have a problem. This is why I went with three 640GB (596GB each) drives my raid0 array. It works out to 1,788GB total, well under the limit.

As for the raid controller, you should have waited. I was going to buy one as well, but after looking at the prices, I decided to try using the on-board motherboard Intel raid controller first. Doing so gives my raid0 array 300MB/s read and write. So a hardware controller is not needed.

You are using Blackmagic Intensity Pro? I am, and can tell you I have zero problems capturing uncompressed 720p and 1080i. You should have no problems as well. My system is a Intel quad core and 4GB memory.

[jeffy]

2TB limit, some reading:
http://www.carltonbale.com/2007/05/how-to-break-the-2tb-2-terabyte-file-system-limit/

[Gabe Logan]

Something you have to consider (as I did), is that the max partition Windows XP Pro 32 bit OS can address is 2TB (2,048GB). So you are over that and might have a problem.

Does this mean that i will lose 94gb or will it not accept it at all?

[Seraphic-]

Does this mean that i will lose 94gb or will it not accept it at all?

It should accept the array, as long you are able to enter the size you wish to use for the array. If it tries to force you to use all 2,094GB, then I'm not sure it the array will be accepted.

If you can enter the size you wish you use then you will just not be using your total amount. In that case, you would not be using 15.3GB or 31.3GB from each drive.

[Gabe Logan]

As for the raid controller, you should have waited. I was going to buy one as well, but after looking at the prices, I decided to try using the on-board motherboard Intel raid controller first. Doing so gives my raid0 array 300MB/s read and write. So a hardware controller is not needed.

I am going to try out both the onboard and the Adaptec and see if shelling out the $100 was worth it or now.How much cpu usage are you using with the onboard RAID?

You said that Windows 32 has a problem with the 2TB limit but what about the 64 version or Vista? From what i have heard Vista has too many things running in the background so maybe it's time to upgrade to the 64 version?

[Seraphic-]

I don't think 64bit XP or Vista have the same limitations as 32bit. Should be fine, but you better google and research first to be sure.

[Gabe Logan]

I would be interested to see how you do on the blackmagic benchmarks. Stripe size and NTFS cluster size will likely have a large impact. I would set them as large as possible since you're going to be creating ginormous uncompressed HD files :) Who knows though, maybe this controller has a sweet spot.

Let us know how it goes!!

~MiSfit

Well i could not get the adaptec card to work out (maybe another day) so i went with the onboard RAID0 and only 2 750 Seagates connected and will see if i can get a third hooked up tomorrow.I can use three in a raid0 right?

Anyways my benchmarks for the RAID came out to this:
8353

I used the compression from the Intensity to capture game content which came really stable but nothing more than that so now it's off to try uncompressed :eek:

[Seraphic-]

Well i could not get the adaptec card to work out (maybe another day) so i went with the onboard RAID0 and only 2 750 Seagates connected and will see if i can get a third hooked up tomorrow.I can use three in a raid0 right?

Anyways my benchmarks for the RAID came out to this:
8353

I used the compression from the Intensity to capture game content which came really stable but nothing more than that so now it's off to try uncompressed :eek:

I think something might be wrong with your raid0 setup. Those speeds seem really... slow.

Here is my raid0 result. Wish I could add a fourth drive, but then I would be over the 2TB limit for 32bit XP Pro. Rather not deal with those issues.

[Gabe Logan]

You are correct Seraphic.After i posted those numbers i added a third drive but it came out the same so i knew i had messed up somewhere and since this is the first time i have ever setup a RAID i eventually found out that i had to install the drivers before setup of XP.Anyways long story short i got it working with 2 750 Seagates and this is what i am getting.

[Seraphic-]

You are correct Seraphic.After i posted those numbers i added a third drive but it came out the same so i knew i had messed up somewhere and since this is the first time i have ever setup a RAID i eventually found out that i had to install the drivers before setup of XP.Anyways long story short i got it working with 2 750 Seagates and this is what i am getting.

Still can't see image. But yeah, you need to press F6 to install raid drivers from floppy disk when XP menu first loads for install.

[Gabe Logan]

You can't see the attachment pic i included in the post?

[Seraphic-]

You can't see the attachment pic i included in the post?

I can see it now, speed is much better. :cool:

[Gabe Logan]

I think something might be wrong with your raid0 setup. Those speeds seem really... slow.

Here is my raid0 result. Wish I could add a fourth drive, but then I would be over the 2TB limit for 32bit XP Pro. Rather not deal with those issues.

I have since gotten Windows Vista 64 so i can get rid of that 2TB limit but i am having trouble with anything more than 2 drives in raid 0.I use the P35 chipset and use the Intel Matrix Storage Manager to bring the drives together but if i try with my 3 750gb windows will not see it but 2 of those drives work just fine.

What kind of mobo you got working for you?maybe i will change to yours instead.Iv'e spent way too much time on this thing.

[Seraphic-]

I'm using a Gigabyte X48 motherboard (Intel/ATI).

[easy2Bcheesy]

One thing to get clear here. RAID controllers are all about using hardware to make the computations for backup flexibility (eg RAID5) without introducing a CPU burden. In short, the compression work required for RAID5 is carried out by the controller card without adding extra workload to the CPU.

RAID0 requires zero computational work in that it's basically streaming to hard disk and doing nothing else. Therefore an integrated on-chip RAID solution is perfectly fine for HD capture. No card required.

Use Samsung F1 Spinpoints if you want the ultimate in speed.

Or just save yourself a lot of time, money and effort and buy the CineForm codec.

[Blue_MiSfit]

Or just save yourself a lot of time, money and effort and buy the CineForm codec


+10!

Cineform is ridiculous. It doesn't seem logically possible. No serious HD content creator should be without it.

[Seraphic-]

+10!

Cineform is ridiculous. It doesn't seem logically possible. No serious HD content creator should be without it.

You do know that Cineform is lossy, right (however slight). Any serious HD content creator should capture lossless. Not lossy.

Seems like a waste of $250 to buy a proprietary capture program when for a little more, if not almost the same, you can do a raid0 with on-board raid and capture lossless.

But, to each his own.

[Blue_MiSfit]

Sure - Cineform is lossy. But it does a few amazing things:

1) It's real-time in Premiere, and extremely responsive
2) Its compression is excellent. No more RAID, you can capture to a freaking laptop in the field (!)

I also disagree that every serious HD content creator should capture lossless. It's not always feasible to have a multi-terabyte striped RAID array at your disposal. I've been involved on a project where we had to capture 8 bit 1080i60 @ 4:2:2 uncompressed, using nothing more than a MacPro with 4x 500 GB SATA drives. It was not fun! Cineform or an equivalent would have saved our butts (no ProRes yet)

Sure, if you have all kinds of storage available - why not - but Cineform is so much more flexible. ProRes / DNxHD compare very nicely in my opinion, so pick your codec to match your NLE - Avid gets DNxHD, Final Cut gets ProRes, and Premiere gets Cineform. Sure, you have to pay for Cineform, but it's 100% worth it if you're on any kind of budget for a film production, and plan on using Adobe as your editing app.

~MiSfit

[easy2Bcheesy]

You do know that Cineform is lossy, right (however slight). Any serious HD content creator should capture lossless. Not lossy.

Absolute rubbish. The massive majority of professional HD content creators are using a lossy solution. Even a megabucks HDCAM-SR digital tape deck - the most expensive and high quality solution a content provider can use - is not lossless.

Planet Earth - perhaps the most spectacular piece of HD programming ever made - was made using a lossy workflow. Parts of it were even filmed using DVCPRO-HD cameras - the codec used there is considerably inferior to CineForm.

More than that, all of the capture cards you've bought and used are using YPrPb 4:2:2, which even if you do capture losslessly is still tossing away 33% of the video information your video game sources are providing. If we're talking about serious content provision then you'd be capturing 4:4:4 or the native 24-bit RGB colour space of your video game sources.

Basically unless your final presentation is going to be played back losslessly, the difference is moot. The last phase of your workflow is going to involve enormous amounts of compression, making the difference between lossless and CineForm visually lossless at the penultimate stage basically irrelevant.

[Seraphic-]

Absolute rubbish. The massive majority of professional HD content creators are using a lossy solution. Even a megabucks HDCAM-SR digital tape deck - the most expensive and high quality solution a content provider can use - is not lossless.

Planet Earth - perhaps the most spectacular piece of HD programming ever made - was made using a lossy workflow. Parts of it were even filmed using DVCPRO-HD cameras - the codec used there is considerably inferior to CineForm.

More than that, all of the capture cards you've bought and used are using YPrPb 4:2:2, which even if you do capture losslessly is still tossing away 33% of the video information your video game sources are providing. If we're talking about serious content provision then you'd be capturing 4:4:4 or the native 24-bit RGB colour space of your video game sources.

Basically unless your final presentation is going to be played back losslessly, the difference is moot. The last phase of your workflow is going to involve enormous amounts of compression, making the difference between lossless and CineForm visually lossless at the penultimate stage basically irrelevant.

I mainly put in the "Any serious HD content creator should capture lossless", as a joke since he put out "No serious HD content creator should be without it" (CineForm). But also bcause I disagreed.

But anyway, I see the point. CineForm allows those with current systems, that are not that powerful or on one hdd, to capture near lossless HD on their current systems for $250.

Would still say that anyone building a new mid to high end system should go for the raid0 with three or four drives, for not much more $$$. Then you have a lot of diskspace to work with and can capture lossless.

Yes, all the cards available at a normal consumer price capture YPrPb 4:2:2 8Bit. I found one that did YPrPb 4:2:2 10bit, but was not worth it since it didn't offer at least 1080p30. I looked around the web for awhile when buying the hardware for my system. If I found one I would have picked it up, but found no HDMI cards that offer anything over 4:2:2 YPrPb at 10Bit (all 720p/1080PsF - no 1080p).

The one 10Bit card I found was the new DeckLink HD Extreme for $995.
(HDMI Color Precision - 4:2:2 10 bit Y,B-Y,R-Y and 10 bit RGB)
However, when I called to ask about it, they said it won't do RGB over HDMI (even though it lists it).

And even if you had pure 4:4:4 RGB, would would be the point? There are no encoders that support it (at a normal consumer price). Using x264 drops you to 4:2:0 YV12 no matter what, so you lose the extra quality. There are such things as high profile 4:2:2 and 4:4:4 encoding, but no encoder that doesn't cost a few thousand dollars supports it.

Also, on a side note. easy2Bcheesy, what would be a 4:3 resize of 853x672? (to mimic how it would look with 640x448)

[easy2Bcheesy]

But anyway, I see the point. CineForm allows those with current systems, that are not that powerful or on one hdd, to capture near lossless HD on their current systems for $250.

Would still say that anyone building a new mid to high end system should go for the raid0 with three or four drives, for not much more $$$. Then you have a lot of diskspace to work with and can capture lossless

You're still kind of missing the point here. CineForm offers compression approximately 10:1 going up to 15:1 for less demanding video. 1080p30 8-bit 4:2:2 is approx 144MB/s. For every one minute of video on your hard disk capturing uncompressed, CineForm will provide 10-15 minutes. This allows for realtime editing of multiple video streams simultaneously which you won't be able to do with your RAID array. CineForm using one disk will hold more video than your four disk RAID array. It's a mathematical fact. There's also a question of value for time, as well as value for money. Working with CineForm files is much faster than working with uncompressed. Edits will be completed much more quickly.

You could use Huffyuv, but you're still only looking at 2:1 to 3:1 compression there and even then, Huffyuv can't decode in realtime making it useless for editing.

Plus of course a 3-4 disk RAID0 array is 3-4 times more likely to lose your data due to a drive malfunction too.

And even if you had pure 4:4:4 RGB, would would be the point? There are no encoders that support it (at a normal consumer price). Using x264 drops you to 4:2:0 YV12 no matter what, so you lose the extra quality.

Well that's a really poor argument. x264 won't retain the quality of your uncompressed capture, so you might as well capture with CineForm as the raw input you'd be giving the x264 encoder would be virtually identical any way.

[Seraphic-]

Well that's a really poor argument. x264 won't retain the quality of your uncompressed capture, so you might as well capture with CineForm as the raw input you'd be giving the x264 encoder would be virtually identical any way.

But are you talking about feeding the raw capture video right into the x264 encoder or into a program like Adobe Premier for editing before moving to the encoder?

Like in my case, I need to use Adobe Premier to add video/audio/image overlay to the video before final encoding. So what I was somewhat unsure of is how to output the file before feeding to the encoder. Be it the same as input format or YV12.

Also, why would he x264 encoder (megui) not be able to retain the raw uncompressed quality of UYVY/YUY2 for input but somehow retain CineForm? Don't understand that.

On a side note. If you have a chance, easy2Bcheesy, what would be a 4:3 resize of about 853x672? (to mimic how it would look with 640x448)

[easy2Bcheesy]

The point is this. The output of x264 is never going to match the quality of your uncompressed capture, unless you use the lossless profile which is going to require inordinate amounts of bandwidth. It's never going to match CineForm either. So you may as well take advantage of all of the plus points of a CineForm workflow since your final video is going to be inherently compromised. There are so many compression variables at work with x264 that the difference in the quality level between the input provided by lossless and that of CineForm is basically irrelevant.

I have no idea what you are asking about Premiere and how it would be different from what you must be doing now any way. You'd simply export as a CineForm AVI, then use an Avisynth script for MeGUI exactly as you do now any way.