30 Terabyte PC Computer System Suggestions Ideas

No the subject field is not in error.

I have a possible client interested in buying a Windows PC with a storage capacity of 30 Terabytes. It will need to be extremely fast access etc too.

It will be used for a specific application the client wants/wishes, I am sure the price will scare the living hell out him, but still any ideas/concepts?

Thanks in advance

WMS

I've been off SOTA so long I can't really help you but if you don't mind me asking.

What the devil does he need 30 terabytes of highspeed storage for?

16 closed circuit video cameras recording at 30 fps for 365 days at 640 x 480 resolution.

30 1 Terabyte hard drives are ~250.00 each so 30 of them is 7500.00 dollars.

WMS

I fear the case that can hold and cool 30 1 terabyte HDs. Would you need to RAID all that? I'm not clear on the mechanics but that seems like an awful lot of constant writing. And what the hell is he going to do for backups? That seems like an awful lots of eggs in the proverbial basket.

http://en.wikipedia.org/wiki/Network-attached_storage

Try Aberdeen Computers or go directly to SUN...they both sell systems that can take up to 48 drives (or more, in the case of SUN). You will be getting RAID cards of some sort with this which adds to the price considerably. Due to the extreem writes & access you may have to go SCSI or SAS which would pretty much add a zero or two to the tag.
My best advice would be to toss the the customer to EMC (or SUN) for a SAN but, if you really want to do this...put together a simple 5 drive HW RAID and test how many feeds you can reliably get on it. Scale from there.
An alternate option would be to get the large case with a ton of SCSI drives and use Open Solaris/linux with ZFS to pool/control the drive arrays and put windows into a VM on top.
Mr. Man has as great idea with the NAS/SAN and that is most likely the right solution...its going to cost though.

This really isn't the place to discuss this. You'd be far better suited using a computer hardware dedicated site like Anandtech.

As Mr. Man suggested, you're also FAR better off going with some sort of NAS set up than you would be with trying to hook all of this up to a single PC.

Having said that, you're going to need to spend quite a bit more than $7500.00 just on drives. With 30 drives, your chance of individual failure within 1 year gets into the probable range. So, you're going to need to be using at least RAID 5.

It's been a while since I've set up a large RAID, so I'm not sure what the maximum number of drives you can stick into a single array is. Going by memory, I know there's a SATA Promise RAID controller that holds 6 drives. If you set that up to RAID 5, then you'd have 1 extra drive for every 5. Which would mean 36 1TB drives to create six 5TB arrays. You'd have to point separate cameras at separate arrays OR set up your program so that every two months it automagically moved on to the next array.

As far as cases go, you're going to want an external cases for each array. There are a number of decent options out there. Make sure that your cases support hot-swap, so that you don't have to take the whole system down every time a drive crashes and burns. Also

A slightly harder point to address is finding a MB with enough PCI-e slots to accommodate 6 RAID controller cards. Good luck with that one.

Two other things to consider --
1) Windows is going to choke all to hell on this much storage. You're going to want some flavor of Unix that runs a file system more robust than NTFS.

2) Does he *really* need 365 days of feeds all on-line simultaneously? It'd probably be dramatically cheaper in terms of maintenance, electrical, and other costs to get a Blu-Ray Burner and just do a regular archive. By my math, you're expecting a usage of 80-ish GB daily (which seems EXTREMELY low for the amount of video you describe). Assuming no errors, that'd be 4 single-sided BD's per day. Those typically cost around $15 each, so you're looking at a cost of $60 in blank disks.

Finally, about 5 years ago, I was involved in a project to put together a storage system for my department. We were dealing with what were then some extremely large files. So, we decided to put together a TB of storage. This ran into the thousands of dollars and created a massive server with numerous RAID arrays all mounted in it. It worked well, and it was definitely necessary for what we were doing. Now, that server is obsolete, and I can get the same amount of storage for $250. Your contact might consider that he's a couple of steps ahead of himself in technology and that if he can wait on this for a few years, he might be able to pick this up for a couple of orders of magnitude less money.

@The Dunner
The daily needs of 16 30 fps 640 by 480 video is 124.8 Giga. The video is MPEG-4 compression, the best compression of the 3 available options.

RAID 5 does appear to the one way to go.

Thanks I have been pointed to Aberdeen Computers, they have systems that are excellant, pricey but excellant. But I have to find out what if any video is on board and any slots left over.

One idea is to use a SAN/NAS with Ubuntu and SAMBA so windows will see it as storage.

My guess is the client will go into sticker shock on the price and be more "reasonable" in his video storage needs.

WMS

that's an utterly ridiculous amount of time. one YEAR of recorded surveilance tapes?

2 months should be all you need. make it 4 months worth of storage space Just-In-Case, if you don't know if you need it by then, you don't need it.

2 sets of 2 month-total storage drives, swap out every 2 months. have fun replacing dead hard drives.

As I am learning my clients/customers have some "unique" needs at times.

Like one what wants all the camera feeds/network feeds to be fiber optic, and yes for this location and equipment rooms, for this I can see a reason why. This quote request was from a agency with a four letter name abbreviation.

WMS

hmmm...you may want to do some checking at the NSA & CERT sites to see what the standards are then. Things are changing these days and the NSA is now in charge of setting standards for all Fed. Gov. facilities as well as responding to cyber violations. If this is a state agency...well...you can probably do anything you want as each & every state (and agency in each state/city) seems to do things differently.

The Dunner had a good idea with the offline storage...in truth all you really need is a temp storage for the new data and the rest can be offline. If they want a couble of days online that ain't much of a probelm and you can still use SATA RAID. In this case I'd go for RAID 6 or a RAID 50 (5+0) as you wont need much more hardware then a desktop system. The RAID cards will add to the price but, not anywhere near the price of a SAN/NAS that could handle the data/traffic they want.

Well until I can examine the image file structure, and finding out if there any limitations on the DVR image storing software, alot of this is fairly moot.

As for checking with No Such Agency(NSA), getting information out of them, tends to be pedantic at the best of times.

As far as I know TEMPEST is not involved. This is not a state agency so the less restrictive rules/guidelines of such do not apply.

TEMPEST
TEMPEST

But some of the principle tech maybe applied.

I agree that if RAID is needed that RAID 6 or such will be needed.

There is a SATA RAID card that will support 24 RAID Drives but requires a PCI-X slot, which is in mostly in servers but some non servers offer PCI-X, note this is not PCI Express.

I have seen 1.5 TB hard drives available, so 24 hard drives are a possible use. But an external rack or racks would be needed. Along with "spare" hot swappable 1.5 TB hard drives, in case one of the 24 main drives under goes a failure.

Thanks

WMS

for true highspeed access for that size your only choice is a SAN or NAS. EMC and Equalogic have both. I bought one for work today for about 40k (3 tb of sas and 7tb of SATA 750's). You can change the drive config on an AX4-5i to 3 shelves, use 1 terabyte HDD's (will be out ina month tops per EMC) will get you close (4 vault drives, 1 hot spare, and 31 production drives in raid 5). If they ever change thier mind they can always start a datacenter or run a bunch of VM's off of it