Miscellaneous Operating Systems/Hardware

The plural of "anecdote" isn't "data", but I'll add another anecdote to the pile anyway.

At home I have a 16-disk array consisting of 8x 750GB, 4x 1TB and 4x 1.5TB SATA drives configured in mostly-RAID10 using ZFS. I can't say I've had any more issues with the larger drives than the smaller ones - all 16 seem to be fully functional. However, once I exceeded the PSU's maximum 12V current output, the 1.5TB units dropped offline first. I'm not sure why - maybe they need more juice, or are just more sensitive to power fluctuations. That said, after installing a beefier PSU, they came back without any permanent damage. By the same token, this server only gets used by 5-7 people, and doesn't get hit all that hard.

As for how to set it up, I'd also advise against RAID5 unless the data is read-mostly, and even if it is, I'd prefer RAID6 over RAID5+hotspare. (RAID5 without a hotspare is, IMHO, too risky unless there are /very/ frequent backups). Space-wise, RAID6 will get you 10.5TB - which you'd need 14 disks to match with RAID10. As much as I prefer RAID10 for both performance and reliability reasons, RAID6 might make more sense in your case if you can't afford/use 14 disks.

It may be too late now, but I've heard it suggested that your drives should come from a mix of manufacturers, to reduce the odds of multiple drives puking simultaneously due to the same manufacturing fault/firmware bug/etc. Honestly I don't know how much this helps, but intuitively it makes sense.

As for filesystems, that's always a question. Recently I've been partial to ZFS, but if you're using Linux, and can't use FreeBSD or Solaris, XFS is a very good choice. ext4 or jfs might work well, too, but I'd have to look at some benchmarks and such to be sure. (Also, ext4 is the least well tested of the lot. It does seem to deserve the 'stable' label, but that's still something to consider)

Just FYI, ZFS RAID does not perform quite as well as Linux software RAID or a RAID controller, despite its oft-hyped variable stripe width. While ZFS shines in many areas (replication, snapshotting, quotas, and so on), its RAID is not one of its strong points. The recovery and hot-spare support in RAID-Z is also relatively new and unproven, and for quite a while was just flat-out broken. I wouldn't trust RAID-Z with mission-critical data, especially as it's prone to loss if the ZFS metadata gets damaged.
Of course, there's nothing that keeps you from running ZFS on a controller-backed volume besides cost.

jade_angel wrote: As much as I prefer RAID10 for both performance and reliability reasons, RAID6 might make more sense in your case if you can't afford/use 14 disks.

If you don't mind a massive overhead in parity calculations, RAID-6 does the job.
RAID-DP (dual-parity in NetApp) gets around this problem by doing all the calculations in NVRAM and write all the data on all the involved disks *once*, and that's it.

It may be too late now, but I've heard it suggested that your drives should come from a mix of manufacturers, to reduce the odds of multiple drives puking simultaneously due to the same manufacturing fault/firmware bug/etc. Honestly I don't know how much this helps, but intuitively it makes sense.

There's a good point in that, since that would have saved us from our RAID-6 failure when three drives died.
Syncing up a RAID-6 raidset takes forever and that added load on the drives took the other drives as well...

SAQ wrote:

mila wrote: On a AMD backup server I have had two 1TB Seagate SATA drives go down in a few days in a single raidz2 6 disk pool, I had one spare drive on the shelf popped that in
and then the second drive failed after a few days that was scary waiting for more spare drives

Haven't made the terabyte leap yet for (among other reasons) the fact that they seem to die much more often than smaller drives.

How long had they been running? (i.e. is this something that should have been caught in factory QA if they still did QA, or was it something that cropped up after install+test+a noticeable amount of use - 6months or better).

I would say 1.5 year, Seagate 1TB ES drives. There could be issues with FW(or ZFS!), since I have managed to use the two failed drives again!

/michael

bri3d wrote: Just FYI, ZFS RAID does not perform quite as well as Linux software RAID or a RAID controller, despite its oft-hyped variable stripe width. While ZFS shines in many areas (replication, snapshotting, quotas, and so on), its RAID is not one of its strong points. The recovery and hot-spare support in RAID-Z is also relatively new and unproven, and for quite a while was just flat-out broken. I wouldn't trust RAID-Z with mission-critical data, especially as it's prone to loss if the ZFS metadata gets damaged.
Of course, there's nothing that keeps you from running ZFS on a controller-backed volume besides cost.

Actually, I thought about doing that - my array is using an Adaptec 31605 SAS RAID controller right now, but I'm running it in JBOD mode. I'm using software mirrored pairs of disks right now, since that gives me the ability to swap out disks for more capacity (I did that in this array's predecessor, swapping 4x250 for 4x750 on the fly), but I was thinking about using the controller since I'm rebuilding the system soon anyway, once I get back from deployment. I thought of maybe setting the drives up as 4x1000 RAID5 + 4x1500 RAID5 + 4x(2x750) ZFS mirrors, but maybe that makes no sense. I'm still thinking and doing the math. (The array is read-mostly, and I can easily add an OCZ Vertex SSD as a separate ZFS intent log device, which will massively speed up writes)

This article explains the whole issue in a 12TB+ RAID-5 array: http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162

jade_angel wrote: The plural of "anecdote" isn't "data", but I'll add another anecdote to the pile anyway.

Not apropos of anything in this thread, but that is the best quote-worthy line I have seen in a while.

Well played!

tunkaflux wrote: This article explains the whole issue in a 12TB+ RAID-5 array: http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162

wow, interesting. thanks for sharing

eh, Robbin Harris is a well known boob. his treatment of the facts in this case is typical of the sensationalist journalism he's known for. for modern drives you ARE fine with RAID 6, now and going into the future until Unrecoverable Read Error Rates are revised in the next technology bump.

btw EVERYONE in the industry knew this 5-6 years ago. that's how long it takes to become 'general knowledge'.

skywriter wrote: eh, Robbin Harris is a well known boob. his treatment of the facts in this case is typical of the sensationalist journalism he's known for. for modern drives you ARE fine with RAID 6, now and going into the future until Unrecoverable Read Error Rates are revised in the next technology bump.

btw EVERYONE in the industry knew this 5-6 years ago. that's how long it takes to become 'general knowledge'.

Thats also why Sun started working on ZFS for 5-6 years ago since it takes along time to get a mature product, and a filesystem is really a mission critical thing you can't just replace it in a matter of hours it takes days!

/michael