Miscellaneous Operating Systems/Hardware

Trying to make a decision on a backup plan.

Currently have a dying NAS that is configured RAID0+1 for "pseudo-versioning" backups (not really versioning, but it will give me "file X as it existed at 21:00 on the last day in November" if I do something stupid). In concert with this there is the primary filesystem (RAID-1 mirrors) and a rotating offsite image backup.

Replacing the NAS, and I was considering initially going with RAID-6 (really the "2-disk redundant" Synology setup) so I can potentially recover from any two drive losses (rather than any 1-drive loss and some 2-drive losses with RAID 0+1), but then I started thinking. What is the likelihood of ever needing that level of redundancy for a "midlevel online" backup? Would I be better off doing RAID-5 (Synology "1-disk redundant)? The NAS (new one is Synology DS413) isn't in a area where it gets "bumped" physically, electrically or thermally - so the risk of loosing everything (i.e. all backups+ original data) is very minor.

I'm not expecting personalized consultation here, just interested in what others are doing. If you have a NAS in your backup plan what do you have it set to?

Hey, we've all got more and more bits in our lives. And unless we just close our eyes and trust in The Cloud, we need to answer these questions.

I feel like the quality/reliability of drives is dropping along with the price. If you didn't have the off-site backup I'd definitely be thinking about the more survivable configurations. I guess part of the question is how frequently that off-site is updated, and what is it's drive failure tolerance, age of disks involved, etc.

I've only got a 1U 4-bay external case for my home/office store. While I'm nervously debating prices on the 3 and 4TB drives versus my now ~3 year old 1.5TB drives, I'm wondering if I don't want more bays for the kind of options you mention... One of those SGI/Rackable Omnistor 3016s comes to mind, though I'm not thrilled at running more drives over the same 4 SAS ports. Though if the expander(s) could use the full 6Gbps/channel to the HBA it might be a wash over the situation today, which is drives limited to 3Gb and/or whatever rate they can stream off the platters...

A NAS is not a backup solution, I use tapes for backup (on multiple OS including, but not limited to, Irix of course).

I don't really back anything up anymore. I stopped caring a couple years ago (when I started saving photos to facebook, have now switched to flickr...) Sure I have time machine running, to prevent "oops" moments but I don't see the need to keep photos etc copied in triplicate on different continents and in space. If I loose my music collection, I can always download it again. Shakira and Frank Zappa aren't going to magically disappear from itunes (or piratebay or the library or the bookstore etc) the instant my hard disk is zapped by lightning.

My Aperture library is about 70gb, maybe 1/5 of it is stored privately on flickr as a backup. The rest of it is stuff that I don't feel need to backup because it's crap, but I don't have the balls to delete straightaway

I also have a disk full of operating system images and software that is either expensive to replace (i.e. I acquired it cheaply as a student or use it daily as a tool) or is simply not available anymore (i.e. some Irix software etc).

Bottom line, if it's something I've created and worth sharing, I've shared it already. If I haven't shared it online or elsewhere, does it really need to be backed up? It's useless to backup if somebody else created it as I can always download it again. Either I have a proof of purchase for the data and can download it again, or I pirated it in the first place.


Ignore this advice obviously if the backup is for business purposes How long do tax records need to be stored? 7-10 years?

+1


Seven.

It can be part of a backup solution - in this case the part to deal with the days when something gets inadvertently deleted, overwritten or corrupted.

The offsite images are for the disaster recovery should we need it.

Yeah, tape's great for backups, but terabyte+ capacity drives are several thousand dollars so we're on hard drives for the high capacity stuff. This setup's going in a nonprofit that I do computer support for, and I don't want to spend my off time tape swapping or trying to teach someone how to do it - with a simple hard drive swap I can get one of the employees to manage the offsite backup.

Heck, I'm on rotating media for backups at home because swapping TZ87's gets real boring real fast.

Do not use raid5 for any purpose whatsoever.

Drives are too big to survive a rebuild these days. raid5 is useless, and beyond 4TB drives, raid6 is getting close as well...

If it's four physical disks, just do raid10, since raid6 gives you the same amount of space, with better performance.

If it is 6 disks or more, use raid6.

If you are from the future and reading this with disks larger than 6-8 TB each, raid6 is useless. Use raidz3 on ZFS or ... whatever exists in 2016

For service storage I'd only consider mirrors or RAID-6. Service filesystems get too much of a beating, and the risks of a error during rebuild of a large RAID-5 are getting too high - and restoring from dumps is not fun. I'd even prefer RAID-6 to RAID 0+1 now for a 4+ drive setup, since RAID-6 allows any two drives to fail, whereas if a mirrored pair fails in RAID 0+1 it's restore time .

The limiter now is still platter speed. At 6Gbps with 4 channels you can throw a lot of drives on there before you start to notice performance loss, unless you're running stuff that fits in the disk caches Disk transfer rates are still in the 200MB/s rate for 15K drives so if you balance across all channels that's what, about 12 drives before you have a potential cut in performance under ideal situations?

If you want to archive to disk, then you should either run amanda (to disk), or a worm-like filesystem, like plan9port's venti/vac or, eventually, plan9's cwfs64.

If that's too exotic for you (I ran 36TB on venti with no problem), then don't do raid, do zfs with 2 parity disks, exported to Irix via nfs. Zfs has block checksum, which LVM/XVM/whatever don't have, then don't forget to collect smartmon data from your drives, you might be surprised.

Don't buy disks too fast (15krpm), they trend to have more *silent* errors than 7.2 or 10krpm, heat and power are also of concern on faster spindles.

I've been using raid5 for 'disposable' data (can be recovered from the 'net, but takes time and effort) and raid10 for 'private' data.


The raid5 failed twice, and it's been able to recover both times. 3TB drives. No issue whatsoever. I did make sure not to put any load on the machine while rebuilding though. 90%+ capacity available for the rebuild, otherwise it'll be hell.

But... yep, once you go beyond 4 drives, a raid6 starts to sound more interesting

I'm thinking I'll go with RAID-6 instead of 10 (even though it's slower) for one big reason:

On the last NAS we had a disk dropping off the array from time to time (not a specific disk, disk worked fine, disk listed as supported for the NAS, we're not sure what was up there). If a disk had dropped off and another failed then there's a 1/3 chance that we'll loose all the data on RAID10 (since we have 3 disks, 2 of which are mirrored and 1 not mirrored). If we're on RAID-6, one disk dropps off and one fails then we can still rebuild.

With a Synology (as opposed to Western Digital) NAS the chances that it will work reliably are much higher, but we can take the performance hit for the improved recovery.

Wow, just hearing this gives me chills.

This part NAS is just for someone who accidentally deletes something or corrupts something. The "real" backups are rotated offsite for disaster recovery. They're not as redundant and they're harder to get at, hence this midlevel layer. There is always a recent copy of the current filesystem offsite, so if the building burns we can be back up the same day we get new equipment. However, people will be out of luck if they also ask for the e-mail they accidentally deleted a week before to be restored.

Yeah, I gave them a scenario for tape, but it would need to be LTO-5 or LTO-6 for single-tape backups, and between the cost for equipment and the discussion of tape rotation strategies interest waned significantly. I set them up with something that will get rotated out regularly for offsite (external hard drives) and will, via the NAS, allow for speedy recovery of userbloopers. High on my list is the fact that it's also easy enough for the people who work there to take care of it once I set it up. I don't begrudge them the time it takes to fix things, but given the choice between coming home after work vs. running over to their spot to fiddle with something on a daily basis...

Hmm, how much (more) of my idiocy do I want to expose here... The two bays in the 1U server are for the mirrored system disks. That only leaves me the four drives to work with - which are in a ZFS raidz1 with weekly scrubbing, in hopes that scrubbing might warn of approaching danger. Going with raidz2 /RAID6 would in this case give up 50% of raw capacity, which is harsh... Not as harsh as failure, but that's always easier to accept afterwards.

Just checked the "Power_On_Hours" and they've actually only been running about 1.8 years - but they are Seagate's of a troubled 2009-2010 vintage/model (ST31500341AS). Some folks have had so much trouble with those drives they'd run screaming right there, but I think there's some luck involved and the reported issues are exacerbated by bad interactions with hardware RAID controllers vs. my 9212 HBA. That said, when I check "Reallocated_Sector_Ct" I see values of [2, 53 , 1, 0]. Hmm - and smartd(1m) was mis-configured until recently, so I don't know if that 53 grew slowly over time, or has recently entered a death spiral... <_<

Well, guess I'd better plan for that transition - the SE3016 would have enough bays to get redundant. I'm unlikely to be looking at anything spinning faster than 7.2k, so I doubt raw bandwidth will be the limiting factor even at 3Gb with 8 drives.

My iMac is my home server with all of my home files and such and I do 4 stages.

1: The data is on a Mediasonic 4 drive FW800 array using raid 1+0 (4x1TB for 2 TB usable, divided into a few volumes).
2: The data volumes I care about the most are watched by Time Machine with a 2TB FW800 single drive.
3: My data is mostly replicated between all of my computers, so at any point, any other computer is a restore source.
4: Every few months I pull out another 2TB drive and do a flat copy of the data.

Seems like a lot but mediasonic array was really cheap and I don't have 100% faith in it, especially now that it's a few years old. I just got that new Mac Mini and will likely move to a tbolt solution on it.

Since drives have become so large, I pretty much only look to RAID-1 these days, and often with software RAID rather than hardware RAID. Besides the usual concerns, some things to also consider:

• What happens when a hard drive fails? How are you notified? If you have software notification through email or some other mechanism, can you really trust it?
• How are these backups being made, and can you trust the backup programs / scripts to never fail? If they do fail, how are you notified?

To me, the difference between RAID-5 and RAID-6 is not very big compared to these other matters, which are often neglected. Each part of the chain needs to be strong, or it won't hold up. Each time I set up a new system that will be used for backups, there are two types of scripts I end up writing: [1] scripts that notify me if something fails, and [2] scripts that send me regular reports about the status of the backup files. These are scheduled to run regularly with cron, and the scripts are always carefully written and tested. For example, all scripts are organized into shell functions, and all functions use local variables and explicitly return an exit status. Commands like tar , cd , etc., should always have their exit status checked. Critical errors caused by any command failing should be caught, and relevant information should be emailed to the appropriate people automatically.

You're forgetting the most important part - get in there and check it out "by hand" regularly (every week or two).

Ah, that's a good point, but I generally try to relegate a lot of that to the daily reports. For example, which filesystems are mounted, the amount of disk usage for each, long listings of any backup files ordered by date (so I can check for any straggling backup files), RAID array statuses, etc. Basically I try to put in the reports anything that I would type into a shell when checking manually, so I can just peruse through an email report and get the gist of the situation. I do log in from time to time for applying system software updates, though.

Random anecdote.... At one time, a guy I worked with had a backup program running on a Windows server that he never really checked, but was responsible for backing up PCs at the company, as well as a fairly important file server. Two of the external hard drives had died without anyone knowing about it, and the last external drive only had backup files from two years before . He only found out about the situation after the machine itself died. When I found out about the system and told him about the multiple levels of failure, he looked pretty embarrassed, so I didn't say anything to make the situation worse. The moral of the story is to keep a close eye on your backup servers.

Is anyone here using Amazon Glacier for his backups? I thought last year about building a FreeNAS array, but making a RAID-Z2 array would require me to buy 6 HDDs and of course all the other parts. Amazon charges $0.01/GB per month so having a terabyte archive would mean $10 monthly and not having to worry about hardware/configuration/power. It's managed through a web service, but if you don't want to write your own client, there are several to choose from. I'm using FreeGlacier , but I can change it at any time, since it's not tied to the service. The only downside is that Glacier works more like a tape backup - you can't just download your files, you request such operation and wait 6 hours for the file to be prepared for download.

Although my main server is RAID5, it's three relatively small 73GB SAS 15K drives, so if one goes boom it should be able to rebuild quickly enough when I get the spare installed.

The file server is also RAID5 with three 500GB SATA drives, with a hot spare online. The main server does a daily filesystem dump to it, which is versioned, plus more frequent snapshots of the mailboxes. My workstation also does a daily dump to it only of specific files. I then throw a copy of the fileserver in an undisclosed offsite location at regular intervals.

Neither have had issues in years, but threads like this make me sweat.