Backup plans for faulty drives

I've been meaning to ask you all how you all deal with downtime if a drive happens to fail.

Do you have a backup server per server? If so, how do you do this with minimal downtime?

Would be interested in anyone's insights.

Well, in my case i use RAID for Disks so if one fail the system will not fail itself.

But, aswell, every weekend i move all-users backups to an external server via FTP.

For the restore in case of doube-disk fail i did never had that, but will mean make a new server up, so i suppose some hours.

That's why i'm stuindy the clustering for those system, so, if one is down, no downtime "show up".

regards

You're just lucky, Andrea. We've had server failures (due to drive failures) twice, both times even though we have RAID running. We still believe in RAID.

On our new servers we backup nightly locally. If a drive fails we'll install a new one into the RAID array. If a server fails we'll take the drives out of the failed server and plug them into a new server (which is available on standby at the datacenter. (And reroute the IP#s if necessary, since we'll be using the original IP#s.)

If the whole RAID array fails we'll restore from backups onto that spare machine.

Jeff

Personally I rent High Availability VPS servers. They automatically move a node when a hardware failure occurs. But I still make daily DA backups as well, moving them every night to a local FTP server, just in case. Every x days I also download them to another server which keeps them in separate folders to build up a retention.

Of course, I do backups of everything every morning religiously. It's just that the technicians in the datacentre could take an age to change a drive. Had a drive fail last July, and it took them just under 24 hours to change.. It's not the data I'm on about, it's the physical work that's done which generates the most downtime.

Arieh said:

Personally I rent High Availability VPS servers. They automatically move a node when a hardware failure occurs.

Click to expand...

But then do you get storage redundancy? It seems to me over years of experience that storage fails more often than anything else.

Jeff

nobaloney said:

But then do you get storage redundancy? It seems to me over years of experience that storage fails more often than anything else.

Jeff

Click to expand...

They have 2 raid controllers per san and hot swap raid 6. But what I like about a provider like this; all they do is manage servers and maintaining their clusters. So if there's something wrong it's routine for them to bring it back up. It's a good solution for me since I don't have too many (virtual) servers.

I think it just comes down to having a good system, if there's something wrong you need to have it fixed fast and no surprises and panicking.

Peter Laws said:

Do you have a backup server per server? If so, how do you do this with minimal downtime?
Would be interested in anyone's insights.

Click to expand...

If you've got to recover data on a new server, and you have no a backup server for it, that might take 3-4 hours to complete the job. Of course there are some factors which might change the amount of time which you might need.

Since 2005 I've done some recover jobs, among the others I changed hot-swap drives to launch the old server on a new hardware; and moved backups to restore them on a new server; re-installed a server.

If to speak about dedicated server, I prefer to use 3 HDDs (hot-swap) platforms: 2 HDDs are used in a mirror (RAID) the third one is either a spare drive or a disk for backups. So you can always move the third drive with backups to another platform if it's needed.

And yes, High Availability VPS servers are a good choice, as you don't need to worry about hardware at all. That's what I use for some projects as well.

all the sites/domains (up to 10 but only 5 with real usage) on server are mine so my setup not typical.
hot swap raid, dedicated drive for backups, and some domains do auto offsite backups for files and databases.
have had drive fail. never even noticed from way sites were running.
sent note to datashack staff and had new drive in place in moments. did take a bit to sync it all but the performance hit wasn't really noticeable.
dedicated drive for backups has saved my butt a few times.
would lve to be rich enough to play with high availabilty load balanced stuff but....not happening LOL

We use raid-10 arrays for all our servers. Also backup daily with barracuda backups https://www.barracuda.com/products/backupservice

If you are poor I would suggest something like http://aws.amazon.com/glacier/

Greetings,

We as well have RAID configured on all our servers. We as well configured nightly backups. I would recommend checking out the folks at Backupsy, there service has been phenomenal, reliable, and is extremely affordable. In fact, there is a 40% off for life coupon right on their homepage. We have been using them for 5 or so months now and have zero complaints.

Can never have enough backups!

^TG

I like to use bacula4hosts which uses pools of cheap backup storage servers. Mainly do it for the Bare Metal Restore Option so I won't have to worry about losing all the tickering I've done to the server over time. Account backups is a nice side effect. I also like the fact that I can restore the server almost anywhere in minutes in case of a total data center getting killed by a killer tornado or something. Only thing to worry about if changing data centers is the IPs and the propagation times.

Unless bare-metal backup also let's you restore individual files, you also likely need a separate solution; youdon't always need to restore an entire machine.

Jeff

nobaloney said:

Unless bare-metal backup also let's you restore individual files, you also likely need a separate solution; youdon't always need to restore an entire machine.

Jeff

Click to expand...

I'm like everyone else, individual accounts are offloaded to remote locations for safe keeping. But with hard drive crashes being my main worry, bare metal restore is the only thing I truly care about. While I don't need to restore an entire machine every time, I don't want to give that up if I had to choose.

bakula4hosts ...

I just looked at the site and it's not as easy for me to see as it should be... are they selling hardware, software, or a service? Where are the backups stored? Locally to you, on your hardware? Or at their datacenter?

Thanks.

Jeff

nobaloney said:

bakula4hosts ...

I just looked at the site and it's not as easy for me to see as it should be... are they selling hardware, software, or a service? Where are the backups stored? Locally to you, on your hardware? Or at their datacenter?

Click to expand...

It is quite confusing at first I must admit. But this is how it works.

Server to be backed up has a small agent installed onto it kinda like R1's solution, but cleaner. It serves up all incremental changes to a storage pool (pool of servers) that you set up preferably at a DC far away. Those servers also get a small agent install on them. bacula4host does not yet provide servers to use for storage but I think it is in the plans one day. I found a great storage server for $7/mo for 250gigs at one DC and a few other good deals in other places. Anyhow, you login to bacula4hosts's GUI and it is from there that you control the going and coming of data. If you have a cPanel server, there are plugins for the end user who can actually get different versions of their files. There is also a whmcs plugin where users can get those same files from right within the billing application. In case you have kvmip, they have bootable media as well.

I did have a hard drive failure on a server which I've paid an enormous amount of money on all kinds of third party scripts and whatnot. I have software installed that while I might have a license to, I'm not quite sure how to reinstall without relying on the authors which would result in more money going outbound to get the server back to the way it was before the HD crashed.

Right after the DC replaced the HD, I installed that agent again and used bacula4host's GUI which put the thing exactly to how it was. Custom kernel and all.

I their forums, I've read that you can even perform a bare metal restore on dissimilar hardware, which I find fascinating. They say its not easy unless its a VPS, but it is possible to do it cleanly.

I chose them for the price. Can't beat it. I stay because the support really rocks and the product lives up to what its advertised.

While it may be confusing at first, its not anymore confusing than running DA for the first time from a background of cPanel. Drastically different but very functional.

r3chn3r said:

I found a great storage server for $7/mo for 250gigs at one DC

Click to expand...

Tell me more . Is that just for storage (bandwidth and processor power extra) or all-inclusive? If for raw storage, I could actually compete (should I just to do so) but if for all exxclusive maybe I should buy from them .

Jeff

Well it isn't a dedicated solution, its a VPS made for storage.
Found them here:
http://8ng.us/storage

Hi

How much cost bacula4host for backup one server in one ofsite backup server ?

The prices on their site are very confusing.

roberto

rpereyra said:

Hi

How much cost bacula4host for backup one server in one ofsite backup server ?

The prices on their site are very confusing.

roberto

Click to expand...

I pay $19 a month for the first server. Each additional server is $5/mo but don't forget the cost of a storage server which you'll have to find elsewhere and it'll need centos on it.