Backup/Restore Scenario

[DAAdmin2005]

Hello,

We have a DA server running in production in a datacenter. We are doing nightly backups and ftp'ing the files offsite. However, we have never actually done a restore nor do we have a solid gameplan if that server goes down. Perhaps others are in this boat.

Here are some things we'd like to accomplish. Maybe we can share ideas on this thread.

1. Prepare a server offsite that is in "warm standby" mode. The daily backups can be restored to the WS server and it can be brought up in a matter of hours.

2. Prepare a "hot standby" server. The HS should be replicated in real-time or near real-time with the production server, so if the main server goes down it is ready to go with an IP switch or whatever. Could be costly.

3. Other ideas.

Please share your ideas especially on #1, or what you are doing to provide some level of disaster recovery or fault-tolerance to your DA server.

Thanks!

 

[hostpc.com]

Well, after Hurricane Wilma this year, our datacenter took a severe hit - as did all of south florida... so ya, I've got a bit of experience in disaster recovery

We're now maintaining servers in two datacenters, with nightly backups in each, and rsync'd backups for customers that pay for added site security in a 3rd, on an hourly basis. Costly? Ya, a bit - but worth it for security.

Worst case, we restore the DA created backups on the servers in the alternate datacenter - change DNS IP's - and we're golden within the time it takes for DNS to propogate.

What we learned from day one however is probably the most important part - keep your customers updated - give them hourly updates - even if there's nothing to report - tell them that.

Use the "MultiServers" option in DA... share your zones with another DA host, that'll help too.

 

[DAAdmin2005]

Hi Joe,

Thanks for the reply. I am going to have a bunch of follow-ups as I try to get this implemented.

One thing I am unsure of. In the case of the warm standby, or what you call "we restore the DA created backups on the servers in the alternate datacenter", do you have a DA license on that WS server?

 

[hostpc.com]

Yep, every server has licenses ready to go

 

[harro]

when you restore a back-up, does that mean the system is as-it-was on the backed-up server (in functionality)

Do you have to separately backup users homedirs and the system, or do you use the "back-up homedirs" option?

Interesting topic, because my experience learns that it never goes as expected (which doesn't mean you shouldn't try to prepare for it ;-) )

Last question: if you go to an alternative server, do you need another license, or can you email DA to ask to switch IP on your license? Since you will only run one working server at a time...

 

[nobaloney]

Joe, do you do multiple reseller backups, or do you do the admin server backup?

Have you ever tested for total restore time?

Have you developed a restore scenario for the admin server backup, or are we still on track to start on that soon?

harro, you can certainly ask DA to change the IP# on your existing license, but as that could take a while, you'd be better off if you had a license purchased in advance, ready to go.

Jeff

 

[hostpc.com]

when you restore a back-up, does that mean the system is as-it-was on the backed-up server (in functionality)

Do you have to separately backup users homedirs and the system, or do you use the "back-up homedirs" option?

Interesting topic, because my experience learns that it never goes as expected (which doesn't mean you shouldn't try to prepare for it ;-) )

Last question: if you go to an alternative server, do you need another license, or can you email DA to ask to switch IP on your license? Since you will only run one working server at a time...

1. I restore all user accounts and resellers exactly as they were at the last backup

2. I use the reseller backup function - it backs up everything including emails/databases/conf/etc

3. You theoretically COULD do that (switch license IP's) - but I dont want to have to wait for that to happen (murphy's law says whatever WILL go wrong will go at the most inopportune time) - weekends, or times when it may take 12-24 hours to get a response for a license modification. I can start restoring immediately and be back online as fast as DNS updates.

 

[hostpc.com]

Joe, do you do multiple reseller backups, or do you do the admin server backup?

Have you ever tested for total restore time?

Have you developed a restore scenario for the admin server backup, or are we still on track to start on that soon?

harro, you can certainly ask DA to change the IP# on your existing license, but as that could take a while, you'd be better off if you had a license purchased in advance, ready to go.

Jeff

My resellers (most of them) have a cron established in their account which backs up all their customers data nightly... then I pull all those backups. Some of my resellers pay for hourly rsync options - even better.

Total restore time for a regular hosting server is around 2 hours max - then again, I dont overload servers. I haven't had the need to test the restore time of a reseller server (thank God) - but I imagine it to be close to that.

As for the admin server backups - Only the necessary data is transferred - user data. If the quotas/counters are reset, that's not a major concern for me - my objective is to get back online asap. I do pull a server wide backup nightly if I need to restore an image - but dont anticipate needing that - most likely it's gonna take time to reset ip's throughout the backup files (from the old, etc) - using the DA reseller backup is much faster, and more efficient.

Fortunately, we work with an excellent datacenter (bocacom) that provides immediate hands on service - I call, I talk to the owner - no questions asked - it's done and done right the first time.