Backup / Disaster Recovery

[bami82]

Hi Everyone,

I have a question about backup and disaster recovery and am intresting to know other people's opinion or sollutions.

Let say I have a server with Debian and DA installed and I have a backup server availlable with enough storage capacity. The server with Debian and DA got 2 SATA disks running on software RAID1 (using LVM).

Now I want to be able to do 2 things.

1. Restore files if someone deletes something on accident.
2. Bare-Metal Recovery in case of a real crash.

I assume that for point 1 most people will recommend using DA backup option from the reseller level (Because admin backup/transfer is still beta).

I am very intrested in point 2. I would say an image of the system is needed (OS+DA) and together with the files from the reseller backups its possible to restore the complete system. How do I make this image? Would it be smart to take 1 harddrive out of the system (its raid1 so it contains a exact copy already) and put in a new one and store this harddrive in a save spot?

Intresting to hear other opinions, please let me know.

Regards,

Sander

 

[nobaloney]

Obviously the only way to do a bare-metal recovery is to rebuild completely with a copy of everything, including OS, kernel, etc.

To do that you need to have a full copy, let's say today. I'm not sure if breaking the RAID and removing the drive will give you a bootable drive or not; you should try it. Why? Because your setup may not give you the boot files (grub/lilo/whatever) on both drives.

Once you have that drive it has to be in a separate machine (to avoid the problems that could occur with a catastrophic failure of the server), and you need to keep all the other files updated; you can do that by rewriting sysbk backup system included in DA to backup the files as it does now, but directly to the right places on the backup drive (the other drive in the other server). Use either rsync, ftp, scp, or nfs, or a combination thereof or write your own system to synchronize files in a reseller backup to the backup drive. You can't modify how the reseller backup works because you don't have the source-code; it's written in compiled C++.

Jeff

 

[bami82]

Jeff,

Thanks for your answer. Our software raid is including grub. Pulling 1 drive out and rebooting it, after sync try the other disk all works fine thats pretty easy to configure with grub, if anyone ever intrested in knowing how to do that just let me know.

I was thinking about storing the drive in a safe place, not inside a system and label it with the name of the server.

With DA backup you mean the reseller lvl backup right? I was thinking about using nfs to mount it over the 2nd NIC (internal network) or just use FTP as the DA backup utility got a standard ftp function. Why Rewrite it? I was thinking about empty DA on the "Image disk" and just restore the reseller backup on that disk if something go's wrong. So in short the plan for emergency would be.

1. Put in disk with bootable image (Incl. Debian and empty directadmin)
2. Recover resellerbackup inside directadmin

Are there people that would do it different, please let me know I am looking for best sollution to do a fast bare-metal recovery. (Rebuild complete system within 5 hours)

Regards,

Sander

 

[nobaloney]

Thanks for your answer. Our software raid is including grub. Pulling 1 drive out and rebooting it, after sync try the other disk all works fine thats pretty easy to configure with grub, if anyone ever intrested in knowing how to do that just let me know.

You're welcome to post your solution under Off-Topic Discussion.

I was thinking about storing the drive in a safe place, not inside a system and label it with the name of the server.

Then how can you update it in real time?

With DA backup you mean the reseller lvl backup right?

I wrote about two solutions, one being the sysbk admin-level backup, and the other being the reseller backup.

I was thinking about using nfs to mount it over the 2nd NIC (internal network) or just use FTP as the DA backup utility got a standard ftp function. Why Rewrite it? I was thinking about empty DA on the "Image disk" and just restore the reseller backup on that disk if something go's wrong.

That works.

Except that you're not backing up any OS updates or DA updates.

But if i were you, I'd "empty" the backup (as you put it) before need.

So in short the plan for emergency would be.

1. Put in disk with bootable image (Incl. Debian and empty directadmin)
2. Recover resellerbackup inside directadmin

Before recovering a reseller backup you have to create the resellers on the new system with all their setup information.

Are there people that would do it different, please let me know I am looking for best sollution to do a fast bare-metal recovery. (Rebuild complete system within 5 hours)

We can get it done in approximately the same amount of time (plus travel time to data center, which you'd need as well) by building a new system, installing DA to that, and restoring our sysbk backups. However as I've pointed out before, restoring sysbk backups requires a high level of sysadmin knowledge (we have RHCEs on 24/7 call).

Jeff

 

[bami82]

Jeff,

I agree, this situation misses the backup of OS patches and DA patches. However this should be fairly easy to do if we keep track of all patch levels. We are very strict on patches and low prio. patches or patches that are not security related are always tested on our development machine, so we got a high lvl of patch control, so if we have all documented shouldn't be to hard to upgrade.

Alternative is perhaps swapping disks at least once in 3 months, that way we always got backup disks

So now I would have something like:

Preparations
1. Replace current harddrive with a new one and resync.
2. For current servers empty DA on a test machine, for new server replace drive
when DA is fresh installed.
3. Save that harddrive in safe place, labelled with disk name
4. Swap disks once in 3 months.

When disaster strikes
1. Insert bootable harddrive with image OS+DA
2. Upgrade with neccesary patches (incl. DA version)
3. Create Resellers Manually
4. Restore backup per reseller

Regards,

Sander

 

[nobaloney]

Test it first .

Jeff