danellison
Verified User
I have read many threads in the forums relating to the sysbk utility and how to restore a full system. So far the only answer to how to restore had been "manually." I didn't really like that solution too much so I developed the embedded script (below) for automatically maintaining an image of your active server on a backup system. In my case the backup system is a symetrical heartbeat cluster with DRBD replication onto a "backup" backup server (there cannot be too much redundancy in my book.) The backup cluster is remote to the active server and I use the scp method for my daily full backups.
The script is highly configurable to allow full restore or only specific parts of the restore to be implemented by just changing variables from "no" to "yes" and vica-versa. In it's current incantation I do NOT image /etc/sysconfig but should be no problem to add it,
I make no warranty or representation of one if this script breaks your installation (but it works for me
) My platform is centos-4. Slight modification could (likely() be needed for other non-redhat installations.
As presented this script will operate in "dry run" mode so you can see
what it WOULD do if DEBUG is set to "off"
My script:
I hope this is formatted such that it can be cut and pasted with a minimum of fixes. I tried anyway! If not - let me know and I will put it up for download on my server somewhere. This script is certainly not as efficient or elegant as it could be - but it works! Hope some of you find it useful.
Regards,
Dan Ellison
Concepte of Illinois
The script is highly configurable to allow full restore or only specific parts of the restore to be implemented by just changing variables from "no" to "yes" and vica-versa. In it's current incantation I do NOT image /etc/sysconfig but should be no problem to add it,
I make no warranty or representation of one if this script breaks your installation (but it works for me
) My platform is centos-4. Slight modification could (likely() be needed for other non-redhat installations.As presented this script will operate in "dry run" mode so you can see
what it WOULD do if DEBUG is set to "off"
My script:
Code:
#!/bin/bash
#
# live_restore.sh - pull the data out of our daily backup and into
# the right places on our
backup systems
#
# Author: Dan Ellison Copyright (C) 2008 Concepte of Illinois
#
# Published under the GPL license - do with it as you wish
# and don't blame me/us if you get different results than
# you might have expected - this was developed in the centos-4
# environment and will likely need some tweeks on non-redhat
# platforms.
#
# I use the following custom.dirs/custom.files in my application
# using directadmin v1.32.2's sysbk function - I list each reseller
# individually instead of backing up the entire /home tree
#
# /usr/local/sysbk/mod/custom.dirs:
# /var/named
# /var/log
# /var/mail
# /var/spool/cron
# /var/spool/mail
# /var/spool/virtual
# /var/www
# /etc/mail
# /etc/virtual
# /etc/ssh
# /home/admin
# .....
# /home/whatever-individual-resellers-you-have
# /usr/lib/apache
# /usr/local
# /usr/share/ssl
# /srv # my svn tree - you might not need
#
# Could include /etc/sysconfig and then exclude it unless FULL_RESTORE
# is set to "yes" - if only there was more time in a day!
#
# /usr/local/sysbk/mod/custom.files:
# /etc/exim.conf
# /etc/exim.cert
# /etc/exim.key
# /etc/exim.pl
# /etc/group
# /etc/gshadow
# /etc/hosts
# /etc/httpd/conf/httpd.conf
# /etc/httpd/conf/ips.conf
# /etc/named.conf
# /etc/passwd
# /etc/proftpd.conf
# /etc/proftpd.passwd
# /etc/proftpd.vhosts.conf
# /etc/resolv.conf
# /etc/shadow
# /etc/ssh/sshd_config
# /etc/system_filter.exim
# /usr/local/directadmin/conf/mysql.conf
#
# Your backup cronjob needs to generate a file backup_sem containing
# the name of the directory created for the backup - ussually the date -
# and place in into the BASE_DIR location (or you could do it manually.)
# I modified my crontab on the production server to include a call to
# a very small script that scp's this semaphore onto the backup server.
# So instead of /usr/local/sysbk/sysbk -q mine
# reads /usr/local/bin/mysysbk
# mysysbk simply runs the sysbk script and then creates and scp's
# the semaphore file to the remote server
#
# /usr/local/bin/mysysbk:
#
# /usr/local/sysbk/sysbk -s && /usr/local/bin/send_backup_sem
#
# /usr/local/bin/send_backup_sem:
#
# echo -n `date '+%m-%d-%y'` >/tmp/backup_sem
# scp /tmp/backup_sem root@backupcluster:/usr/tmp/backups/
#
# Adjust the above to match your setup - trusted keys are presumed
#
# I have successfully imaged our online server onto our active/passive
# remote backup cluster so if needed I can modify the /etc/sysconfig
# scripts to match my production server and use the backup cluster
# directly replacing the production cluster with full reseller/user
# information intact. There is no doubt that there are more improve-
# ments that could be placed here but this works for my purpose -
# a plug-n-play standby server - hope this is useful for some of
# the DA users out there.
DEBUG="yes"
VERBOSE="yes" # whether to echo messages or not
NUM_TRYS=120 # number of time to chec
DEL_ARCHIVE="no" # whether or not to clear old backup
BACKUP_BASE="/usr/tmp/backups/" # where is the backup stored
#
# the following define cluster elements using heartbeat V2 cib
#
# It is assumed that you have all necessary directories linked
# to the right locations with the exception of the /etc stuff
# which we will scp to the inactive node at the end of this
# script.
#
# Otherwise set USING_HA to "no" if not using a heartbeat V2 cluster
#
USING_HA="yes" # are we using heartbeat high availability
MASTER="drbdmaster" # whatever the name of your master is
SLAVE="drbdslave" # whatever the name of your slave is
#
# the following control whether or not the entire brach of the backup
# tree is processed
#
FULL_RESTORE="no" # do a full restore of ENTIRE backup
APACHE_RES="yes" # /etc/httpd/config and friends
MYSQL_RES="yes" # process the mysql tree branch
BIND_RES="yes" # process the bind tree branch
CUSTOM_RES="yes" # process the custom tree branch
#
# these control access to the custom tree
#
USR_RES="yes" # restore /custom/usr tree
ETC_RES="yes" # restore /custom/etc !!caution!!
VAR_RES="yes" # restore /custom/var tree
HOME_RES="yes" # restore /custom/home directories
#
# the following control whether specific archives within the enabled trees
# defined above will be restored
#
# The features in this block will typically only be needed if your
# are about to take the backup image servers (this cluster) to place
# them in active status at our hosting location
#
ROOT_RES="no" # restore the /root directory
SSH_RES="no" # want to change keys?
CRON_RES="no" # restore /var/spool/cron crontabs
LOG_RES="no" # restore /var/log
SPOOLMAIL_RES="no" # restore /var/spool/mail (/var/mail)
RESCONF_RES="no" # restore /etc/resolv.conf (change dns servers!)
#
# The following determines if /usr/local tree gets restored
# This would include /usr/local/lib, /usr/local/directadmin
# etc etc. Might not change frequently but could...
#
# If I use it I would move the existing directory to a temp
# location before restoring, restore and "build all y" to be sure
# EVERYTHING is the same as it was. If in a hurry - put the machine
# cluster into active operation and then do the build. Moral
# of the story - don't make mods on the backup cluster - do them on
# the active one.
#
SAVEUSRLOCAL="no" # move existing /usr/local before populating
USRLOCAL_RES="yes" # restore /usr/local tree (not ussually)
#
# The following block determines whether to restore sql dumps
# or actual database archives.
#
SQL_RES="no" # restore databases from SQL dump
DB_RES="yes" # restore SQL data bases
#
# Most of these actions will want to be done daily....
#
SRV_RES="yes" # restore SVN Repository
ADMIN_RES="yes" # restore /home/admin could be large....
DNS_RES="yes" # restore dns zone records
SYSFILES_RES="yes" # restore /etc/passwd and friends
VIRTMAIL_RES="yes" # restore /var/spool/virtual email
ETCMAIL_RES="yes" # restore /etc/mail files
EXIM_RES="yes" # restore /etc/exim files
PROFTPD_RES="yes" # restore /etc/proftpd files
ETCVIRT_RES="yes" # restore /etc/virtual files
NAMEDCONF_RES="yes" # restore /etc/named.conf
VARWWW_RES="yes" # restore /var/www tree
SSL_RES="yes" # restore /usr/share/ssl tree
HTTPDCONF_RES="yes" # restore /etc/httpd/conf/*
MYSQLCONF_RES="yes" # restore directadmin/conf/mysql.conf
APACHELIB_RES="yes" # restore /usr/lib/apache modules
############# NO MODS BEYOND HERE #############
BASE_DIRS="" # backup tree root - set later
getTree() {
local DIRS=`find $1 -maxdepth 1 -mindepth 1 -type d -print`
echo "$DIRS"
}
myname() {
local name=`echo "$1" |awk '{i=1;
split($0, p, "/");
for(j in p) { ++i; };
print(p[i-1]);}'`
echo $name
}
splitme() {
local myval=`echo $1 |awk '{ split($0, t, "."); print(t[1]); }'`
echo $myval
}
tarme() {
local TAR="tar xzvf "
local DEBUG2="no"
if [ "$VERBOSE" = "yes" ]; then
echo "$1 restore into $2"
fi
if [ "$DEBUG" = "yes" ]; then
if [ "$DEBUG2" = "yes" -a "$VERBOSE" = "yes" ]; then
echo "Extraction command: $TAR $1 -C $2"
fi
else
$TAR $1 -C $2
fi
if [ $? -ne 0 ]; then
echo "Error restoring file $1."
fi
}
restore_me() {
local MYNAME=$(myname $1)
local PWD=`pwd`
# echo "Now processing file: $1."
case $(splitme "$MYNAME") in
root)
if [ "$ROOT_RES" = "yes" -a "$FULL_RESTORE" = "yes" ];then
tarme "$1" "/"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
srv)
if [ "$SRV_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
tarme "$1" "/"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
admin)
if [ "$ADMIN_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
tarme "$1" "/home"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
zone_records)
if [ "$DNS_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
tarme "$1" "/"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
named)
case "$MYNAME" in
"named.conf.tar.gz")
if [ "$NAMEDCONF_RES" = "yes" -o $FULL_RESTORE"= "yes" ];
then
tarme "$1" "$2"
fi
;;
*)
if [ "$DNS_RES" = "yes" -o "$FULL_RESTORE" = "yes" ];
then
tarme "$1" "/var"
fi
;;
esac
;;
sshd_config)
if [ "$SSH_RES" = "yes" -o "$FULL_RESTORE" == "yes" ]; then
tarme "$1" "/etc/ssh"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
ssh)
if [ "$SSH_RES" = "yes" -o "$FULL_RESTORE" == "yes" ]; then
tarme "$1" "/etc"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
cron)
if [ "$CRON_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
tarme "$1" "/var/spool"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
log)
if [ "$LOG_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
tarme "$1" "/var"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
passwd | group | shadow | gshadow | hosts)
if [ "$SYSFILES_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
tarme "$1" "/etc"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
exim | system_filter)
if [ "$EXIM_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
tarme "$1" "/etc"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
proftpd)
if [ "$PROFTPD_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
tarme "$1" "/etc"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
resolv)
if [ "$RESCONF_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
tarme "$1" "/etc"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
mail)
case "$2" in
"/etc")
if [ "$ETCMAIL_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]
; then
tarme "$1" "/etc"
else
echo "SKIPPING file $MYNAME. Restore NOT enable
d."
fi
;;
"/var/spool" | "/var")
if [ "$SPOOLMAIL_RES" = "yes" -o "$FULL_RESTORE" = "yes" ];
then
tarme "$1" "$2"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
esac
;;
virtual)
case $2 in
/etc)
if [ "$ETCVIRT_RES" = "yes" -o "$FULL_RESTORE" = "yes" ];
then
tarme "$1" "$2"
else
echo "SKIPPING file $MYNAME. Restore NOT enable
d."
fi
;;
/var | /var/spool)
if [ "$VIRTMAIL_RES" = "yes" -o "$FULL_RESTORE" = "yes" ];
then
tarme "$1" "$2"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled
."
fi
;;
esac
;;
apache)
case "$TREENAME" in
lib)
if [ "$APACHELIB_RES" = "yes" -o "$FULL_RESTORE" = "yes" ];
then
tarme "$1" "$2"
else
echo "SKIPPING file $MYNAME. Restore NOT enable
d."
fi
;;
*)
if [ "$APACHE_RES" = "yes" -o "$FULL_RESTORE" = "yes" ];
then
tarme "$1" "$2"
else
echo "SKIPPING file $MYNAME. Restore NOT enable
d."
fi
;;
esac
;;
www)
if [ "$VARWWW_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
tarme "$1" "/var"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
local)
if [ "$USRLOCAL_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
if [ "$SAVEUSRLOCAL" = "yes" ]; then
mv -f /usr/local /usr/local.bak
fi
tarme "$1" "/usr"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
ssl)
if [ "$SSL_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
tarme "$1" "/usr/share"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
httpd | ips)
if [ "$HTTPDCONF_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
tarme "$1" "/etc/httpd/conf"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
mysql)
if [ "$MYSQLCONF_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
tarme "$1" "/usr/local/directadmin/conf"
else
echo "SKIPPING file $MYNAME. Restore NOT enabled."
fi
;;
*)
if [ "$2" != "" ]; then
tarme "$1" "$2"
else
echo "Do not know where to restore $1 to...."
fi
;;
esac
cd $PWD # switch back to where we started from
}
function restore_tree() {
local TREENAME=$(myname $1)
local MYFILES=`find $1 -maxdepth 1 -type f -name \*.tar.gz -print`
local MYSUBS=`find $1 -maxdepth 1 -mindepth 1 -type d -print`
# echo "TREENAME: $TREENAME"
case "$TREENAME" in
custom)
if [ "$CUSTOM_RES" = "yes" -o "$FULL_RESTORE" = "yes" ];
then
for f in $MYFILES; do
restore_me "$f" "/"
done
for f in $MYSUBS; do
restore_tree "$f"
done
fi
;;
apache)
if [ "$APACHE_RES" = "yes" -o "$FULL_RESTORE" = "yes" ];
then
for f in $MYFILES; do
restore_me "$f" "/"
done
fi
;;
bind)
if [ "$BIND_RES" = "yes" -o "$FULL_RESTORE" = "yes" ];
then
for f in $MYFILES; do
# echo "Restoring file $1."
restore_me "$f" "/"
done
fi
;;
mysql)
if [ "$MYSQL_RES" = "yes" -o "$FULL_RESTORE" = "yes" ];
then
if [ "$SQL_RES" = "yes" ]; then
# echo "processing SQL restoration."
for f in $MYFILES; do
if [ "$f" -ne "full-mysql.tar.gz" ]; then
restore_me "$f" "/"
fi
done
else
if [ "$DB_RES" = "yes" -o "$FULL_RESTORE" = "yes" ];
then
if [ "$USING_HA" != "yes" ]; the
n
if [ "$DEBUG" = "yes" ]; then
echo "SQL: /etc/init.d/mysql stop"
else
/etc/init.d/mysql stop
fi
else
if [ "$DEBUG" != "yes" ]; then
/usr/sbin/crm_resource -r mysql -p \
target_role -v stopped
mysql_stopped="false"
while [ "$mysql_stopped" = "false" ]; do
mysql_status=\
`/usr/sbin/crm_resource -W -r \
mysql | \
awk '{ split($0, t, ":");
print t[1] }'`
if [ "$mysql_status" != "resource mysql is running on" ]; then
mysql_stopped="true"
else
sleep 10 # give the system 10 seconds to get it together
fi
done
fi
fi
restore_me "$CURRENT_BASE_DIR/full-mysql.tar.gz" "/"
if [ "$USING_HA" != "yes" ]; then
/etc/init.d/mysql start
else
/usr/sbin/crm_resource -r mysql -p target_role -v started
fi
fi
fi
fi
;;
###
#
# second level directory processing routines....
#
###
var)
if [ "$VAR_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; th
en
# echo "processing /var restore operation."
for f in $MYFILES; do
# echo "Restoring file $f into /var dire
ctory."
restore_me "$f" "/var"
done
for f in $MYSUBS; do
restore_tree "$f"
done
fi
;;
etc)
# echo "processing /etc restoration"
if [ "$ETC_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; th
en
for f in $MYFILES; do
restore_me "$f" "/etc"
done
for f in $MYSUBS; do
restore_tree "$f"
done
fi
;;
home)
if [ "$HOME_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
# echo "processing /home restoration"
for f in $MYFILES; do
# echo "Restoring file $f into /home directory."
restore_me "$f" "/home"
done
fi
;;
usr)
# echo "processing /usr restore operation."
if [ "$USR_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
for f in $MYFILES; do
restore_me "$f" "/usr"
done
for f in $MYSUBS; do
restore_tree "$f" "usr"
done
fi
;;
###
#
# third level directories here
#
###
httpd)
for f in $MYFILES; do
restore_me "$f" "/"
done
for f in $MYSUBS; do
restore_tree "$f" "httpd"
done
;;
ssh)
for f in $MYFILES; do
restore_me "$f" "/etc/ssh"
done
;;
lib)
for f in $MYFILES; do
restore_me "$f" "/usr/lib"
done
for f in $MYSUBS; do
restore_tree "$f"
done
;;
local)
for f in $MYFILES; do
restore_me "$f" "/usr/local"
done
for f in $MYSUBS; do
restore_tree "$f"
done
;;
share)
for f in $MYFILES; do
restore_me "$f" "/usr/share"
done
;;
spool)
for f in $MYFILES; do
restore_me "$f" "/var/spool"
done
;;
directadmin)
for f in $MYFILES; do # shouldn't be any
restore_me "$f" "/usr/local/directadmin"
done
for f in $MYSUBS; do
restore_tree "$f" "directadmin"
done
;;
conf)
case "$2" in
"directadmin")
if [ "$MYSQLCONF_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
for f in $MYFILES; do
restore_me "$f" "/usr/local/directadmin/conf"
done
else
echo "SKIPPING file $f. Restore
NOT enabled."
fi
;;
"httpd")
if [ "$HTTPDCONF_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
for f in $MYFILES; do
restore_me "$f" "/etc/httpd/conf"
done
else
echo "SKIPPING file $f. Restore
NOT enabled."
fi
;;
esac
;;
*)
echo "DEFAULT restore_tree: $1"
for f in $MYFILES; do
restore_me "$f"
done
for f in $MYSUBS; do
restore_tree "$f"
done
;;
esac
}
#
# only run on primary node
#
if [ "$USING_HA" = "yes" ]; then
IAMPRIMARY=`drbdadm state home` # replicated /home is assumed
if [ "$IAMPRIMARY" = "Secondary/Primary" ]; then
exit 0
fi
fi
NUM_TRIED=0
SEM_FOUND="false"
while [ "$SEM_FOUND" = "false" -a $NUM_TRIED -lt $NUM_TRYS ]; do
if [ -f "$BACKUP_BASE/backup_sem" ]; then
SEM_FOUND="true"
else
echo "Unable to locate backup semaphore. Retry in 60 seconds."
NUM_TRIED=`expr $NUM_TRIED + 1`
sleep 60 # wait one minute before looking for semaphore again
fi
done
if [ "$SEM_FOUND" = "true" ]; then
BACKUP_EXT=`cat $BACKUP_BASE/backup_sem`
BACKUP_DIR="$BACKUP_BASE$BACKUP_EXT"
BASE_DIRS=$(getTree $BACKUP_DIR)
for i in $BASE_DIRS; do
CURRENT_BASE_DIR=$i # globally know what tree we are in
restore_tree $i
done
#
# sync up files not shared via drbd shared volumes
# like everything that was restored into /etc....
#
if [ "$DEBUG" = "yes" ]; then
echo "DEBUG Run completed."
else
if [ "$USING_HA" = "yes" ]; then
case `uname -n` in
$SLAVE)
DEST="$MASTER"
;;
*)
DEST="$SLAVE"
;;
esac
if [ "$HTTPDCONF_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
scp -r /etc/httpd/conf root@$DEST:/etc/httpd/
fi
if [ "$SYSFILES_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
scp /etc/passwd /etc/group /etc/shadow /etc/gshadow /etc
/hosts root@$DEST:/etc/
fi
if [ "$EXIM_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
scp /etc/exim* root@$DEST:/etc/
fi
if [ "$ETCVIRT_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
scp -r /etc/virtual root@$DEST:/etc/
fi
if [ "$NAMEDCONF_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
scp /etc/named.conf root@$DEST:/etc/
fi
if [ "$ETCMAIL_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
scp -r /etc/mail root@$DEST:/etc/
fi
if [ "$APACHELIB_RES" = "yes" -o "$FULL_RESTORE" = "yes" ]; then
scp -r /usr/lib/apache root@$DEST:/usr/lib/
fi
fi
#
# cleanup
#
if [ -f "$BACKUP_BASE/last_restore" -a "$DEL_ARCHIVE" = "yes" ]; then
echo "Clearing previous backup directory."
# rm -rf $BACKUP_DIR/`cat $BACKUP_BASE/last_restore`
mv -f $BACKUP_BASE/backup_sem $BACKUP_BASE/last_restore
else
rm -f $BACKUP_BASE/backup_sem
fi
exit 0
fi
else
echo "Backup semaphore could not be located. Giving up after $NUM_TRYS attempts."
exit 1
fiI hope this is formatted such that it can be cut and pasted with a minimum of fixes. I tried anyway! If not - let me know and I will put it up for download on my server somewhere. This script is certainly not as efficient or elegant as it could be - but it works! Hope some of you find it useful.
Regards,
Dan Ellison
Concepte of Illinois
