HOWTO: Swift & painless server migration using Admin backup/restore

[Jackiegoal]

We, as a company, migrate DA servers on a very regular basis. Sometimes because a server is past its expiration date, sometimes because clients move to our services. Anyway, because we feel we've conquered this quite nicely so far, I decided to share our knowledge and maybe start a discussion on how to improve this technique.

It's just a basic checklist based on many other howto's and knowledgebase items, but I'll try to make it a comprehensive set of "tools" making sure you minimize your downtime.

We start four to six hours before our server is at its quietest by lowering the TTL. As DA support has a guide for this, I decided not to go too deep into this. Don't lower it to 100 though, lower it to 301. Some access providers tend to overrule a TTL lower than or equal to 300, 301 has the best results, although I must add to that that these experiences are with Dutch APs.

These four hours give us plenty of time to setup the new server. Of course, you have experience with setting up a server and you will want to do it your way, I respect that. Basically, what we do is we install our OS (CentOS 5), do a custombuild DA install and proceed from there. Add all the IPs you need to the server in advance. Then you could look into using ELS as your basis for securing the server. After that I would advise on securing Exim with ClamAV, SpamAssassin and SpamBlocker (an obsolete version is currently provided with DA, look here for the current stable version (v2) or the latest beta (v3)). Guides can be found on the forums. But again, this is just one of many ways. Of course, install the packages you need, you'll know better what you want.

Fine, then you'll have some spare time left, get a nice cup of coffee, start testing whether everything is setup properly. Make a test reseller account and try giving it two nameservers and one user (with a domain) on those nameservers. Check whether they are functioning normally using your favorite nameserver checking service (Google will find it for you). We've had some problems with bind/named in the past, so I always advise to do so. If everything works, remove the reseller and the user(s).

At least four hours should have passed and it should be an off-peak time by now. If not, get another cup of coffee and go do something useful.

Right before your start backup up, there's a very important decision to make: whether to shutdown httpd/mysql/exim/dovecot etc. on the old server or not. Basically, that's a choice between downtime and information loss. And it's your choice. I always stop all services just before I make the backup, inform all clients on it two days in advance and depend on my mail relay service for relaying e-mail. But again, it's your choice. It should all be offline for no longer than 30 - 40 minutes.

Now, we'll start by backing up all accounts on the old server. I don't know how your server is setup, but I always advise not to transfer the admin account along with the other accounts. If you're experienced with DA, you'll know not to use the admin account too much and use a reseller account for all the rest, but if you didn't, well, you'll have to backup the admin account as well. Use the admin backup/transfer tool. System backup is nice to use for redundancy, but it's very hard to restore a system backup, so this way is better imho. You can have DA send the backup to the new server using FTP, but you could also have DA just focus on the backing up process and throw each individual backup to the other server as soon as it's done. The last way is probably faster as DA doesn't backup and transfer at the same time (thanks to jlasman for pointing that out!).

Login to the new server and wait until the old server's DA tells you the backups have been successful.

After the backup's done, start restoring the backups on the new server. Make sure you check that the new IP is being used for restoring. If certain users have their own IPs, restore them individually or change it later on. DA will have it done quite fast in my experience, but it could take a while longer or fail. If it fails, make sure to restore the failed accounts again.

Then, as fast as you can, fire up an SSH session to the old server and change the IPs in the DNS configurations. This is to make sure that all traffic is now going to the new server, even if an AP overrided the new TTL. To do that, use the following steps:

# cd /var/named
# perl -pi -e 's/1.2.3.4/4.3.2.1/' *.db
# service named restart

1.2.3.4 is your old IP, 4.3.2.1 is the new one. Make sure you execute the perl command for every IP that needs to be changed, so also for the nameserver IPs and dedicated IPs.

Just to be sure, stop all processes on the old server except for DA, named and SSH. To speed up the process, you could change the IP for the domain that has the nameservers at your registrar's. After you have made sure everything resolves fine, you can do with the old server whatever you want.

After making sure everyting works like it should, you're done. Great stuff, give yourself a nice pat on the back!

 

[nobaloney]

Great work!

Sometimes because a server is past its expiration date

I threw away some milk yesterday because it was past it's expiration date, but I've never thrown away a server for that reason. Do you mean your server is old and you've decided to migrate to a new server?

We start four to six hours before our server is at its quietest by lowering the TTL. As DA support has a guide for this, I decided not to go too deep into this. Don't lower it to 100 though, lower it to 301. Some access providers tend to overrule a TTL lower than or equal to 300, 301 has the best results, although I must add to that that these experiences are with Dutch APs.

Here in the U.S. we've always lowered TTL to 600, and left it there forever. Ten or more years ago such a low TTL was thought to cause too much traffic on the 'net, but these days it's a reasonable practice, and enables us to change IP#s as well, if necessary, without advanced planning.

SpamBlocker (an obsolete version is currently provided with DA).

Hopefully your excellent How-To will last long enough into the future so that you might want to avoid the use of the word currently. DirectAdmin will soon be using a newer version of SpamBlocker, but the latest recommended version will always be available from my site. Would you consider giving two links in your How-To? One for people who want the latest official version, and one for people who want the latest beta version? Or perhaps three versions, adding a link to my latest bleeding-edge version?

You might also want to add a note about updating/customizing SpamBlocker; it's really necessary and the latest beta and bleeding-edge versions are commented extensively; much more so than the official version, to make it easy to do.

Don't store the backups locally, but send them to the new server's admin backup space using the FTP function.

We've found that this doesn't save time; it doesn't move the previous backup while backing up the next. So if you save locally and FTP (or SCP for security) backups manually, you'll save time.

Then, as fast as you can, fire up an SSH session to the new server and change the IPs in the DNS configurations. This is to make sure that all traffic is now going to the new server, even if an AP overrided the new TTL.

Since restoring on the new server will restore with new IP#s in DNS, I'm wondering if you didn't mean fire up an SSH session to the old server. Please clarify.

Also, you didn't specify how you handle the possibility of people using sites on the old server, while doing the backups, which could create database changes which won't be copied over. Have you tried suspending sites on the old server before backup and unsuspending them on the new server after restore, and if so, does this work for you?

Have you considered turning of the ftp daemon on the old server before starting backups so users can't upload changes?

Again, great work!

Jeff

 

[floyd]

What I do is suspend a user before backing up. If its not too many users I will suspend all of them and back up all of them.

Also I change the suspend message to state that the user and domain is being moved to another server and to check back in a little while.

If its going to take long time to back up everybody then I will start with all the users that begin with "a" or maybe "a" through whatever.

If you backup locally first and then scp over you can do the transfer and restore on the new server while DA is backing up the next user.

If you have control over the router or you can get the DC to work with you then you can use an extra ip and assign it to the old server, assign the users to the new extra ip, have their web sites answer to both the old and the new ip for a day and then when you do the back up and transfer you can transfer the ip as well, clear the arp on the router, and have virtually no downtime at all.

If you don't clear the arp then the ip may not get routed to the new server.

The key is good planning. Write it down in advance if you have to that way you don't have to think about it while you are doing it.

 

[nobaloney]

What I do is suspend a user before backing up. If its not too many users I will suspend all of them and back up all of them.

I haven't done this because I didn't know if the transfer of a suspended site would work properly. It's good to know it would. Perhaps we'll try this if/when we have to move sites again.

Also I change the suspend message to state that the user and domain is being moved to another server and to check back in a little while.

Great idea!

If its going to take long time to back up everybody then I will start with all the users that begin with "a" or maybe "a" through whatever.

Another great idea!

If you backup locally first and then scp over you can do the transfer and restore on the new server while DA is backing up the next user.

I believe I mentioned this; downtime is a lot less than if you have DirectAdmin move everything.

If you have control over the router or you can get the DC to work with you then you can use an extra ip and assign it to the old server, assign the users to the new extra ip, have their web sites answer to both the old and the new ip for a day and then when you do the back up and transfer you can transfer the ip as well, clear the arp on the router, and have virtually no downtime at all.

I'm thinking about this, as we have our own routers and switches, and enough IP#s. However, I don't understand how this would keep users from writing to the old database while you were moving to the new. Has this ever been a problem for you?

If you don't clear the arp then the ip may not get routed to the new server.

That's a good point but in our situation the switches are where the ARP is cached, and they clear themselves (though I'm not sure of how often; I'll have to verify that with our network administrator (but not today; he deserves his holiday).

The key is good planning. Write it down in advance if you have to that way you don't have to think about it while you are doing it.

Yet another great idea!

Jeff

 

[floyd]

What I do is suspend a user before backing up. If its not too many users I will suspend all of them and back up all of them.

I haven't done this because I didn't know if the transfer of a suspended site would work properly. It's good to know it would. Perhaps we'll try this if/when we have to move sites again.

It works. You just have to unsuspend the user after doing the restore.

If you have control over the router or you can get the DC to work with you then you can use an extra ip and assign it to the old server, assign the users to the new extra ip, have their web sites answer to both the old and the new ip for a day and then when you do the back up and transfer you can transfer the ip as well, clear the arp on the router, and have virtually no downtime at all.

I'm thinking about this, as we have our own routers and switches, and enough IP#s. However, I don't understand how this would keep users from writing to the old database while you were moving to the new. Has this ever been a problem for you?

I guess I should explain more. I still suspend the user before backing up so that nothing new gets written. But time is saved from the user's point of view because even though you have moved his site over and its working he still thinks it down because his isp has cached the old ip. But by changing the ip in advance and then also moving the ip to the new server the cache is not a problem since the ip has not changed during the move. The only downtime is when the user is suspended during the backup and restore.

 

[nobaloney]

Thanks! Good idea. While we use a default TTL of 600, and we've never found an ISP that doesn't honor it, local routers, desktop systems and even browsers can have the problem.

How do you have the sites answer to both IP#s? Do you do that using your firewall to reroute the new IP#? Or do you make changes to the httpd config for each domain?

Thanks!

Jeff

 

[floyd]

Now what if the server has crashed or the hard drive is going bad and the server is now unbootable? In my experience most of the time its just a few files that are making it unbootable. I have had fsck really screw up some files and make things worse.

Correct me if I get some terminology wrong. Hopefully everyone will get enough to follow it even if I get some terms wrong.

You have to be physically at the server to do this. Boot with the cd into rescue mode. Let it mount the system. chroot to the server environment which for me is "chroot /mnt/sysimage" Then start mysqld. Make sure /home/admin/admin_backups exists. Then:

echo "action=backup&local_path=/home/admin/admin_backups&owner=admin&type=admin&value=multiple&when=now&where=local&who=all" >> /usr/local/directadmin/data/task.queue

While you are waiting for it to finish backing up start building your new server.

In the end you should have a backup of all your users. You can check the latest ticket in /usr/local/directadmin/data/tickets to see if there were any problems backing up.

I have successfully done this several times. Once I did have to backup mysql separately. mysqldump --all-databases -uda_admin -p

 

[floyd]

How do you have the sites answer to both IP#s? Do you do that using your firewall to reroute the new IP#? Or do you make changes to the httpd config for each domain?

I create a /etc/httpd/conf/vhosts directory and before make the ip change I copy the users httpd.conf file there and name it user.conf and the in the main httpd.conf I have it include all the files in vhosts directory.

 

[nobaloney]

That will work ... but I think it would be easier to just use the firewall (on Linux, iptables) redirect the new IP#. I'm going to discuss this with my network admin.

Jeff

 

[floyd]

That will work ... but I think it would be easier to just use the firewall (on Linux, iptables) redirect the new IP#. I'm going to discuss this with my network admin.

Jeff

Easier for people who know a lot more than me which that could be a lot of people. I am not familiar with the above.

 

[floyd]

Its also a good idea to run mysqlcheck on all the databases to analyze, repair, and optimize. Look for any problems. Fix any problems before trying to backup.

 

[jackc]

does DA able to backup everything with configuration files except /home/user and mysql?
I was hoping to be able to do that so I can just use DA to restore all the users with their configuration files and then just tar up /home and /var/lib/mysql
it's much faster than backup & restore users.

 

[Jackiegoal]

Great work!

I threw away some milk yesterday because it was past it's expiration date, but I've never thrown away a server for that reason. Do you mean your server is old and you've decided to migrate to a new server?

Of course I mean that. I'm not native English, neither have I ever lived in an English speaking community. I haven't even had sufficient English lessons at highschool, so my expression will be off quite a few times.

However, we actually handle expiration dates. Most of our servers are up for replacement after 3 years. That's also taken in account when setting prices and stuff...

Here in the U.S. we've always lowered TTL to 600, and left it there forever. Ten or more years ago such a low TTL was thought to cause too much traffic on the 'net, but these days it's a reasonable practice, and enables us to change IP#s as well, if necessary, without advanced planning.

Most of the people who actually need this HOWTO will not have their TTLs set to anything but the DA standard and after the migration a new "out-of-the-box" DA install will set every transferred account with the standard TTL. It's just a bit of common sense that I include this information.

Hopefully your excellent How-To will last long enough into the future so that you might want to avoid the use of the word currently. DirectAdmin will soon be using a newer version of SpamBlocker, but the latest recommended version will always be available from my site. Would you consider giving two links in your How-To? One for people who want the latest official version, and one for people who want the latest beta version? Or perhaps three versions, adding a link to my latest bleeding-edge version?

You're absolutely right, I will add those links right away. That will make this HOWTO a bit more interchangeable as well, as people on the forums will probably already know where to find SpamBlocker.

You might also want to add a note about updating/customizing SpamBlocker; it's really necessary and the latest beta and bleeding-edge versions are commented extensively; much more so than the official version, to make it easy to do.

Maybe, but linking to SpamBlocker should do the trick. Documenting SpamBlocker is more your thing, I feel.

We've found that this doesn't save time; it doesn't move the previous backup while backing up the next. So if you save locally and FTP (or SCP for security) backups manually, you'll save time.

I've actually put quite some thought into this. I felt this way to be more straight-forward, but lately we have had some issues with DA spitting up error messages during such an automated transfer. I might add the alternative way to the HOWTO.

Since restoring on the new server will restore with new IP#s in DNS, I'm wondering if you didn't mean fire up an SSH session to the old server. Please clarify.

You're absolutely right. You see, writing in a second or third language can make it a little bit harder to spot mistakes in your own text, so this feedback is absolutely necessary

Also, you didn't specify how you handle the possibility of people using sites on the old server, while doing the backups, which could create database changes which won't be copied over. Have you tried suspending sites on the old server before backup and unsuspending them on the new server after restore, and if so, does this work for you?

Yup, we've tried that. You can even, as floyd suggests, change the suspend message. It's a way. We have different policies for different servers. Some servers rely on uptime, some rely on data accuracy. It's important to make sure which way is right for you. Suspending and unsuspending is one way, shutting down certain services is another (you could argue e-mail and database services should always be shutdown). This HOWTO is not the only way, it's just a way.

Have you considered turning of the ftp daemon on the old server before starting backups so users can't upload changes?

I now see the order of the HOWTO is a bit off. Shutting down those services should be done before the backups indeed.

Again, great work!

Thank you Jeff, it's just my 2c of course, all additions to this are more than welcome. Thank you for yours!

 

[Jackiegoal]

does DA able to backup everything with configuration files except /home/user and mysql?
I was hoping to be able to do that so I can just use DA to restore all the users with their configuration files and then just tar up /home and /var/lib/mysql
it's much faster than backup & restore users.

Well, there's the system backup functionality. I have always felt it's not really suitable for server migrations because there's not really a fast way to restore those backups. System backups are better suited for backing up just what you need as a sysadmin at a specific moment, as it can be used to backup practically everything.

There's always the option of a manual backup, but I feel this is the only way to migrate a live server as it can be done in under an hour most of the times.

 

[floyd]

does DA able to backup everything with configuration files except /home/user and mysql?
I was hoping to be able to do that so I can just use DA to restore all the users with their configuration files and then just tar up /home and /var/lib/mysql
it's much faster than backup & restore users.

There is a work around. Untested and is my theory only.

Rename the user to something else and then create a new user directory that has the right directories but they are all empty. That DA does not error out. You can do the same thing for mysql too.

 

[Jackiegoal]

floyd, as I said, your suggestions are greatly appreciated, it's always good to improve these kind of "routine" jobs, which is another reason why I wrote this HOWTO. It can get the discussion going.

If you have control over the router or you can get the DC to work with you then you can use an extra ip and assign it to the old server, assign the users to the new extra ip, have their web sites answer to both the old and the new ip for a day and then when you do the back up and transfer you can transfer the ip as well, clear the arp on the router, and have virtually no downtime at all.

You are absolutely right and if you're moving users one by one you will actually have every user experience a shorter downtime this way. However, it's important to realize that the real downtime is not in the IP switching, it's in the time between the start of the backup and the actual restore. I agree that your way is better and if you have the possibility to do it your way, you're absolutely right to do so. I don't believe, however, that it does anything for a sysadmin who migrates his server in one big swoosh. From the first backup to the last restore (not really true, but that's a technicality) all domains on the server have to be unaccessible or at least FTP, MySQL and Exim should not be running.

 

[floyd]

I am just throwing out options. As you also agree there is no one right way to do it. It all depends on the situation. Some like to backup the entire server and move it at once. If there are a lot of users using a lot of disk space running a backup could take all day. I had one doing daily scheduled backups where the next backup run would start before the first one finished. I had to stop doing daily backups on that server until I moved some people. When I have to retire that server I will have to move people one at a time or maybe 5 at a time. I am glad I have a over 2500 ip addresses.

I know the real downtime is in the backup. But transferring the ip as well will reduce the downtime a few more minutes. I do it both ways. Sometimes I transfer the ip and sometimes I don't. It depends on the situation.

all domains on the server have to be unaccessible or at least FTP, MySQL and Exim should not be running.

That is why I say suspend the user before backing up. But not all the users, only the ones that are currently being backed up.

There is an easy way and a hard way. The easy way requires more downtime. For less downtime the job becomes harder and more to keep track of.

I am adding to your post, not taking anything away.

 

[Jackiegoal]

I'm not saying you're taking anything away, I'm just trying to explain why I chose these steps specifically. Again, I appreciate your input.

This HOWTO is not for the experienced multi-server administrator, again, it's for the ones that have a single DA server running and need a basic idea on how to do a successful server migration. Your way is great, I'm only trying to explain why I didn't include a step like that in the HOWTO itself.

 

[floyd]

Sorry. Maybe I should start a thread "Swift & painless server migration using Admin backup/restore for the advanced admin."

 

[nobaloney]

I've thought some more about the use of an IP# (or more) on the old server and then moving that same IP# to the new server.

I'm no longer sure I want to do this, since we currently have each server on it's own virtual subnet, for security purposes.

I know you can have multiople virtual subnets on the same server but I've never done it. Has anyone had experience with this?

Jeff