rsync setup

[Duboux]

Hi,

I got a short question.
I believe there's 2 ways to run rsync to backup files on an external backup server..

1. The webserver runs the command to rsync to the backup server
2. The backup server runs the command to rsync from the webserver

Would it save resources on the webserver if the rsync command was run from the backupserver ?

 

[mkniskanen]

As far as I know the rsync task at the sending end will require scanning the file system, packing the files and sending them in both cases. Therefore there should be not much difference performance-wise.

It is a good idea to split the process into several subprocesses which run during a longer time span with pauses in between. I am using the "request update" method and the backup takes place one home directory at a time and the rest of the directories (if needed) in a different batch.

I have two DA servers. When creating a new user I also create an exact duplicate at the backup server end and rsync the home directories once a day, starting the backup at 3:00AM, doing each at one minute intervals. I also back up the /etc/virtual directory and have database backups at /home/<user>/backup/daily/ (and /weekly plus /monthly).

The one missing still is an automated database refresh at the backup server. Once that is running a total primary server crash can be recovered by changing the DNS

 

[zEitEr]

Just my two cents, I've never compared that, but I know that both servers must have rsync installed, and the B-server might scan the file system. I guess debugging might enlighten the situation.

A server --- syncs all to ----> B server

 

[mkniskanen]

From the rsync man page:

"...rsync uses a lqquick checkrq that (by default) checks if each file's size and time of
last modification match between the sender and receiver..."

Server A scans the given file system (or a subdirectory branch) and server (B) must
do the same so the process is practically the same in both situations.

Using --skip-compress with a list of files that need no compression might lower the CPU use, say:

--skip-compress=gz/jpg/mp[34]/7z/bz2/zip/ogg

but that, again, depends on the number of files involved.

 

[Duboux]

Thanks for the intel =)

btw, I dont'think u can really rsync databases, unless they're dump-files. (as tables may chance every second _+ there's the tables that run on your memory)
If you want to live-sync them (I think you want to have a cluster), u'll need to edit some MySql settings to have it run its queries again on a slave server.

I'm also looking into it, but it's not yet on top of my list, so unfortunately I can't help you there yet as good as you helped me

for the record, I'm almost trying to do the same as you, except for the "creating a similar user on the backup server" part.
But clustering servers is on my things to do list, so I'll get there eventually.. (month or 2)

 

[SeLLeRoNe]

Actually da will (in a NON-SHORT time) implement clustering, they working on it but i dont know in which way.

I would use rsync for users backups, disabling gzip on backup and running a script that gzip (with the rsync needed option) the backup and sync the compressed backup.

Im thinkin to work on it in one of this way:

1- da backup -> after backup script that compress -> push rsync to backup server
OR
2- da backup -> after backup script that compress <- backup server that check once per day each server for take the "differences"

Regards

 

[Jan_E]

Is there any reason not to use MySQL replication? If you run phpMyAdmin with the da_admin user you can can turn replication on/off at any desired moment.

 

[NoBaloney2]

Have you tried it? When we tried it years ago it worked successfully only within an intranet. When we tried it across the public internet it would fall out of synchronization without warning.

I'd still prefer backups in time. But you may feel differently.

Jeff

 

[Jan_E]

Yes, I am doing this for quite some time now, between two Windows servers (no DA) in Amsterdam and Brussels. And between two Debian servers with DA, both in different datacenters in Amsterdam. I have never had any real replication problems. As always with backups you'll have to check once in a while if it it still working. And a DA backup besides that will not harm anyone (or rather DA backup's for every reseller on your server).

The reason I ended up in this thread is that my backup server had very serious problems with the RAID controller. Something similar like in this Youtube clip: http://www.youtube.com/watch?v=WNyjVWeehrk.

The end of the story was a complete new installation of the server. No harm done, because we had a double backup, but a new installment gives you the option to implement new backup strategies as well. I already had rsync running on the /home dir, but was looking for a complete rsync script. Together with MySQL replication that should be safe enough. rsync every day and you will loose at most some file uploads.