Admin Backup/Transfer Script & Rsyncable

D

davezeking

Verified User
Joined
Jan 4, 2007
Messages
16
Hi,

I am looking at setting up the following backup system on my DA server:

1) Use Admin Backup/Transfer to create a daily local backup of my DA users and settings in a single folder
2) Using an offsite machine, connect to the DA server using Rsync to mirror this folder on the connecting server

Since only a small subset of the data changes each day I am hoping this will enable me to have one full backup of the server each day (I will handle the archiving of this data for multiple backups on the local machine)

In order to do this I need to make sure the GZIP bundles that the Admin Backup/Transfer script outputs are created using the gzip --rsyncable option. Can anyone tell me where the backup script is located by default and if it would be possible to alter this to add the --rsyncable flag?

I realise that if DA is updated on my server I am going to have to update this script each time but that seems an acceptable tradeoff for the amount of transfer I will save on a daily basis.

Regards,

Dave.
 
Same question

I am having the same problem. Does anyone knows where to set the --rsyncable flag?
 
Script Location

Could anyone at least tell me the location of the script that is run for the Admin Backup/Transfer so I could take a look at the script?
 
Hello,

DA doesn't use a script to create backups, it'll all done in the binaries.

However, you are in luck, there is a flag you can specify to add that option to the tar command:
http://www.directadmin.com/features.php?id=792

eg:

extra_backup_option=--rsyncable

Which would make the tar command look like:

tar czfp /path/to/user.tar.gz --rsyncable -C /backup/location/username backup -C /home/username domains

John
 
Hi,

ive a symple question, with rsyncable option you will be able to import only the "difference" between two archives?

You will do a public script for that or ive to work on it by myself?

Regardds
 
Unfortunately I am still having some trouble getting an rsyncable backup sorted.

I have added the --rsyncable argument in the directadmin.conf file but last night my backups failed with the error:

Error Compressing the backup file admin.root.admin.tar.gz : /bin/tar: unrecognized option `--rsyncable'
Try `/bin/tar --help' for more information.
Click to expand...

I think I understand what is wrong here, the rsyncable flag should only apply to GZIP and not TAR, as far as I am aware there isn't an --rsyncable option for TAR as this doesn't have the sames issues when rsyncing that gzipped files do?

Is there any way to apply this argument to the gzip part of the operation only and not the tar command?
 
Last edited:
SeLLeRoNe said:
Hi,

ive a symple question, with rsyncable option you will be able to import only the "difference" between two archives?

You will do a public script for that or ive to work on it by myself?

Regardds
Click to expand...
Once I have this working I will let you know what 've done :)

There shouldn't be any scripting on the server side (you will need to install and configure rsyncd on your DA server however), just an automatic Admin/Transfer backup to a local folder on your DA server and then a remote machine can login and duplicate the files to itself only downloading the changes via rsync.

You should then have a backup copy of your server data on your remote machine.
 
Thanks for your reply :) that should be awesome and should be exactly what i need for remote backups for dont stress out bandwidht ^^

I hope it will work as you saying and as i hope ^^

Regards
 
Unfortunately this is still not working as I'd like, each backup downloaded using RSYNC is downloading the entire file for each domain every time.

The --rsyncable flag does not apply to the TAR command only the GZIP command, is there any way to apply this flag to the GZIP elements of the backup or would this be negated by the TAR command anyway?

Really need some help with this one - there must be a way to put together a backup that can be downloaded using the benefits of RSYNC?
 
mmh, should be stupid what im going to say...

BUT.. what about a post_backup script that do gzip -d archivetar.gz | gzip -rysncable archive.tar?

i know is a "waste" of time and resource for the server on backup but, should work at least for transfer just the changes?
 
Hello,

I was actually going to suggest the same thing.
Since the tar.gz is sitting there, you can use this:
http://www.directadmin.com/features.php?id=951

and then do whatever you'd like with the backup file. (eg: extract it and rysnc just the extracted data, uncompressed, file by file. That's the original rsync design anyway.

The user_backup_post.sh script is run right after the tar.gz is created, and before the tar.gz is uploaded with (if you have ftp enabled).

John
 
davezeking said:
Hi,

I am looking at setting up the following backup system on my DA server:

1) Use Admin Backup/Transfer to create a daily local backup of my DA users and settings in a single folder
2) Using an offsite machine, connect to the DA server using Rsync to mirror this folder on the connecting server

Since only a small subset of the data changes each day I am hoping this will enable me to have one full backup of the server each day (I will handle the archiving of this data for multiple backups on the local machine)

In order to do this I need to make sure the GZIP bundles that the Admin Backup/Transfer script outputs are created using the gzip --rsyncable option. Can anyone tell me where the backup script is located by default and if it would be possible to alter this to add the --rsyncable flag?

I realise that if DA is updated on my server I am going to have to update this script each time but that seems an acceptable tradeoff for the amount of transfer I will save on a daily basis.

Regards,

Dave.
Click to expand...

First, you should have your backupserver pull the the incremental backups from your da server either with rsync, ssh, rsnapshot of whatever. This wil give you the option to keep multiple backups, maybe on a daily basis or even more than once a day.

Besides this, you should just keep the directadmin backups running (local on the da server ofcourse), for fast access to your latest backup.

Now the users (or the admin) can restore their last backup with very little trouble. Just use 'restore' in the directadmin panel.
Meanwhile you should be able to hack a script together to create a tarball on your backupserver from a specific incremental backup and upload this tarball to the da server, ready to restore. If your user needs a backup from the day before yesterday, you just run your script (and bill your customer...)

Don't forget to exclude the backup direcories on the DA server in your rsync command or your direactadmin generated tarballs will get rsync'd and we don;t want that, do we :)
 
Thank you to all for the suggestions.

SeLLeRoNe:
Thanks for post_backup idea, I might be able to do something with this. Your suggestion for deflating and then re-zipping with the correct flag should work I think. I will test this and post feedback here.

John:
Thanks for the link, I think what I will need to do is unzip each backup archive and either rsync directly at this point or rezip on the DA server using the rsyncable flag. Is there any possibility of DA being updated so that zipping the backup becomes optional? i.e. only a tar and not tar.gz is created (removing the -z option in the backup binary) - I realise this might not be possible since not many users would need this.

sysdev:
Thanks for the outline, I have a system quite close to what you are proposing, the item we are working on is how to rsync the local DA backup to the backupserver since the DA backup is gzipped which doesn't rsync well (sends the entire backup archive each time!).
 
davezeking said:
sysdev:
Thanks for the outline, I have a system quite close to what you are proposing, the item we are working on is how to rsync the local DA backup to the backupserver since the DA backup is gzipped which doesn't rsync well (sends the entire backup archive each time!).
Click to expand...

That's why you should exclude those DA backups from your rsync.
If you mirror the users homedir, you should be able to create the required tarballs on your backupserver, ready to restore at any time.
 
sysdev said:
That's why you should exclude those DA backups from your rsync.
If you mirror the users homedir, you should be able to create the required tarballs on your backupserver, ready to restore at any time.
Click to expand...

I see, so what you are suggesting is manually creating the backups by shadowing the original directories i.e. the /home directory in my case?

My only worry here is when restoring, I am unsure, but I wonder if the Admin/Transfer backup is a better bet to restore from than just overwriting directories on the DA server?
 
davezeking said:
I see, so what you are suggesting is manually creating the backups by shadowing the original directories i.e. the /home directory in my case?

My only worry here is when restoring, I am unsure, but I wonder if the Admin/Transfer backup is a better bet to restore from than just overwriting directories on the DA server?
Click to expand...

If your create the tarball on the backupserver, you should be able to restore it using the da admin backup. Just enter the ftp and credentials from your backupserver.

DA also uses a tar command to generate the backups, so it shouldn't be to hard to copy the syntax on your server OS to your backupserver.
 
sysdev said:
If your create the tarball on the backupserver, you should be able to restore it using the da admin backup. Just enter the ftp and credentials from your backupserver.

DA also uses a tar command to generate the backups, so it shouldn't be to hard to copy the syntax on your server OS to your backupserver.
Click to expand...

I understand what you mean, but I'm not sure the Admin Backup/Transfer files only contain the /home folder for each user, I think they are also able to recreate a user entirely. So, for example, if my server HDD was hosed I could install DA on a new server then upload each .tar.gz and it would create that user from scratch setting up emails, quotas, permissions etc.

Maybe my wording is wrong as this could be termed as "Disaster Recovery" rather than merely a "Backup", I could also be totally wrong with my assumptions of how the Admin Backup /Transfer works :o but this is the sort of local copy of the server I would like to keep.
 
Is this script correct?

Hi There,

Following on from the discussions above, I've created a script as below. It seems to be working in the sense that I can watch the tar.gz files being decompressed and then re-created.

However, I'm not convinced that anything has actually changed as my rsnapshot report (rsync) shows that the whole archive is moved over, not just the changed bytes.

Would someone mind checking the below for correctness?

Code: 
#!/bin/bash

gzip -d $file
gzip --rsyncable ${file%.tar.gz}.tar

exit 0;
 
You must log in or register to reply here.
Back
Top