Admin Backup/Transfer Script & Rsyncable

Backup Script

I have been meaning to post an update here for some time, I have also developed a small script that defaltes the GZIP archive leaving only the TAR archive, I then RSYNC these files and this is currently working well except for one particular TAR file (which is > 4GB) that fails every night.

The first backup is quite large due the uncompresed files but with subsequent backups only the differences are sent. I also investigated the -rsyncable option but this never seemed to work well and I didn't like adding the extra load of rezipping on the server after each backup.

My script is as follows:

Code: 
#!/bin/bash

RSYNC_BACKUPS_PATH=/home/admin/admin_backups/local_rsync_backup

# Check	the $file variable is set
if [[ -n "{$file}" ]] ; then

    # Only manipulate files that are in our RSYNC_BACKUPS_PATH
    # otherwise we may break other backups (e.g. FTP backup)
    if grep -q $RSYNC_BACKUPS_PATH <<<$file; then

        # Check the $file exists
        if [[ -e $file ]] ; then
            gunzip -f $file
        fi
    fi
fi

The script is installed as follows:
Code: 
nano /usr/local/directadmin/scripts/custom/user_backup_post.sh
chown diradmin:diradmin /usr/local/directadmin/scripts/custom/user_backup_post.sh
chmod 755 /usr/local/directadmin/scripts/custom/user_backup_post.sh

If you don't mind the larger filesize try this out and let me know how it goes.
 
rBackup 0.5 - 4 Sept 2007

Doesnt look like its actively worked on...I wouldnt use it.
 
SeLLeRoNe said:
ive a question, why a 4+GB file dont work? i got many customer's account higher then 4GB, any log file to analize?
Click to expand...

I'm not sure if the problem is because of the 4GB+ size, it's just I have one particular file where the only difference to the other files is that it is larger than 4GB and it consistently fails to backup incrementally.

When I delete the destination file so the whole file is downloaded it transfers fine, but the next time the backup is run where an incremental should occur I get the following error in the rsync log:

Code: 
WARNING: user.regalback.metrol.tar failed verification -- update discarded (will try again).
user.regalback.metrol.tar
ERROR: user.regalback.metrol.tar failed verification -- update discarded.

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1524) [generator=3.0.5]

Also, rsync seems to be adding the incremental difference at the end of the file as the destination gets larger and larger until I delete it and force a new transfer.

Perhaps you could run the script and see if it does this with your files?

By the way I am transferring from ext3 to ext3 so I don't think there should be a problem with the filesize due to the filesystem type?
 
Hello,

It could be the filesystem, but most modern filesystems *should* be able to handle large files. What's likely is that the binaries used are not compiled to handle the larger file sizes. I can't comment too much on rsync or gzip, but for DirectAdmin, I had to add this compiler option to get it to handle the larger file sizes
Code: 
-D _FILE_OFFSET_BITS=64
if you want to try recompiling rsync, I'd recommend checking to ensure that option is present.. or some equivalent.

John
 
im sorry i cant try atm that script cause ive not a backup server running, just using two hard disk each box... i would install a backup machine when that script will work, but, i think is a file system problem too
 
Well, just unzipping the tar.gz's to tar's:

Code: 
gunzip -f $file

...doesn't seem to help rsync (rsnapshot) to reduce file size during backups, it's still dragging an entire new version of the archive over each day so for now best option I can see is just to sync over the tar.gz's that DirectAdmin creates.

I'd love to hear if anyone has a better option.

Does anyone know if rdiff-backup would take up significantly less space than rsync/rsnapshot does? And does anyone have a script example to do so over SSH?
 
Not sure why you are not seeing the speedup as my system is definitely gaining some advantage. From my logs last night:

Code: 
sent 3807669 bytes  received 6791656549 bytes  823343.34 bytes/sec
total size is 11959234560  speedup is 1.76

This is showing that it's only downloading 6.3GB rather than the 11.1GB that is on the server, an almost 50% reduction. This also includes re-downloading the broken file (> 4GB) that I mentioned in an earlier post. If I could fix that problem I would only be downloading approx. 2GB out of a possible 11GB - a huge improvement.

Are you certain you are seeing the .tar files in the temporary backup area and not just .tar.gz files?
 
davezeking said:
Once I have this working I will let you know what 've done :)

There shouldn't be any scripting on the server side (you will need to install and configure rsyncd on your DA server however), just an automatic Admin/Transfer backup to a local folder on your DA server and then a remote machine can login and duplicate the files to itself only downloading the changes via rsync.

You should then have a backup copy of your server data on your remote machine.
Click to expand...

Thanks for your reply that should be awesome and should be exactly what i need for remote backups for dont stress out bandwidht ^^

I hope it will work as you saying and as i hope ^^
 
davezeking said:
I have been meaning to post an update here for some time, I have also developed a small script that defaltes the GZIP archive leaving only the TAR archive, I then RSYNC these files and this is currently working well except for one particular TAR file (which is > 4GB) that fails every night.

The first backup is quite large due the uncompresed files but with subsequent backups only the differences are sent. I also investigated the -rsyncable option but this never seemed to work well and I didn't like adding the extra load of rezipping on the server after each backup.

My script is as follows:

Code: 
#!/bin/bash

RSYNC_BACKUPS_PATH=/home/admin/admin_backups/local_rsync_backup

# Check	the $file variable is set
if [[ -n "{$file}" ]] ; then

    # Only manipulate files that are in our RSYNC_BACKUPS_PATH
    # otherwise we may break other backups (e.g. FTP backup)
    if grep -q $RSYNC_BACKUPS_PATH <<<$file; then

        # Check the $file exists
        if [[ -e $file ]] ; then
            gunzip -f $file
        fi
    fi
fi

The script is installed as follows:
Code: 
nano /usr/local/directadmin/scripts/custom/user_backup_post.sh
chown diradmin:diradmin /usr/local/directadmin/scripts/custom/user_backup_post.sh
chmod 755 /usr/local/directadmin/scripts/custom/user_backup_post.sh

If you don't mind the larger filesize try this out and let me know how it goes.
Click to expand...

That code works great but i have some trouble with setting up rsync. Could you share some setting with me?
 
Update: New Script

After quite a long period of testing I have an updated script which resolves the problems when individual backup files are > 4GB

This has been working successfully for me over the last few months, on a good (quiet) day I am seeing a speedup of over 300x e.g.

sent 3483078 bytes received 32126261 bytes 43935.03 bytes/sec
total size is 13094801808 speedup is 367.74
Click to expand...

The script has a couple of new options:

SPLIT_SIZE : This is how large each segment in the split should be
SPLIT_THRESHOLD : How large does the file have to be before we attempt to split it

I have set these as 1GB and 3GB respectively, hopefully this will suit most people.

The script now also creates an md5 file for each tar file in the backup directory so you can check that a file has been successfully rsynced down to your local backup area.

You can put a split file back together using the following command:
Code: 
cat user.x.domain.tar.0* > user.x.domain.tar

Where x is the reseller and domain is the DA username for the account.

The script uses the following utilities/programs that you will need installed in a place where your PATH variable can see them:

grep
gunzip
md5sum
split

Your server will also need to act as an rsync server (I'm sorry but I cannot help with this as I am not really a sysadmin and I think there will be many different ways to install this on your server - check out your package management options or the DA forums for help with this)

Finally, the script must be named user_backup_post.sh and placed in your DA custom scripts directory which on my server is: /usr/local/directadmin/scripts/custom

Download Link for Script is: http://www.regalsoftware.co.uk/downloads/user_backup_post.zip
 
Hello,

Does this script still work and if so can you repost the link to it. The above link no longer works.

thanks

Jon
 
Hi Jon,

Sorry for the delay, been away on holiday :)

I've re-added the script to my server so the link above now works, I'm still using the same script with DA 1.40.1 and I haven't seen any problems as yet.

Regards,

Dave.
 
You must log in or register to reply here.
Back
Top