backup - to compress or not to compress

M

merlinsilk

Verified User
Joined
Feb 16, 2009
Messages
32
I learned from John (http://www.directadmin.com/forum/showthread.php?p=194797) - thanks, John - that the admin backup of users has the tar command (mostly) hard-coded. That means I always get backup files in the form user.admin.xxx.tar.gz where xxx is the user's name and gz means that I get a zipped file.
Generally that is a good solution, with two exceptions as far as I can see:

1. I have plenty of disk space - there is no need to invest the resources to zip possibly huge tar files - creating the tar instead of the tar.gz is quite a bit faster and much easier on the processor - and disk space is getting cheaper and cheaper.

2. If I want to use rsync to maintain a copy on a different machine. zipping the files negates rsync's feature to send only the differences over the (possibly paid for) line to another machine. Change only a single byte in the source for a zipped file and the whole zip changes.

For these two reasons I would like to request an option to create either .tar.gz or just plain, uncompressed .tar files.

I did a test for my VPS and here are the results:
about 1.5GB in user backups. Not much has changed on the sites during the day, a few blog posts added and log files and stats. When using rsync to copy these files over to a mirror machine, nearly 1.5GB had to be transferred.

When I ungzipped those files before the rsync I used just a bit over 1% of that bandwidth - 16MB instead of 1.5GB!

So, pretty please, can I get this option for backup?

Cheers,
Merlin
 
I would like to request bzip2 instead of gzip.

Why are you even using the backup system at all if you are going to use rsync in that way?
 
yes, massive, that's right! DA has the great feature of collecting everything I need to restore - just what you need after a disaster. and with the uncompressed tars rsync works very nicely, just transferring 1 to 2% of the source data to update the destination.
As I have it right now - before we will get the option I requested - I do have to uncompress the tar.gz files to .tar which takes a considerable amount of time and resources. Should disaster strike I will have to compress the .tar to .tar.gz on the destination first before I can use DA to restore, but I will have a mirror up and running within one or two hours.
This, by the way, is one of the very few disadvantages of compiled DA over the other php or python control panels - I can't just change the source code to do myself what I need - have to wait for DA to do it for me.
But on the other hand, DA is really doing it's part to green the planet: Read some article a while back that if all the php code running out there was compiled code we would save a considerable amount of energy as data centers would need a lot less servers to do the same work. By far most of the servers out there run linux on intel so it really would not be a big deal to have all those wordpresses et all being compiled instead of interpreted code.
 
I'am also wondering how to rsync to another machnie. DA backup advantage is quick recovery of single account, but in case of whole server loss, You loose data and backups at the same time. So it is good to have spare backup server, for doing rsync backup.

I think best solution at this moment is to run rsync directly on user files and not backed up tar.gz.

I would also like such option to be implemented to DA: choose tar.gz oraz tar for backup.
 
The way, far best solution would be for DA to implement an rsync directly - but this would be probably wishing too much ;-)

Currently the admin backup give me an option to save the backups via ftp to another machine. As I found out that works but is very bandwidth intensive because the whole of the user data is copied over the net every time (OK, if it's a local backup server, but not if you have to pay for the bandwidth.)

By the same right, a more efficient alternative to ftp would be great - rsync. The only thing that needed to be manually set up would be a key login to the destination machine, then DA could run something like

rsync -avvz -e 'ssh -l user' /temp-dir/ [email protected]:/backup-other-machine

If the main machine goes down then a quick DA restore on desthose.com would make this machine able to take over the work very quickly.

'stars', just rsyncing will not do the job because today most or many sites use dynamic data in databases and you would not back that up by just copying the user files. All your wordpress or joomla sites will get over severely crippled.

Cheers,
Merlin
 
Ok, databases are fragile, but in case of server disaster, where you have to use extrernal backup, few databases which didn't survived doesn't make me worry.

We use primarly DA backup for everyday use. Rsync and external backup is in case of real emergency and for my peacefull sleep ;)
 
Hello,

Changing cvzf to cvf (and xzf to xf) based on a directadmin.conf option would be easy enough. I'm not sure I'd make it an interface option, but a directadmin.conf option wouldn't be too hard.

John
 
Hello,

I was wondering if the compress/don't compress option in the conf file is ready?

Also, I agree that an incremental back-up system would be great. For now, though, being able to DA not to compress the tar's would be useful.

Thanks!

~ Jeremy
 
Yes you can choose to dont compress backups.

Actually i dont know if there is a backup_post.sh script that start after backup complete, should be helpful for start custom script (for example rsync for send incremental to remote server).

Regards
 
It's an option in directadmin.conf. My recollection is that It defaults to nonexistent, which means the files are compressed.

Google either at site:www.directadmin.com or at site:help.directadmin.com.

Jeff
 
You must log in or register to reply here.
Back
Top