backup improvement feature request

[thoroughfare]

Hi,

For the DA staff:

When you make a backup on DA, it always stores to the server before you can download it. Problem is, that this feature relies on the customer having enough space free to store a backup. Can you add a link to download a backup straight from the control panel without creating any files in the customer's account, like you can on CPanel?


Thanks!
Matt

 

[DirectAdmin Support]

Hello,

Backups are owned by root and do not affect the users disk space.

John

 

[thoroughfare]

I thought that might be the case, since the stats for space etc don't decrease when you add a new backup.

I tried testing it tho, by continually making new backups to see if it would keep letting me make them past the users allocated space. After 8 backups-worth of 92mb of data (the account is allocated 800mb space), DA came up with an 'Unable to execute - unknown error" message.

Any ideas?

Much appreciated,
Thanks
Matt

PS I am a reseller, not an admin.

 

[DirectAdmin Support]

Ok, my mistake. We *used* to save the backup files as root. They are now saved as the user... so that would explain your problem. We'll add the feature to the list. At least we know quotas work The stats are computed nightly, so you won't see any immediate changes.

John

 

[thoroughfare]

Thanks mate

 

[nobaloney]

Ok, my mistake. We *used* to save the backup files as root. They are now saved as the user... so that would explain your problem.

What were you thinking, John ?

Backups stored as the user will always limit the user to approximately one-half his/her allotted space.

Please figure out a way to allow backups of their entire allotted space.

Pretty please .

Jeff

 

[DirectAdmin Support]

Backups stored as the user will always limit the user to approximately one-half his/her allotted space

Well yes. If they want backups, they should pay for it .. the feature is mainly aimed at users, and not backups for server admins who want to backup the server (users can delete the backups after download). I'll look into a tempfile for the backups so that the backup can be stored temporarily somewhere for a given period of time at not cost to the user.

John

 

[nobaloney]

Well yes. If they want backups, they should pay for it

Yes, they should. But they already do, built into the price of the plan.

A current plan (for example) offers 100 Megabytes of site space, ten 20 megabyte mailboxes, and nightly backup.

Obviously we do the backup.

But because things can go wrong, our terms of service specify that we're not responsible for lost data; we suggest that our clients do their own backups and keep them at their location rather than on our server.

Imho, to require that they pay for twice the sitespace so they can do their own backup, that's only going to be on the server long enough for them to make it (after which they'll download it to their own system), isn't reasonable.

One option would be to give them twice as much site space, but then of course they'd use it without realizing it, and end up not being able to make a backup.

Another option would be to sell them twice as much site space, and hope they understand that they're only supposed to use half of it on a regular basis, so they can use the other half to back up their site.

Both of those options appear to me to be easily misunderstood, and to cause extra customer support.

So what I'd prefer would be a system of backing up directly to an offsite location, possibly using temporary space on the server that's not part of the site.

the feature is mainly aimed at users, and not backups for server admins who want to backup the server

Yes, we do our own backups, which we store offsite.

(users can delete the backups after download).

As well they should. Which makes it (from their point of view) hard to understand why we'd want to make them "pay for it" as you put it.

I'll look into a tempfile for the backups so that the backup can be stored temporarily somewhere for a given period of time at not cost to the user.

Imho, that would be the best scenario.

Jeff

 

[DirectAdmin Support]

Well, we have few options we can work with....

1) only ever have 1 copy of a backup, so each backup will overwrite the last, and store it out of their home directory, as a different username so it won't count against their disk usage. This would solve all most all of the problems.

2) overwrite the last backup and keep it as their username; it will use some of their disk space (I guess we are trying to avoid this).

3) have both options, 1) as well as leaving the current system as is...probably just a waste of space.

So... 1) is looking like the choice most likely to be voted in, the only thing is that incremental backups won't be as easy to manage (unless you keep downloading it and storing it as a seperate file as you create the backups).

So let me know what you guys want. also add any additional options you can think of that you would want.

John

 

[The Prohacker]

Hmm...

Maybe there could be automatic daily backups stored in say... /backup

Inside each user's home directory would be a symlink to the tar ball like:
/home/username/sitebackup.tar.gz that would actually be /backup/username-backup.tar.gz.. Or have a sub directory in their home dir...


/home/username/backups

And inside that be symlinks to the last 5 days worth of backups? Possibly configurable by the admin on how many days of backups to store... They would of course be owned by the backup user so not to apply to the quota but would need to only be visiable by the user who owns the site....

 

[nobaloney]

I'm not so sure I like Prohacker's idea of automatic backups; if so it'd want it to be an admin or reseller option, and if done as an admin option perhaps charged against the reseller's space.

I do like his idea of the symlink, though, and in general option 1, for users who do decide to make their own backups.

Only one, though; if they want to save multiple backups, they need to move it off our system.

I'd expect my clients to keep their backups somewhere else besides on our system; hopefully some of them will realize that a backup on the same system is at risk for most of the same issues as the web directory itself.

Let the users download multiple backups or pay for space to keep them elsewhere on the 'net, perhaps on another system at the same data center.

Jeff

 

[thoroughfare]

Erm... John,

Is it not possible just to create the backup dynamically and feed the file to the browser, like CPanel does? It requires no server space, so you don't have the problem of users taking up loadsa space.

You could make it work the same way dynamically-called images do, like so:

getimage.php?id=ds43b843on

Also, there's a bug in the backup system. If the user is not allowed annonymous FTP, and then they try to make a backup with FTP settings and accounts selected, DA brings up an error

Cheers,
Matt

 

[The Prohacker]

Is it not possible just to create the backup dynamically and feed the file to the browser, like CPanel does? It requires no server space, so you don't have the problem of users taking up loadsa space.

Doing live backups isn't always a good idea... If the site is large... 200MB+.. Its going to have to copy all those files.. Dump the databases.. Move them into a common directory structure.. Tar them.. And then feed them to the browser..

You have to fight browser timeouts for one thing...

Then also.. What if the server load is high when they click the download backup link.. So it then hits up tar and drives your load even higher.. As it is right now.. DA uses a system queue of commands.. And executes them in order as they are sent in.. This allows for load checking and delaying of proccesses till the load drops..

You couldn't do this with a dynamic backup system.... For small sites it would likely work great but many sites have huge databases....

 

[nobaloney]

You have to fight browser timeouts for one thing...

I've got to agree on this one. I've been in the Sun Cobalt RaQ support business for many years now, and I can tell you that most RaQ-owners have all but abandoned web-based backups; they don't work. Specifically because of browser timeouts.

Sun Cobalt created a "CMU" (Cobalt Migration Utility) system which can be used to back up either all the userspace and sitespace on the system, or individual sites. Some one packaged it with a utility that adds MySQL export, and many of us use it to backup or RaQ content anywhere from nightly to weekly.

Some of us have made web-based interfaces to it that can be used to back up individual sites.

So I guess we have to rethink a lot of this .

I'm open for ideas.

Jeff

 

[thoroughfare]

Ah ok, I understand, explains why DA seems better than CPanel then

Cheers guys,
Matt

 

[nekote]

Good discussion.

Could I add one additional perspective?

As a very small time reseller, I'd like the ability to backup each individual client site one at a time. And for the necessary space to somehow come out of "my" (unused / unallocated?) reseller space. Don't see how to square that with permissions, though.

I envision downloading successive backup files while proceeding with the next client's backup.
Potentially waiting to initiate the next backup if enough space not yet available.
Deleting each backup image after it is download.

That would be particularly useful if / when sites need to be moved to a new (DA) server.