I'm in a world of hurt backing up to S3 bucket via FTP - need help

[jkirker]

I'm in a world of hurt backing up to S3 bucket w/ FTP and I need help.

I have several accounts whose backups are multiple gigs. The problem using FTP and S3 is that FTP sends files in 100M increments. So a 2 gig file is sent as 20 separate files that grow incrementally, especially when versioning is on.

ie: File 1 is 100M
File 2 is 200M
File 3 is 300M and so on.

The end result? a 2G file ends up taking 21G of storage - unless the chunk files are deleted.

So this month and for the next 90 days I'm facing hundreds of dollars in fees until the files I find are deleted and have been deleted for 90 days. This issue ultimately will cost me several thousand.

Why is there no backup to s3 (or s3 compatible) systems using the S3 API? And where can I increase the FTP chunk size in an attempt to mitigate my losses.

Thanks,
John

 

[BillyS]

I'll let someone else answer your question, but how does storing 21G cost you thousands when S3 is $0.023 per GB per month? Thousand for 90 days would be like 43TB of information. Unless I am missing something, your losses would be like $1.50.

 

[Ohm J]

backup via ftp protocol to cloud storage is very bad idea.

If you don't have software engineer to write your own backup to s3 bucket. I suggest using jetbackup.

S3 Compatible — JetBackup 5 Documentation documentation

docs.jetbackup.com

 

[jkirker]

@BillyS It'd be great if it were only one file. That's just an example of how the carnage can be exponential. Here's the actual damage. Across all systems nearly a petabyte of fragments. Fortunately I'm only being billed for a ~60TB. But in reality I should only be paying for about 10TB of actual/usable data.

@jamgames2 - Couldn't agree more about FTP. It's like conducting banking transactions via email. Does that integrate and replace the DA backup system - and is it compatible with existing backups??

I had replied to this in 2019 looking for same solution. It's surprising to me that DA hasn't itself integrated proper S3 compatibility by now.

DirectAdmin Amazon S3 Backup Support

When DA will add support to add Amazon S3 backup so we can easily transfer backup to Amazon S3 ? I have see that feature in Virtualmin but its missing from DA , I didnt see any major feature in DA since I bought it which is like 8 years or more now.

forum.directadmin.com

GitHub - powerkernel/directadmin-s3-backup: DirectAdmin S3 Backup

DirectAdmin S3 Backup. Contribute to powerkernel/directadmin-s3-backup development by creating an account on GitHub.

github.com

 

[Ohm J]

It's recommend in price page. so it's plugins for fully replacement backup system.

DirectAdmin Web Control Panel Pricing

www.directadmin.com

You can turn off admin backup and switch to use jetbackup.

 

[jkirker]

You can turn off admin backup and switch to use jetbackup.

Great. Can you tell me if will be compatible (mainly for restore from functions) with preexisting DA backups that already live at the endpoints? Or will I have to run things concurrently until existing backups expire so as to keep in line with my client SLAs?

 

[Ohm J]

It's not same as admin backup. Jetbackup have their own lifetime logics checking.

you can get trial license and testing by yourself.

 

[jkirker]

It's not same as admin backup. Jetbackup have their own lifetime logics checking.

you can get trial license and testing by yourself.

Thanks. For our workflow we need the same system, just FTP replaced by S3 via API. Fortunately, I was able to get the issue resolved with the S3 provider on cost.