I'm in a world of hurt backing up to S3 bucket via FTP - need help

jkirker

Verified User
Joined
Nov 22, 2012
Messages
124
I'm in a world of hurt backing up to S3 bucket w/ FTP and I need help.

I have several accounts whose backups are multiple gigs. The problem using FTP and S3 is that FTP sends files in 100M increments. So a 2 gig file is sent as 20 separate files that grow incrementally, especially when versioning is on.

ie: File 1 is 100M
File 2 is 200M
File 3 is 300M and so on.

The end result? a 2G file ends up taking 21G of storage - unless the chunk files are deleted.

So this month and for the next 90 days I'm facing hundreds of dollars in fees until the files I find are deleted and have been deleted for 90 days. This issue ultimately will cost me several thousand.

Why is there no backup to s3 (or s3 compatible) systems using the S3 API? And where can I increase the FTP chunk size in an attempt to mitigate my losses. :(

Thanks,
John
 
I'll let someone else answer your question, but how does storing 21G cost you thousands when S3 is $0.023 per GB per month? Thousand for 90 days would be like 43TB of information. Unless I am missing something, your losses would be like $1.50.
 
@BillyS It'd be great if it were only one file. That's just an example of how the carnage can be exponential. Here's the actual damage. Across all systems nearly a petabyte of fragments. Fortunately I'm only being billed for a ~60TB. But in reality I should only be paying for about 10TB of actual/usable data.

@jamgames2 - Couldn't agree more about FTP. It's like conducting banking transactions via email. Does that integrate and replace the DA backup system - and is it compatible with existing backups??

I had replied to this in 2019 looking for same solution. It's surprising to me that DA hasn't itself integrated proper S3 compatibility by now.

 

Attachments

  • IMG_8942.jpg
    IMG_8942.jpg
    160.3 KB · Views: 13
You can turn off admin backup and switch to use jetbackup.
Great. Can you tell me if will be compatible (mainly for restore from functions) with preexisting DA backups that already live at the endpoints? Or will I have to run things concurrently until existing backups expire so as to keep in line with my client SLAs?
 
It's not same as admin backup. Jetbackup have their own lifetime logics checking.

you can get trial license and testing by yourself.
 
It's not same as admin backup. Jetbackup have their own lifetime logics checking.

you can get trial license and testing by yourself.
Thanks. For our workflow we need the same system, just FTP replaced by S3 via API. Fortunately, I was able to get the issue resolved with the S3 provider on cost.
 
Back
Top