DirectAdmin corrupted backup files (tar.gz)

[Ucreation]

Hi,

We have a big problem with tar.gz files that are created by DA on our webserver (CentOS 5.5 64 bit), the backups that are created contains the following errors when we restore a backup or when we extract it manually:

Command: tar xvzf tarfile.tar.gz

gzip: stdin: invalid compressed data--crc error

tar: Child returned status 1
tar: Error exit delayed from previous errors

When we extract a other backup, we get other errors:

Command: tar xvzf tarfile.tar.gz

gzip: stdin: invalid compressed data--format violated
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

(Yes we have more) extracting a other backup says:

Command: tar xvzf tarfile.tar.gz

tar: Skipping to next header
tar: Archive contains obsolescent base-64 headers
tar: Error exit delayed from previous errors

Some information about the server:
* Fresh CentOS 5.5 installation from a brand new quality DVD (2 days old) and a new fresh downloaded ISO
* Directadmin 1.35.1 (updated o recent version)

* Intel Q8200
* Intel DG45ID
* (2x) WD Black 640gb (sata)
* 4x2GB OCZ PC6400 800mhz

Important information:
* Before the new installation we used CentOS 5.3/5.4 and we also had the same problems with the tar.gz files from the DA backup in the last few months, before these period it was always valid.
* The files we tried to extract are not transfered with ftp so these files are local saved by DA (the files that are transfered via ftp to a remote server are also damaged).
* There are files that we extract whithout errors but the content is incorrect (a example: mysql tables are restored back in the wrong database, mysql tables are missing and some files in the public_html folder are missing).
* The files that we have recover with (this) are 100% valid without missing data or corrupted databases etc.

Our question:
We are not looking at a solution to fixed the files we already have now (we have already fix it with god luck) but we are looking at a solution to generate valid tar files but we dont know what the problem is.

Who can help us?

 

[DirectAdmin Support]

Hello,

A crc error usually implies a corrupted tar.gz, generally caused during a transfer. DA only calls the "tar" command, so any corruption would happen either by tar itself, or during the transfer of the tar.gz file.

Check the validity of the file on the system before it's transferred, and then again on the other system after it's transferred. Use an md5sum and ensure the values are the same, eg:

md5sum file.tar.gz

any change in the output would mean the file is becoming corrupted during the transfer.

Also test the extraction of a sample file on the backup system to ensure it's not created in a corrupted state.

If you want, you could try recompiling tar:
http://help.directadmin.com/item.php?id=220

but I'm not yet convinced the issue is with tar itself.
I'd be leaning towards an transfer corruption issue at this time.

If you're using the ftp transfer tool in DA, you can try a different ftp upload client:
http://help.directadmin.com/item.php?id=111

John

 

[Ucreation]

Thanks for your reply!

My first question: Where is the threat itself?

We did some research because we thought that one single user was responsible for all corrupted backups in the same backup job. But it wasn't so. The backups we take via the DA backup system on admin level, with a option 'when' set on 'now' are NOT corrupted. Either, the backups we take with the option 'when' as cron schedule are corrupted. In both cases the backups are stored locally (no ftp transfer).

But, when we used option 'when' as cron schedule and tranfered to a external ftp server (in binary mode of course) the backups are NOT corrupted.

The backups that are corrupted are often above 1.5GB in size. The small backups are never corrupted.

We dont know what the problem is, but we think that TAR is not responsible for the corrupted data. If we compress data to a tar on commandline the tar isnt corrupted.

 

[ranz]

are you running your DA server on a VPS? does your VPS have enough RAM? does your VPS have enough HDD space?

just a thought - but I've seen this time and time again with our clients who go for small VPS accounts - and try to do massive things (RAM & FILE SIZE).

 

[Ucreation]

@ranz

We're running DA directly on our physical server (CentOS 5.5 x64), we don't have a VPS or virtual machine.

The server has 8Gb of RAM (4x2Gb) so I don't think that this would be the problem, our largest tar.gz file is approximately 3.5Gb in size.