backup_exclude_paths: unnecessary files copied to temporary backup directory

[websafe]

Patterns defined in .backup_exclude_paths are used ONLY by the final
tar command, which creates the resulting backup archive.

This is OK, but I think backup_exclude_paths was meant to speed
up backups by excluding unnecessary folders/files.

This works only partially, because:

1. DA copies ALL user files temporary to /home/tmp/admin.XXX/user/backup/
even those excluded via /home/user/.backup_exclude_paths

2. DA creates the final archive with --exclude-from=/home/user/.backup_exclude_paths
- this is expected and OK.


But the first step, copying for example three gazillions of files
(spam/trash/etc.) to the temporary location (/home/tmp/admin.XXX/user/backup/)
so they can be ignored/excluded by the final tar command
makes no sense, because this copying to the temporary directory is
what takes sometimes very log, especially when you're using NFS shares.

So I would suggest to somehow rethink/modify this behavior.


Related threads:

1. [Feature request] Use backup_tmpdir for "local" backups too (locally mounted NFS)
2. [Feature request] backup_include_paths in addition backup_exclude_paths

 

[websafe]

I'm aware that solving this problem would require to reimplement
the way tar is handling entries in `--exclude-from=/home/user/.backup_exclude_paths`
by the DA procedure which performs copying files to the temporary location,
to have these files excluded "earlier" - during the creation of the temporary
structure.

So maybe a solution like

tar \
  --exclude-from=/home/user/.backup_exclude_paths \
  -cpf - \
  -C /home/user/ \
  . \
  | tar -xf - -C /home/tmp/admin.XXX/user/backup/

could be used for copying files to the temporary location...?

 

[Richard G]

Agreed. +1 on this request.
Would be nice if the excluding would take place before the copying, so only real files to be backed up will be copied.

 

[myH2Oservers]

+1. Currently running into an issue with a server that has 40 GB free space but runs out of storage when the backup is executed, it runs out of space as excluded data (30 GB is excluded) is also copied while the backup result is only 3 GB big (as excluded files are not tarred).