CLI backup strategy for incremental backup

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
10,625
Location
LT, EU
Even if tools like ClusterLogics and R1Soft integrate with DirectAdmin, and incremental backup solutions like https://www.modmaster.pro/ exist, if you'd like to use free incremental backup utility (while it gets integrated into DirectAdmin), here is a short description of the idea:
DirectAdmin has "Admin Backup/Transfer" feature, which could create you a backup without any data in /home. This should take minimal space on the system, as main data (emails and website data) is stored in home. We could use any utility (borg, restic, riff-backup) to backup whole /home folder.

The steps would be:
1) Go to "Admin Backup/Transfer" feature, "Schedule" a backup to be ran as often as you'd like to (once a day might be enough), de-select "Domains Directory" and "E-mail data", place your backups to /home/admin/admin_backups for example.
2) Setup restic (supports many backup storage types) or borg (SSH required on the other end). Listing steps for borg, but things would be very similar for restic too.
a) install borg, commands for CentOS:
Code:
yum -y install epel-release
yum -y install borgbackup

Debian:
Code:
apt install borgbackup
b) initialize a local OR remote repository:
local (using /backups, but you can set any directory you want):
Code:
borg init --encryption=none /backups

remote (using borgbackup SSH user on remote server having privileges to /backups/hostname):
Code:
REPOSITORY=borgbackup@[B]YOUR_SERVER_IP[/B]:/backups/`hostname -f`

I'd suggest adding ssh keys from root to that remote server, so that it could connect without a password.

c) create /usr/local/directadmin/scripts/custom/all_backups_post.sh script, which would automatically sync with borg backup server after every backup done, chmod the script to 700, content:
Code:
#!/bin/sh
REPOSITORY=borgbackup@[B]YOUR_SERVER_IP[/B]:/backups/`hostname -f`


# Backup all of /home and /var/www except a few
# excluded directories
borg create -v --stats                          \
    $REPOSITORY::'{hostname}-{now:%Y-%m-%d_%H:%M}'    \
    /home                    \
    /var/www/html                    \
    /etc                    \
    /usr/local/directadmin > /tmp/borg-stat.tmp 2>&1


if [ "$?" -le 1 ]; then
borg prune -v $REPOSITORY --prefix '{hostname}-' \
    --keep-daily=7 --keep-weekly=4 --keep-monthly=6
else
    date >> /tmp/borg-stat.tmp
        mail -s "backup failed on server `hostname -f`" [B][email protected][/B] < /tmp/borg-stat.tmp
fi


# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly
# archives of THIS machine. The '{hostname}-' prefix is very important to
# limit prune's operation to this machine's archives and not apply to
# other machine's archives also.

On any failure it should mail you ([email protected] in script) about problems with it. Retention can be also specified/changed in the script.

This article is just to show alternative (free) ways to setup and incremental backup, until it's fully integrated into DirectAdmin.
 
Last edited:
hey smtalk, for centos 7, it will be this command

yum -y install epel-release
 
I am trying restic. Seems similar with more Backup choices..
 
Even if tools like ClusterLogics and R1Soft integrate with DirectAdmin, and incremental backup solutions like https://www.modmaster.pro/ exist, if you'd like to use free incremental backup utility (while it gets integrated into DirectAdmin), here is a short description of the idea:
DirectAdmin has "Admin Backup/Transfer" feature, which could create you a backup without any data in /home. This should take minimal space on the system, as main data (emails and website data) is stored in home. We could use any utility (borg, restic, riff-backup) to backup whole /home folder.

The steps would be:
1) Go to "Admin Backup/Transfer" feature, "Schedule" a backup to be ran as often as you'd like to (once a day might be enough), de-select "Domains Directory" and "E-mail data", place your backups to /home/admin/admin_backups for example.
2) Setup restic (supports many backup storage types) or borg (SSH required on the other end). Listing steps for borg, but things would be very similar for restic too.
a) install borg, commands for CentOS:
Code:
yum -y install epel-release
yum -y install borgbackup

Debian:
Code:
apt install borgbackup
b) initialize a local OR remote repository:
local (using /backups, but you can set any directory you want):
Code:
borg init --encryption=none /backups

remote (using borgbackup SSH user on remote server having privileges to /backups/hostname):
Code:
REPOSITORY=borgbackup@[B]YOUR_SERVER_IP[/B]:/backups/`hostname -f`

I'd suggest adding ssh keys from root to that remote server, so that it could connect without a password.

c) create /usr/local/directadmin/scripts/custom/all_backups_post.sh script, which would automatically sync with borg backup server after every backup done, chmod the script to 700, content:
Code:
#!/bin/sh
REPOSITORY=borgbackup@[B]YOUR_SERVER_IP[/B]:/backups/`hostname -f`


# Backup all of /home and /var/www except a few
# excluded directories
borg create -v --stats                          \
    $REPOSITORY::'{hostname}-{now:%Y-%m-%d_%H:%M}'    \
    /home                    \
    /var/www/html                    \
    /etc                    \
    /usr/local/directadmin > /tmp/borg-stat.tmp 2>&1


if [ "$?" -le 1 ]; then
borg prune -v $REPOSITORY --prefix '{hostname}-' \
    --keep-daily=7 --keep-weekly=4 --keep-monthly=6
else
    date >> /tmp/borg-stat.tmp
        mail -s "backup failed on server `hostname -f`" [B][email protected][/B] < /tmp/borg-stat.tmp
fi


# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly
# archives of THIS machine. The '{hostname}-' prefix is very important to
# limit prune's operation to this machine's archives and not apply to
# other machine's archives also.

On any failure it should mail you ([email protected] in script) about problems with it. Retention can be also specified/changed in the script.

This article is just to show alternative (free) ways to setup and incremental backup, until it's fully integrated into DirectAdmin.

The restic one is really cool... I hope you all are thinking of using something with AWS support and not just remote server backups.
 
Last edited:
What we need - its not only incremental backup on admin side, but possibility to restore from such backup on user side.
Dou you have such plans in roadmap?
 
@smtalk you copy line for the script instead of correct remote repository initiate command ,

Code:
borg init --encryption=none borgbackup@YOUR_SERVER_IP:/backups/`hostname -f`



I'll not copy/paste you the usual extract command that bloggers post, but and interesting way for to make restorations easy

Strategy for restoration until better tools are added



Uses bor mount option* + mc (midnight commander)
Run

Code:
yum install mc

Code:
mkdir /rbackup

Code:
borg list borgbackup@YOUR_SERVER_IP:/backups/`hostname -f`

You will get result/s similar to this:
server-name-2019-07-14_16:37 Sun, 2019-07-14 16:37:49 [veverve.....





Code:
borg mount borgbackup@YOUR_SERVER_IP:/backups/`hostname -f`::server-name-2019-07-14_16:37 /rbackup


then you can start mc and just browse the files in the /rbackup

Code:
mc

(pic is generic added to give idea how mc look to people that never saw it or the dos nc ;) )

https://upload.wikimedia.org/wikipedia/commons/9/9b/Midnight_Commander_4.7.0.9_on_Ubuntu_11.04.png

and copy back what is needed (single file / files / folders or account backup files that contain the db)

after copying what is needed un mount it

Code:
borg umount /rbackup


*require FUSE
 
How does not backing up /home fulfill the purpose of incremental backup since most incremental changes are in the public_html and emails?
 
How does not backing up /home fulfill the purpose of incremental backup since most incremental changes are in the public_html and emails?

It just clicked for me, you're splitting the backups into two separate processes, using the admin backup tool to backup the settings and restic to handle the incremental files. I see why this is a one way process and restoration is tricky.

Wish there was an option to delete an older post.
 
The idea is as the following: DirectAdmin creates backups of everything except of /home/, and all data under /home/ will be backuped with borg.

So DirectAdmin will create a tar.gz file with users' profiles (i.e. metadata), mysql. And the borg does the rest.
 
The thing with Borg is that it doesn't really support for multi servers. In theory, yes, but Borg might need to rebuild its cache, and prevents simultaneous backups from different sources. So, if backing up for one server, Borg seems being good. For multiple servers, Borg might not be the solution.
 
The thing with Borg is that it doesn't really support for multi servers. In theory, yes, but Borg might need to rebuild its cache, and prevents simultaneous backups from different sources. So, if backing up for one server, Borg seems being good. For multiple servers, Borg might not be the solution.

You can have multiple borg repositories for multiple servers. What's the problem with it? :)
 
thanks, just wanna ask.

is possible to backup on google drive ? because cPanel have this featured.

and, how about jetbackup ? is that support on directadmin?
 
thanks, just wanna ask.

is possible to backup on google drive ? because cPanel have this featured.

and, how about jetbackup ? is that support on directadmin?

restic can be used for google drive. Regardinf Jetbackup - they’re planning to release their plugin for DA soon.
 
Thanks for this!
It will be an amazing feature. Is there any ETA when will be ready or at least a pre release?

@smtalk
I tested your script and i think that you must add the 'ssh://' prefix in the $REPOSITORY so it should be
REPOSITORY=ssh://borgbackup@YOUR_SERVER_IP:/backups/`hostname -f`

and there is one more limitation
borg by default uses /tmp and demands exec permissions which i think its not what we want
 
Last edited:
Back
Top