CLI backup strategy for incremental backup

smtalk

Verified User
Staff member
Even if tools like ClusterLogics and R1Soft integrate with DirectAdmin, and incremental backup solutions like https://www.modmaster.pro/ exist, if you'd like to use free incremental backup utility (while it gets integrated into DirectAdmin), here is a short description of the idea:
DirectAdmin has "Admin Backup/Transfer" feature, which could create you a backup without any data in /home. This should take minimal space on the system, as main data (emails and website data) is stored in home. We could use any utility (borg, restic, riff-backup) to backup whole /home folder.

The steps would be:
1) Go to "Admin Backup/Transfer" feature, "Schedule" a backup to be ran as often as you'd like to (once a day might be enough), de-select "Domains Directory" and "E-mail data", place your backups to /home/admin/admin_backups for example.
2) Setup restic (supports many backup storage types) or borg (SSH required on the other end). Listing steps for borg, but things would be very similar for restic too.
a) install borg, commands for CentOS:
Code:
yum -y install epel-release
yum -y install borgbackup
Debian:
Code:
apt install borgbackup
b) initialize a local OR remote repository:
local (using /backups, but you can set any directory you want):
Code:
borg init --encryption=none /backups
remote (using borgbackup SSH user on remote server having privileges to /backups/hostname):
Code:
REPOSITORY=borgbackup@[B]YOUR_SERVER_IP[/B]:/backups/`hostname -f`
I'd suggest adding ssh keys from root to that remote server, so that it could connect without a password.

c) create /usr/local/directadmin/scripts/custom/all_backups_post.sh script, which would automatically sync with borg backup server after every backup done, chmod the script to 700, content:
Code:
#!/bin/sh
REPOSITORY=borgbackup@[B]YOUR_SERVER_IP[/B]:/backups/`hostname -f`


# Backup all of /home and /var/www except a few
# excluded directories
borg create -v --stats                          \
    $REPOSITORY::'{hostname}-{now:%Y-%m-%d_%H:%M}'    \
    /home                    \
    /var/www/html                    \
    /etc                    \
    /usr/local/directadmin > /tmp/borg-stat.tmp 2>&1


if [ "$?" -le 1 ]; then
borg prune -v $REPOSITORY --prefix '{hostname}-' \
    --keep-daily=7 --keep-weekly=4 --keep-monthly=6
else
    date >> /tmp/borg-stat.tmp
        mail -s "backup failed on server `hostname -f`" [B]your@email.com[/B] < /tmp/borg-stat.tmp
fi


# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly
# archives of THIS machine. The '{hostname}-' prefix is very important to
# limit prune's operation to this machine's archives and not apply to
# other machine's archives also.
On any failure it should mail you (your@email.com in script) about problems with it. Retention can be also specified/changed in the script.

This article is just to show alternative (free) ways to setup and incremental backup, until it's fully integrated into DirectAdmin.
 
Last edited:

bdacus01

Verified User
Even if tools like ClusterLogics and R1Soft integrate with DirectAdmin, and incremental backup solutions like https://www.modmaster.pro/ exist, if you'd like to use free incremental backup utility (while it gets integrated into DirectAdmin), here is a short description of the idea:
DirectAdmin has "Admin Backup/Transfer" feature, which could create you a backup without any data in /home. This should take minimal space on the system, as main data (emails and website data) is stored in home. We could use any utility (borg, restic, riff-backup) to backup whole /home folder.

The steps would be:
1) Go to "Admin Backup/Transfer" feature, "Schedule" a backup to be ran as often as you'd like to (once a day might be enough), de-select "Domains Directory" and "E-mail data", place your backups to /home/admin/admin_backups for example.
2) Setup restic (supports many backup storage types) or borg (SSH required on the other end). Listing steps for borg, but things would be very similar for restic too.
a) install borg, commands for CentOS:
Code:
yum -y install epel-release
yum -y install borgbackup
Debian:
Code:
apt install borgbackup
b) initialize a local OR remote repository:
local (using /backups, but you can set any directory you want):
Code:
borg init --encryption=none /backups
remote (using borgbackup SSH user on remote server having privileges to /backups/hostname):
Code:
REPOSITORY=borgbackup@[B]YOUR_SERVER_IP[/B]:/backups/`hostname -f`
I'd suggest adding ssh keys from root to that remote server, so that it could connect without a password.

c) create /usr/local/directadmin/scripts/custom/all_backups_post.sh script, which would automatically sync with borg backup server after every backup done, chmod the script to 700, content:
Code:
#!/bin/sh
REPOSITORY=borgbackup@[B]YOUR_SERVER_IP[/B]:/backups/`hostname -f`


# Backup all of /home and /var/www except a few
# excluded directories
borg create -v --stats                          \
    $REPOSITORY::'{hostname}-{now:%Y-%m-%d_%H:%M}'    \
    /home                    \
    /var/www/html                    \
    /etc                    \
    /usr/local/directadmin > /tmp/borg-stat.tmp 2>&1


if [ "$?" -le 1 ]; then
borg prune -v $REPOSITORY --prefix '{hostname}-' \
    --keep-daily=7 --keep-weekly=4 --keep-monthly=6
else
    date >> /tmp/borg-stat.tmp
        mail -s "backup failed on server `hostname -f`" [B]your@email.com[/B] < /tmp/borg-stat.tmp
fi


# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly
# archives of THIS machine. The '{hostname}-' prefix is very important to
# limit prune's operation to this machine's archives and not apply to
# other machine's archives also.
On any failure it should mail you (your@email.com in script) about problems with it. Retention can be also specified/changed in the script.

This article is just to show alternative (free) ways to setup and incremental backup, until it's fully integrated into DirectAdmin.
The restic one is really cool... I hope you all i thinking of using something with AWS support and not just remote server backups.
 

s0laris

Verified User
What we need - its not only incremental backup on admin side, but possibility to restore from such backup on user side.
Dou you have such plans in roadmap?
 

DanielP

Verified User
@smtalk you copy line for the script instead of correct remote repository initiate command ,

Code:
borg init --encryption=none borgbackup@YOUR_SERVER_IP:/backups/`hostname -f`


I'll not copy/paste you the usual extract command that bloggers post, but and interesting way for to make restorations easy

Strategy for restoration until better tools are added



Uses bor mount option* + mc (midnight commander)
Run

Code:
yum install mc
Code:
mkdir /rbackup
Code:
borg list borgbackup@YOUR_SERVER_IP:/backups/`hostname -f`
You will get result/s similar to this:
server-name-2019-07-14_16:37 Sun, 2019-07-14 16:37:49 [veverve.....




Code:
borg mount borgbackup@YOUR_SERVER_IP:/backups/`hostname -f`::server-name-2019-07-14_16:37 /rbackup

then you can start mc and just browse the files in the /rbackup

Code:
mc
(pic is generic added to give idea how mc look to people that never saw it or the dos nc ;) )

https://upload.wikimedia.org/wikipedia/commons/9/9b/Midnight_Commander_4.7.0.9_on_Ubuntu_11.04.png

and copy back what is needed (single file / files / folders or account backup files that contain the db)

after copying what is needed un mount it

Code:
borg umount /rbackup

*require FUSE
 

justjosh

New member
How does not backing up /home fulfill the purpose of incremental backup since most incremental changes are in the public_html and emails?
 

justjosh

New member
How does not backing up /home fulfill the purpose of incremental backup since most incremental changes are in the public_html and emails?
It just clicked for me, you're splitting the backups into two separate processes, using the admin backup tool to backup the settings and restic to handle the incremental files. I see why this is a one way process and restoration is tricky.

Wish there was an option to delete an older post.
 

zEitEr

Super Moderator
The idea is as the following: DirectAdmin creates backups of everything except of /home/, and all data under /home/ will be backuped with borg.

So DirectAdmin will create a tar.gz file with users' profiles (i.e. metadata), mysql. And the borg does the rest.
 

gate2vn

Verified User
The thing with Borg is that it doesn't really support for multi servers. In theory, yes, but Borg might need to rebuild its cache, and prevents simultaneous backups from different sources. So, if backing up for one server, Borg seems being good. For multiple servers, Borg might not be the solution.
 

smtalk

Verified User
Staff member
The thing with Borg is that it doesn't really support for multi servers. In theory, yes, but Borg might need to rebuild its cache, and prevents simultaneous backups from different sources. So, if backing up for one server, Borg seems being good. For multiple servers, Borg might not be the solution.
You can have multiple borg repositories for multiple servers. What's the problem with it? :)
 

HostinganID

Verified User
thanks, just wanna ask.

is possible to backup on google drive ? because cPanel have this featured.

and, how about jetbackup ? is that support on directadmin?
 

smtalk

Verified User
Staff member
thanks, just wanna ask.

is possible to backup on google drive ? because cPanel have this featured.

and, how about jetbackup ? is that support on directadmin?
restic can be used for google drive. Regardinf Jetbackup - they’re planning to release their plugin for DA soon.
 
Top