CLI backup strategy for incremental backup

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
8,329
Location
LT, EU
Even if tools like ClusterLogics and R1Soft integrate with DirectAdmin, and incremental backup solutions like https://www.modmaster.pro/ exist, if you'd like to use free incremental backup utility (while it gets integrated into DirectAdmin), here is a short description of the idea:
DirectAdmin has "Admin Backup/Transfer" feature, which could create you a backup without any data in /home. This should take minimal space on the system, as main data (emails and website data) is stored in home. We could use any utility (borg, restic, riff-backup) to backup whole /home folder.

The steps would be:
1) Go to "Admin Backup/Transfer" feature, "Schedule" a backup to be ran as often as you'd like to (once a day might be enough), de-select "Domains Directory" and "E-mail data", place your backups to /home/admin/admin_backups for example.
2) Setup restic (supports many backup storage types) or borg (SSH required on the other end). Listing steps for borg, but things would be very similar for restic too.
a) install borg, commands for CentOS:
Code:
yum -y install epel-release
yum -y install borgbackup
Debian:
Code:
apt install borgbackup
b) initialize a local OR remote repository:
local (using /backups, but you can set any directory you want):
Code:
borg init --encryption=none /backups
remote (using borgbackup SSH user on remote server having privileges to /backups/hostname):
Code:
REPOSITORY=borgbackup@[B]YOUR_SERVER_IP[/B]:/backups/`hostname -f`
I'd suggest adding ssh keys from root to that remote server, so that it could connect without a password.

c) create /usr/local/directadmin/scripts/custom/all_backups_post.sh script, which would automatically sync with borg backup server after every backup done, chmod the script to 700, content:
Code:
#!/bin/sh
REPOSITORY=borgbackup@[B]YOUR_SERVER_IP[/B]:/backups/`hostname -f`


# Backup all of /home and /var/www except a few
# excluded directories
borg create -v --stats                          \
    $REPOSITORY::'{hostname}-{now:%Y-%m-%d_%H:%M}'    \
    /home                    \
    /var/www/html                    \
    /etc                    \
    /usr/local/directadmin > /tmp/borg-stat.tmp 2>&1


if [ "$?" -le 1 ]; then
borg prune -v $REPOSITORY --prefix '{hostname}-' \
    --keep-daily=7 --keep-weekly=4 --keep-monthly=6
else
    date >> /tmp/borg-stat.tmp
        mail -s "backup failed on server `hostname -f`" [B]your@email.com[/B] < /tmp/borg-stat.tmp
fi


# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly
# archives of THIS machine. The '{hostname}-' prefix is very important to
# limit prune's operation to this machine's archives and not apply to
# other machine's archives also.
On any failure it should mail you (your@email.com in script) about problems with it. Retention can be also specified/changed in the script.

This article is just to show alternative (free) ways to setup and incremental backup, until it's fully integrated into DirectAdmin.
 
Last edited:

tan87

Verified User
Joined
Jul 7, 2019
Messages
12
hey smtalk, for centos 7, it will be this command

yum -y install epel-release
 

bdacus01

Verified User
Joined
Jul 22, 2017
Messages
505
Location
Murfreesboro
Even if tools like ClusterLogics and R1Soft integrate with DirectAdmin, and incremental backup solutions like https://www.modmaster.pro/ exist, if you'd like to use free incremental backup utility (while it gets integrated into DirectAdmin), here is a short description of the idea:
DirectAdmin has "Admin Backup/Transfer" feature, which could create you a backup without any data in /home. This should take minimal space on the system, as main data (emails and website data) is stored in home. We could use any utility (borg, restic, riff-backup) to backup whole /home folder.

The steps would be:
1) Go to "Admin Backup/Transfer" feature, "Schedule" a backup to be ran as often as you'd like to (once a day might be enough), de-select "Domains Directory" and "E-mail data", place your backups to /home/admin/admin_backups for example.
2) Setup restic (supports many backup storage types) or borg (SSH required on the other end). Listing steps for borg, but things would be very similar for restic too.
a) install borg, commands for CentOS:
Code:
yum -y install epel-release
yum -y install borgbackup
Debian:
Code:
apt install borgbackup
b) initialize a local OR remote repository:
local (using /backups, but you can set any directory you want):
Code:
borg init --encryption=none /backups
remote (using borgbackup SSH user on remote server having privileges to /backups/hostname):
Code:
REPOSITORY=borgbackup@[B]YOUR_SERVER_IP[/B]:/backups/`hostname -f`
I'd suggest adding ssh keys from root to that remote server, so that it could connect without a password.

c) create /usr/local/directadmin/scripts/custom/all_backups_post.sh script, which would automatically sync with borg backup server after every backup done, chmod the script to 700, content:
Code:
#!/bin/sh
REPOSITORY=borgbackup@[B]YOUR_SERVER_IP[/B]:/backups/`hostname -f`


# Backup all of /home and /var/www except a few
# excluded directories
borg create -v --stats                          \
    $REPOSITORY::'{hostname}-{now:%Y-%m-%d_%H:%M}'    \
    /home                    \
    /var/www/html                    \
    /etc                    \
    /usr/local/directadmin > /tmp/borg-stat.tmp 2>&1


if [ "$?" -le 1 ]; then
borg prune -v $REPOSITORY --prefix '{hostname}-' \
    --keep-daily=7 --keep-weekly=4 --keep-monthly=6
else
    date >> /tmp/borg-stat.tmp
        mail -s "backup failed on server `hostname -f`" [B]your@email.com[/B] < /tmp/borg-stat.tmp
fi


# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly
# archives of THIS machine. The '{hostname}-' prefix is very important to
# limit prune's operation to this machine's archives and not apply to
# other machine's archives also.
On any failure it should mail you (your@email.com in script) about problems with it. Retention can be also specified/changed in the script.

This article is just to show alternative (free) ways to setup and incremental backup, until it's fully integrated into DirectAdmin.
The restic one is really cool... I hope you all i thinking of using something with AWS support and not just remote server backups.
 

s0laris

Verified User
Joined
Aug 10, 2007
Messages
26
What we need - its not only incremental backup on admin side, but possibility to restore from such backup on user side.
Dou you have such plans in roadmap?
 

DanielP

Verified User
Joined
Jun 28, 2019
Messages
82
@smtalk you copy line for the script instead of correct remote repository initiate command ,

Code:
borg init --encryption=none borgbackup@YOUR_SERVER_IP:/backups/`hostname -f`


I'll not copy/paste you the usual extract command that bloggers post, but and interesting way for to make restorations easy

Strategy for restoration until better tools are added



Uses bor mount option* + mc (midnight commander)
Run

Code:
yum install mc
Code:
mkdir /rbackup
Code:
borg list borgbackup@YOUR_SERVER_IP:/backups/`hostname -f`
You will get result/s similar to this:
server-name-2019-07-14_16:37 Sun, 2019-07-14 16:37:49 [veverve.....




Code:
borg mount borgbackup@YOUR_SERVER_IP:/backups/`hostname -f`::server-name-2019-07-14_16:37 /rbackup

then you can start mc and just browse the files in the /rbackup

Code:
mc
(pic is generic added to give idea how mc look to people that never saw it or the dos nc ;) )

https://upload.wikimedia.org/wikipedia/commons/9/9b/Midnight_Commander_4.7.0.9_on_Ubuntu_11.04.png

and copy back what is needed (single file / files / folders or account backup files that contain the db)

after copying what is needed un mount it

Code:
borg umount /rbackup

*require FUSE
 

justjosh

New member
Joined
Oct 13, 2019
Messages
18
How does not backing up /home fulfill the purpose of incremental backup since most incremental changes are in the public_html and emails?
 

justjosh

New member
Joined
Oct 13, 2019
Messages
18
How does not backing up /home fulfill the purpose of incremental backup since most incremental changes are in the public_html and emails?
It just clicked for me, you're splitting the backups into two separate processes, using the admin backup tool to backup the settings and restic to handle the incremental files. I see why this is a one way process and restoration is tricky.

Wish there was an option to delete an older post.
 

zEitEr

Super Moderator
Joined
Apr 11, 2005
Messages
13,853
Location
GMT +7.00
The idea is as the following: DirectAdmin creates backups of everything except of /home/, and all data under /home/ will be backuped with borg.

So DirectAdmin will create a tar.gz file with users' profiles (i.e. metadata), mysql. And the borg does the rest.
 

gate2vn

Verified User
Joined
Nov 9, 2004
Messages
298
The thing with Borg is that it doesn't really support for multi servers. In theory, yes, but Borg might need to rebuild its cache, and prevents simultaneous backups from different sources. So, if backing up for one server, Borg seems being good. For multiple servers, Borg might not be the solution.
 

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
8,329
Location
LT, EU
The thing with Borg is that it doesn't really support for multi servers. In theory, yes, but Borg might need to rebuild its cache, and prevents simultaneous backups from different sources. So, if backing up for one server, Borg seems being good. For multiple servers, Borg might not be the solution.
You can have multiple borg repositories for multiple servers. What's the problem with it? :)
 

HostinganID

Verified User
Joined
May 5, 2016
Messages
28
Location
Indonesia
thanks, just wanna ask.

is possible to backup on google drive ? because cPanel have this featured.

and, how about jetbackup ? is that support on directadmin?
 

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
8,329
Location
LT, EU
thanks, just wanna ask.

is possible to backup on google drive ? because cPanel have this featured.

and, how about jetbackup ? is that support on directadmin?
restic can be used for google drive. Regardinf Jetbackup - they’re planning to release their plugin for DA soon.
 

massive

Verified User
Joined
Feb 16, 2006
Messages
305
Thanks for this!
It will be an amazing feature. Is there any ETA when will be ready or at least a pre release?

@smtalk
I tested your script and i think that you must add the 'ssh://' prefix in the $REPOSITORY so it should be
REPOSITORY=ssh://borgbackup@YOUR_SERVER_IP:/backups/`hostname -f`

and there is one more limitation
borg by default uses /tmp and demands exec permissions which i think its not what we want
 
Last edited:
Top