Solved Need extra files included in admin backups

kristian · Dec 20, 2022

I'm doing some custom things with dovecot and keep some settings in an extra file in /etc/virtual/<domain>/. I need this file to be included in the DirectAdmin Admin Backup so they'll stick around when moving an account to another server. I cannot seem to find any way to tell DirectAdmin about additional files to include.

Anyone know if this is possible?

Richard G · Dec 20, 2022

As far as I know, admin backups can not include root file things. So I guess an option is to create a cron to copy them to the admin account somewhere so they can be backed up.

I don't know if they are included in a system backup or not.

Maybe @zEitEr knows more about this or has a good idea to include it the best way.

kristian · Dec 20, 2022

What I meant by Admin Backup is an Admin backup of a specific user account, so I need it to belong to the user. A workaround could be to copy the file into the user's home directory in a pre script, and move it into position again in a post script, but I'm not really sure I see a pre/post script for this particular use case.

Ohm J · Dec 20, 2022

maybe "user_backup_compress_pre.sh" is your choice.

user_backup_compress_pre.sh, run just before backup tar.gz compression.

www.directadmin.com

but hook script doesn't tell you anything that come from admin backup or reseller or user self backup

maybe this can determine ( This just idea, not full usage script )

Code:

#!/usr/bin/bash
ADMIN_BK_DIR="/home/admin/admin_backups"
ADDITION_BK_DIR="${ADMIN_BK_DIR}/${username}/custom_virtual"

#use regex that $file should start with "admin_backups" directory. otherwise, it should skip.
if [[ ! "$file" =~ "^${ADMIN_BK_DIR}.*" ]]; then
   #not admin backup, should break here,
    exit 0;
fi

#prepare to store custom file/folder
#All admin backup will prepare store file like this
#/home/admin/admin_backups/user01
#exit 1; should treats as fail backup.
mkdir "${ADDITION_BK_DIR}" || exit 1;

#copy your custom file here to prepare file/folder backup
cp -rf "/etc/virtual/somedomain" "${ADDITION_BK_DIR}"

#let script continue to start compress backup.
exit 0;

zEitEr · Dec 20, 2022

Yes, that's a correct hook. I use the same:

- Add to backup: /usr/local/directadmin/scripts/custom/user_backup_compress_pre.sh
- Restore from a backup: /usr/local/directadmin/scripts/custom/user_restore_post.sh

What I'd suggest is using directories. Something like these:

- Add to backup: /usr/local/directadmin/scripts/custom/user_backup_compress_pre/dovecot_custom.sh
- Restore from a backup: /usr/local/directadmin/scripts/custom/user_restore_post/dovecot_custom.sh

This way you won't get your custom scripts overwritten by another plugin.

kristian · Dec 21, 2022

I will give that a try. I didn't see any mention in the docs of what happens to any stdout/stderr output the script might send. Will it be shown anywhere in the message system message perhaps?

kristian · Dec 21, 2022

I'm seeing some .. interesting results using this approach. I think I need to be very careful about where exactly I place these custom files I back up. This is my current script:

Code:

# cat user_backup_compress_pre/dovecot_custom.sh
#!/bin/bash

# This script will add the custom dovecot virtual user settings to an admin
# backup.

admin_backup_dir="/home/admin/admin_backups"

# Check if we're doing an admin backup, because that's the only kind we care
# about.
if [[ ! "${file}" =~ ^${admin_backup_dir}/.* ]]; then
    # This backup is not an admin backup. Exit cleanly.
    echo "$(date) Not admin backup. Exiting." >> /tmp/foo
    exit 0
fi

# For each domain for the user being backed up, store the custom virtual user
# settings, if it exists (/etc/virtual/<domain>/user-settings).
for domain in $(grep -oP "^\K[^:]+(?=: ${username}$)" /etc/virtual/domainowners); do
    if [ -f "/etc/virtual/${domain}/user-settings" ]; then
        mkdir -p "${admin_backup_dir}/${username}/backup/dovecot-custom-user-settings-${domain}"
        cp -a "/etc/virtual/${domain}/user-settings" "${admin_backup_dir}/${username}/backup/dovecot-custom-user-settings-${domain}/"
        # When user-settings was mail:mail and 0600 it wasn't included in the
        # backup for some reason, and +r helped.
        chmod +r "${admin_backup_dir}/${username}/backup/dovecot-custom-user-settings-${domain}/user-settings"
    fi
done

exit 0

My user-settings file is placed as expected into /home/admin/admin_backups/<myuser>/backup/dovecot-custom-user-settings-<domain>/user-settings, and packed up with everything else. All good so far.

When I restore this backup, however, it seems like dovecot-custom-user-settings-<domain> is treated as its own separate domain, causing all kinds of errors (from message from Message System):

Code:

Domain dovecot-custom-user-settings-<redacted> Created: Domain Created Successfully

Domain <redacted> Created: Domain Created Successfully

E-Mail: Unable to read /home/<redacted>/backups/backup/dovecot-custom-user-settings-<redacted>/email/passwd : Unable to open /home/<redacted>/backups/backup/dovecot-custom-user-settings-<redacted>/email/passwd for reading.
No such file or directory

Unable to read /home/<redacted>/backups/backup/dovecot-custom-user-settings-<redacted>/email/aliases Unable to open /home/<redacted>/backups/backup/dovecot-custom-user-settings-<redacted>/email/aliases for reading.
No such file or directory

Unable to read /home/<redacted>/backups/backup/dovecot-custom-user-settings-<redacted>/email/autoresponder.conf Unable to open /home/<redacted>/backups/backup/dovecot-custom-user-settings-<redacted>/email/autoresponder.conf for reading.
No such file or directory

Unable to read /home/<redacted>/backups/backup/dovecot-custom-user-settings-<redacted>/email/vacation.conf Unable to open /home/<redacted>/backups/backup/dovecot-custom-user-settings-<redacted>/email/vacation.conf for reading.
No such file or directory

Unable to read /home/<redacted>/backups/backup/dovecot-custom-user-settings-<redacted>/email/email.conf Unable to get Lock on file:
open error for /home/<redacted>/backups/backup/dovecot-custom-user-settings-<redacted>/email/email.conf.lock: No such file or directory
/home/<redacted>/backups/backup/dovecot-custom-user-settings-<redacted>/email: No such file or directory

Catch all is now set to :fail:

DNS: Unable to read backup_db: /home/<redacted>/backups/backup/dovecot-custom-user-settings-<redacted>/dovecot-custom-user-settings-<redacted>.db : Can't open file /home/<redacted>/backups/backup/dovecot-custom-user-settings-<redacted>/dovecot-custom-user-settings-<redacted>.db for reading

Merge DB File issue, restoring to <redacted>: (restore to ip)

This seems incredibly flimsy, and very undocumented, to me! Is there a safe place to put my custom files?

Ohm J · Dec 21, 2022

this just another idea that come to my head.

Im not sure about restore workflow of Directadmin. but from your log, DA extract the file to user home dir.

So... for safe place, you need to completely seperate compress your custom file from main backup. Like this

/home/admin/admin_backups/reseller.user.{some_user}.tar.gz
/home/admin/admin_backups/reseller.user.{some_user}-custom-dovecot.tar.gz

and when you want to restore, just use hook to extract file to correct location.

but maybe there have a better idea than this.

kristian · Dec 21, 2022

kristian said:
When I restore this backup, however, it seems like dovecot-custom-user-settings-<domain> is treated as its own separate domain

Some testing and guesswork later, I suspect this happened because something in the restore process is doing a lazy matching for anything with the domain name in it, and treating it as a subdomain. I've put the files into a two level custom directory now, and I don't have the same problems.

I also needed to use user_restore_post_pre_cleanup, to have the custom files available to me, and not user_restore_post. I still have some stuff to work out, but I'll post my resulting scripts when I'm satisfied with it. Then you all can tell me what I've still done wrong.

johannes · Dec 21, 2022

kristian said:
Then you all can tell me what I've still done wrong.

I´d more interested to get the final good working result for my own usage

kristian · Dec 21, 2022

This is what I've ended up with, that works the way I need it to. It feels a little weird having to do workarounds with chown/chmod, when it seems like the script runs as root. I don't know why this is necessary, but a good thing to be aware of.

Bit of a background: The intention of this script is to backup/restore my custom passwd-files that I use in my custom dovecot passdb config to be able to set extra fields for each email user. I have some other posts about that: https://forum.directadmin.com/threads/is-there-a-way-to-lock-mail-accounts-for-maintenance.57836/ and https://forum.directadmin.com/threa...s-by-ip-to-force-webmail-usage-and-2fa.65755/

Anyway, here are my first working versions of the backup/restore related scripts:

Code:

# cat /usr/local/directadmin/scripts/custom/user_backup_compress_pre/dovecot_custom.sh
#!/bin/bash

# https://www.directadmin.com/features.php?id=1183
# https://www.directadmin.com/features.php?id=978

# variables:
# username=joe
# reseller=resellername
# file=/path/to/the/user.tar.gz

# This script will add the following extra files to an admin backup:
# /etc/virtual/system-users-settings/<username>/user-settings
# /etc/virtual/<domain>/user-settings

admin_backup_dir="/home/admin/admin_backups"

# Check if we're doing an admin backup, because that's the only kind we care
# about.
if [[ ! "${file}" =~ ^${admin_backup_dir}/.* ]]; then
    # This backup is not an admin backup. Exit cleanly.
    exit 0
fi

# Store the system user's user-settings if it exists.
if [ -f "/etc/virtual/system-users-settings/${username}/user-settings" ]; then
    mkdir -p "${admin_backup_dir}/${username}/backup/dovecot-system-user-settings"
    cp -a "/etc/virtual/system-users-settings/${username}/user-settings" "${admin_backup_dir}/${username}/backup/dovecot-system-user-settings/"
    # user-settings must be world readable to be included in the backup.
    chmod +r "${admin_backup_dir}/${username}/backup/dovecot-system-user-settings/user-settings"
fi

# For each domain for the user being backed up, store the custom virtual user
# settings, if it exists (/etc/virtual/<domain>/user-settings).
for domain in $(grep -oP "^\K[^:]+(?=: ${username}$)" /etc/virtual/domainowners); do
    if [ -f "/etc/virtual/${domain}/user-settings" ]; then
        mkdir -p "${admin_backup_dir}/${username}/backup/dovecot-virtual-user-settings/${domain}"
        cp -a "/etc/virtual/${domain}/user-settings" "${admin_backup_dir}/${username}/backup/dovecot-virtual-user-settings/${domain}/"
        # user-settings must be world readable to be included in the backup.
        chmod +r "${admin_backup_dir}/${username}/backup/dovecot-virtual-user-settings/${domain}/user-settings"
    fi
done

exit 0

Code:

# cat /usr/local/directadmin/scripts/custom/user_restore_post_pre_cleanup/dovecot_custom.sh
#!/bin/bash

# https://www.directadmin.com/features.php?id=2230

# variables:
# $username
# $reseller   (user.conf creator)
# $filename

# This script will retrieve the custom dovecot user settings files from an
# admin backup.

admin_backup_dir="/home/admin/admin_backups"

# Check if we're doing an admin restore, because that's the only kind we care
# about.
if [[ ! "${filename}" =~ ^${admin_backup_dir}/.* ]]; then
    # This backup is not an admin backup. Exit cleanly.
    exit 0
fi

# Retrieve the system user's user-settings if it exists.
if [ -f "/home/${username}/backups/backup/dovecot-system-user-settings/user-settings" ]; then
    mkdir -p "/etc/virtual/system-users-settings/${username}"
    cp -a "/home/${username}/backups/backup/dovecot-system-user-settings/user-settings" "/etc/virtual/system-users-settings/${username}/"
    chown -R mail:mail "/etc/virtual/system-users-settings/${username}"
    chmod 0640 "/etc/virtual/system-users-settings/${username}/user-settings"
fi

# For each domain for the user being restored, retrieve the custom virtual user
# settings, if it exists (/home/<username>/backups/backup/dovecot-custom-user-settings/<domain>/user-settings).
for domain in $(grep -oP "^\K[^:]+(?=: ${username}$)" /etc/virtual/domainowners); do
    if [ -f "/home/${username}/backups/backup/dovecot-virtual-user-settings/${domain}/user-settings" ]; then
        cp -a "/home/${username}/backups/backup/dovecot-virtual-user-settings/${domain}/user-settings" "/etc/virtual/${domain}/"
        chown mail:mail "/etc/virtual/${domain}/user-settings"
        chmod 0640 "/etc/virtual/${domain}/user-settings"
    fi
done

exit 0

johannes · Dec 21, 2022

Wow, thats great, thanks for sharing!

Solved Need extra files included in admin backups

kristian

Verified User

Richard G

Verified User

kristian

Verified User

Ohm J

Verified User

user_backup_compress_pre.sh, run just before backup tar.gz compression.

zEitEr

Super Moderator

kristian

Verified User

kristian

Verified User

Ohm J

Verified User

kristian

Verified User

johannes

Verified User

kristian

Verified User

johannes

Verified User