Issues with backup restore running parallel

ItsOnlyMe

Verified User
Joined
Apr 3, 2009
Messages
124
Location
Netherlands
So I have been playing with this for a while now but i am really out of idea's.

I am trying to have Directadmin taskq run parallel to restore multiple users at once. For this I use the GNU parallel. I notice a lot of random "crashes" of the taskq.
- There are no visible errors
- I am unable to find something in the logs about this.
- There is no message send to the message system about the given restore.
I also end up with a lot of backup restore jobs in the overview on the "admin backup and restore" page. When clicking one of those jobs, it will tell you that it does not run anymore and starts clearing that job data, but said user is not restored.

I am also unable to find hooks for the creation of ftp accounts that its unable to get a lock on /etc/proftpd.passwd . For creation of users you can hook into the process so it will hold off a little bit before throwing a error. As said on https://docs.directadmin.com/developer/hooks/ftp.html you can call these scripts but there is no ftp_create_pre.sh in the documentation. So where would I call the actions to set a custom lock file and loop with a sleep to see if that file is there before proceeding?

The same thing is working for useradd with the user_create_(pre|post).sh. I think there are more files that Directadmin locks / checks if its able to create the user or not. Can I get a full list of files that have this which da checks / calls?

The restoration of Directadmin users is terribly slow hence the reason why i want this to be done in parallel. I am willing to privately share the scripts with Directadmin staff I use regarding the restoration of users if this is needed to debug the crashing of taskq jobs.
I call the taskq with this command:
Code:
/usr/local/directadmin/directadmin taskq --run="$_line"
Where $_line is the complete action line directadmin uses to restore a user.
 
What you mean slow? Maybe your accounts too big? Or disks/cpu slow?
Recently I moved 220 accounts 350gb compressed size and 650gb uncompgressed - restored on Ryzen 5950x and nvme disks in 3 hours.
I'm using tar.gz archives and PIGZ configured to 4 threads.
 
I have a lot of accounts to create over 2K users so the restore time is very long for only a user and mysql settings ( no database data or website data or mail data, just the settings for the users ).

Anyways I took a different route to do this without a Directadmin user restore which is 10000 times faster. So I have no need for this anymore.
 
Anyways I took a different route to do this without a Directadmin user restore which is 10000 times faster. So I have no need for this anymore
Care to share so that others also can benefit ? we are here to help each other right ? :)
 
Well, I cant share the full scripts that I have made for this but, how I did it is, creating all users that i would transfer first on the server with useradd

Code:
ssh -n -T root@"$REMOTEHOST" -p$SSHPOORT 'if [[ $(id '"$USER"' >/dev/null 2>&1; echo $?) -eq 1 ]]; then while true; do if [ ! -f /etc/passwd.lock ] || [ ! -f /etc/shadow.lock ] || [ ! -f /etc/group.lock ] || [ ! -f /etc/gshadow.lock ]; then useradd --no-create-home '"$USER"'; break; else sleep 1; fi; done; fi;';

Then have rsync send over the /home and /usr/local/directadmin/users/$USER
Use rsync without --numeric-ids so that the correct usernames/groups will be set on the transferred data

When all data has been send over, you need to fix the /etc/shadow file to get the correct password of the user back in there, and you need to fix the UID in /etc/virtual/$DOMAIN/passwd otherwise you are going to have issues with mail.
Just make sure you transfer all data from important paths that you can find on https://www.directadmin.com/paths.php .
Also this would only work if you are going to re-use the same IP as the old server like I do. Otherwise you need a few extra steps for replacing/updating the IP's in the DNS files etc.

Sorry that I can't share the full script that I currently use to make these migrations possible. For me it was needed due to CentOS6 -> CentOS7 or higher.
 
Otherwise you need a few extra steps for replacing/updating the IP's in the DNS files etc.

Did you try

Bash:
# /usr/local/directadmin/scripts/ipswap.sh
Usage:
/usr/local/directadmin/scripts/ipswap.sh <oldip> <newip> [<file>]
you gave #0: /usr/local/directadmin/scripts/ipswap.sh

for it? Or do you do it with your script as well?
 
for it? Or do you do it with your script as well?

Not needed for me in this case since I would be keeping the same IP from where the data came from. Only thing I would need to do is, swap the IP's on the nics them selfs.
 
Back
Top