How-to: cPanel to DA migration

I converted 700+ accounts last week with migration script. I would like to leave feedback here about what happened.

- All websites moved with rsync and then restored with DirectAdmin. I was thinking that I had enough space on my SSD drive. But it seems you need to have double space of what you are moving. Because it converts tar.gz files and create DA backup files after it. So when you move 1GB cpanel backup file, it becomes 2GB backup file in total. So everyone should be aware of that, otherwise you will wait for a long time and see that "not enough space error".

- My solution was rsync backup folders as part1, part2, part3. So disk space was not a problem anymore.

- Out of 700 websites, some of them had e-mail password issues. It is fixed by resetting their mail password.

- Some websites had 22 character mysql database name. So I needed to fix them manually by phpmyadmin and their wp-config files.

- Some websites are converted without some MySQL tables because of row sizes. It was same MySQL version on both servers. But some of them didn't want to work on DA. I solve that by changing SQL Row as compressed in cPanel and upload them manually to DirectAdmin. It happened on at least 10 websites I converted.

- With cPanel, many users had ea-72 path entries in their .htaccess files. That websites are showing 403 error when you try to open. When my clients reports this, I manually remove that entry from their .htaccess files and problem solved. So you need to check that if some websites are having errors.

- My biggest issue is with subdomains. Almost none of them was uploaded to directadmin server. It was in backup file, but many of them were not in the folder after extracted. So when client report it, I download their subdomain folder from cPanel backup and extract it to directadmin manually. This is the biggest issue, because files are lost in convert process. I was using migration script 0.28. So maybe it is fixed in new versions.

So my struggle is still with subdomains which I need to download and upload manually. Other than that, everything works without a problem.

I would like to thank DA Staff for making this migration process easier for all.
 
Update on Email transfer

stellar7 said:
The script is working great for me with one exception--it creates email accounts but only will transfer the emails/mail files when the user name matches the account name (which is a rare case for me). Source accounts use maildir. I don't see any errors/warnings except for the warning for merging the user when it matches the account name.

Any idea why they aren't copying? I can copy the mail data manually afterward, but it would be nice to not have to do that.
Click to expand...

I tried again today (after adding some additional logging to the cpanel_to_da script), and it worked, so I'm not sure why it wasn't transferring them before.
 
Code: 
tar: homedir/public_html/billings/.htaccess: Cannot open: Not a directory
tar: homedir/public_html/.htaccess: Cannot open: Not a directory

I always seem to end up with this error while restoring. Anything I can do except renaming the .htaccess to htaccess at backup time & doing the opposite while restoring?
 
I have noticed when people have a catch-all email in Cpanel it will be not migrated to the DirectAdmin server.
 
Restoration via command line

Hi

any of you know any option to restore accounts via command line so that we can automate things ?

for user in `ls /var/cpanel/users/`; do { /scripts/pkgacct ${user} /home/all_backups; rsync -avt /home/all_backups/cpmove-${user}.tar.gz root@your_directadmin_server.com:/home/admin/all_backups/cpmove-${user}.tar.gz; rm -f /home/all_backups/cpmove-${user}.tar.gz ; }; done


I have servers with large amount of accounts and old server is not having enough space and also new server as well cant hold all backups. So I want to fiinsh things one by one and accounts in each of the cpanel servers are like 1000 or some is having more.
 
Thanks so much for creating this guide. I'm wondering if anyone has solved this problem creatively:

Full cPanel backups take time. In fact, a low capacity server of mine takes 24h to complete the backups. In that time many things can change on the source server, but the least of which is that customers continue to receive emails. The maildir format in DA is not the same as the maildir format on cPanel, so you can't just do a 1:1 rsync of /home/user/mail to /home/user/maildir or /home/user/imap. It looks like:

DA: /home/user/imap/domain.tld/user/maildir
cP: /home/user/mail/domain.tld/user

This is some logic that is going to take me a good while to script out I think, to make sure that I'm thinking of everything. But while I'm doing it, I'm curious if someone else has already done it and might be willing to share :D
 
mxroute said:
Thanks so much for creating this guide. I'm wondering if anyone has solved this problem creatively:

Full cPanel backups take time. In fact, a low capacity server of mine takes 24h to complete the backups. In that time many things can change on the source server, but the least of which is that customers continue to receive emails. The maildir format in DA is not the same as the maildir format on cPanel, so you can't just do a 1:1 rsync of /home/user/mail to /home/user/maildir or /home/user/imap. It looks like:

DA: /home/user/imap/domain.tld/user/maildir
cP: /home/user/mail/domain.tld/user

This is some logic that is going to take me a good while to script out I think, to make sure that I'm thinking of everything. But while I'm doing it, I'm curious if someone else has already done it and might be willing to share :D
Click to expand...

Did you see https://forum.directadmin.com/showthread.php?t=58066 ? :) Maildir format matches, so, rsync can be used.
 
Hello!
I've been tryting to transfer few accounts from cPanel to new DirectAdmin server. Apart from other issues with DA, I would like to point out couple things that I noticed:

1) Rsync transfer does not work when PasswordAuthentication in /etc/ssh/sshd_config is set to no (might be obvious for most users, but for new ones that might be tricky to figure out);

2) It is not possible to restore admin user from cPanel, because DirectAdmin already has an automatically set admin user.
Now this one I would say could be changed. I personally would like to restore my admin account with all my personal and business domains onto admin account in DA. I assume others would like that too. How about making this possible, or giving users a choice if they would like to do restore onto same username or different?
Or maybe there is some sort of issue, which forbids of doing this?

I was also thinking of changing the username in cPanel, but it is not recommended, since it can break few things. So I am a bit stuck at this point.
Would changing the transfered backup file name help? For example if I would change cpmove-admin.tar.gz on DA server to cpmove-adm.tar.gz? Would it still be restored as admin, or adm?

(I posted this comment also in here: https://forum.directadmin.com/showthread.php?t=52792&page=3&p=302749#post302749, but I realised that this thread is newer, so please remove the comment in that other thread if needed).

Thank you!

Ačiū labai, tautieti! Šaunuolis!
 
smtalk said:
As we're getting more and more questions on how to use https://forum.directadmin.com/showthread.php?t=52792&p=280613#post280613, I've decided to write this article to make it clear :)

See the 2nd post if you'd just like to try it for a single or couple accounts.

FAQ for new customers: http://forum.directadmin.com/showthread.php?t=58062&p=296460#post296460

---
Known issues:
* RoundCube data is converted from sqlite (like ./homedir/etc/domain.com/user.rcube.db) just since version 0.0.14, make sure you run the latest.
* cPanel mdbox is supported in ver. 0.0.7, but hasn't been tested very throughly.
* Mailman mailing lists do not get converted to majordomo mailing lists
* Please make sure it's you who created cpmove-user.tar.gz backup for security purposes, because at this moment we fully trust the content inside, thus available on admin level only. This might change later.
* DirectAdmin format for mysql usernames/databases is always username_database and username_user. cPanel has the same format, but due to max username/database length (https://forums.cpanel.net/threads/username-length-limits.630671/), is sometime shortens the username part in from of it. cPanel allows myusername_ to have a DB username with myuserna_. DA DB names/usernames always start with myusername_, changing this would create a mess, for example, you could have myuserna user owning myuserna_db1 and myusername user owning myserna_db2. The script converts username to be username in such cases, so, for some accounts you might need to change database username/name in wp-config.php or other files CMS uses. This should be rare though. Script output in such cases:
Code: 
WARNING! us_wordpress cannot be owned by user, renaming database user to user_wordpress
* JetBackup format doesn't include cpmove-user directory inside, so, for now these need to be converted manually, like (would store converted backup in /home/admin/converted_user_backup):
Code: 
jetbackup=yes /usr/local/directadmin/scripts/cpanel_to_da/cpanel_to_da.sh /home/admin/user_backups/cpmove-[B]USERNAME[/B].tar.gz /home/admin/converted_user_backup
chown -R admin. /home/admin/converted_user_backup

Other things to take into consideration:
* It's recommended to leave all the cpmove-user.tar.gz files on the system after restore. If there are bugs, or something goes wrong, it'd still be possible to recover that data.
* Like any software - there might be bugs there, if you notice any - please let us know, and we'll try fixing them as soon as possible.
* DirectAdmin supports a different feature set than cPanel. For example, DirectAdmin supports nginx/openlitespeed, MySQL8, rspamd etc., but it has no support for PostgreSQL or Ruby. So, if you have any sites using them - they'd need to be transferred manually.
* Max username length is 16 characters for MySQL users by default, and 10 for system user. Max username length can be increased in /usr/local/directadmin/conf/directadmin.conf. You can find max length of your username in cPanel easily:
Code: 
ls /var/cpanel/users | awk '{print length, $0}' | sort -nr | head -n1

To increase it in DA to 16, for example, you can simply run:
Code: 
/usr/local/directadmin/directadmin set max_username_length [B]16 [/B]restart
* To make sure your MySQL username/database name is fine, I'd suggest upgrading your MySQL/MariaDB on the server (https://www.directadmin.com/features.php?id=2294), MySQL 5.7+ or MariaDB 10.x is recommended.
* I'd suggest trying it with 1 account first, most preferably - the one with the longest database name/username (guide for this in 2nd post) to make sure you have the nameservers you need and that you don't hit any limitations :)
* If you have user.tar.gz instead of cpmove-user.tar.gz, it will not work! pkacct forms cpmove-users.tar.gz. If you have a list of user.tar.gz files, and want to rename them all in current directory:
Code: 
for i in `ls *.tar.gz | grep '^[A-Za-z0-9]*.tar.gz'`; do { mv -vf ${i} cpmove-${i}; }; done
* If you have backup-...._user.tar.gz instead of cpmove-user.tar.gz, it will not work! pkacct forms cpmove-users.tar.gz. If you have a list of backup-...._user.tar.gz files, and want to rename them all in current directory:
Code: 
for i in `ls backup-[0-9]*.*.tar.gz`; do { USERNAME=`echo $i | cut -d_ -f3 | cut -d'.' -f1`; mv -v $i cpmove-${USERNAME}.tar.gz; }; done
* If you don't see the user in "Move users between resellers" list, please apply: https://help.directadmin.com/item.php?id=251
* If you did not restore resellers first, all users were assigned to "admin". If you want to re-assign them to correct resellers after restore:
Code: 
cd /home/admin/all_backups
for i in `ls user*.tar.gz`; do { RESELLER=`echo $i | cut -d. -f2`; USER=`echo $i | cut -d'.' -f3`; /usr/local/directadmin/scripts/move_user_to_reseller.sh $USER admin $RESELLER; }; done
echo "action=cache&value=showallusers" >> /usr/local/directadmin/data/task.queue
/usr/local/directadmin/dataskq d
---

MIGRATION STEPS:
Steps to transfer ALL the accounts from cPanel server to DirectAdmin:
cPanel server:
1) Create a backup of all cPanel accounts on cPanel server and place them to /home/all_backups (make sure you have enough of space for them, if not, please read the suggestions at the end of the article):
Code: 
mkdir -p /home/all_backups
for user in `ls /var/cpanel/users/`; do { /scripts/pkgacct ${user} /home/all_backups; }; done
2) Transfer /home/all_backups to DirectAdmin server:
Code: 
rsync -avt --delete /home/all_backups/ root@[B]your_directadmin_server.com[/B]:/home/admin/all_backups/

We're done with cPanel server now. Let's connect to DirectAdmin server now.

DirectAdmin server:
1) Make sure your DirectAdmin version is at least 1.57.4 (as of time of writing, pre-release: https://help.directadmin.com/item.php?id=408), make sure backups at /home/admin/all_backups are owned by "admin" and that you have latest version of convertor script
Code: 
cd /usr/local/directadmin/custombuild
./build update
./build cpanel_to_da
chown -R admin. /home/admin/all_backups
2) We finally have DirectAdmin backups in /home/admin/all_backups :) Now we can simply go to DirectAdmin "Admin Backup/Transfer" section, check "Backup/Restore Settings" (defaults should be fine) in Admin level and restore all backups from "/home/admin/all_backups" directory.

That's it!

-----
If you have enough of space, I'd suggest leaving /home/admin/all_backups still there on DA server for a couple of weeks/months, if there is something missing in DA - you'd always have it in that backup.

If your cPanel server doesn't have enough of space - it's easy transfer accounts one-by one directly to the home folder on a new server. The most convenient way to do this is to generate SSH keys on cPanel server using:
Code: 
ssh-keygen

Then you'll have your public key placed in /root/.ssh/id_rsa.pub. Just copy the content of it to your DirectAdmin server, file /root/.ssh/authorized_keys (if /root/.ssh directory does not exist - create it). This will make it possible to connect to DA server without any password (from your cPanel server). Now instead of steps 1) and 2) on cPanel server, just do this:
Code: 
for user in `ls /var/cpanel/users/`; do { /scripts/pkgacct ${user} /home/all_backups; rsync -avt /home/all_backups/cpmove-${user}.tar.gz root@your_directadmin_server.com:/home/admin/all_backups/cpmove-${user}.tar.gz; rm -f /home/all_backups/cpmove-${user}.tar.gz ; }; done
Click to expand...

This would work fine if you had 2 servers. 1 with cpanel installed and another with DA installed, but what do you do if you just have 1 server and want to reformat the server with DA

I have all then cpmove files saved on an external HDD
 
jhoney said:
Might be easier to convert them first back to maildir on the cPanel server as it's built in?
Click to expand...

Just rsync:

From your cpanel:
Code: 
rsync -rvz /home/cpusername/mail/domain.tld/user/cur/ root@daserver:/home/dausername/imap/domain.tld/user/Maildir/cur/

Repeat this step for another user
and then correct owner in your DA:

Code: 
chown dausername: /home/dausername/imap/domain.tld -R
 
Thanks for writing up this guide. In case it helps further development or others doing these migrations, I'd like to share a few things that became important to my customers after migrating from cPanel to DirectAdmin.

1. Email catch-alls didn't migrate. They use the same syntax in the alias files and could potentially be migrated.
2. Autoresponders did migrate, but in a bad way. The alias files end up piping emails to a cPanel binary that does not exist on the DA system, and the migration did carry those over. They had to be manually cut out of the alias files.
3. Roundcube contacts did not migrate. This is to be expected, given that DA sticks to RC defaults and uses MySQL, while cPanel now uses SQLite.
4. Email filters did not migrate. This also makes sense, given that the syntax is not completely compatible and cPanel customers can use reg-ex, making it nearly impossible to account for all variables in such a migration.
5. Mailman aliases ported over as well, and those are not functional as they also pipe to a cPanel binary similar to the autoresponders.
 
Hi,

This tool is not working, on most of the accounts I am trying to migrate I end up with the following error:
 
Code: 
Error with query:
ALTER DATABASE agri_wp883 CHARACTER SET 'utf8mb4' COLLATE 'latin1_swedish_ci'
COLLATION 'latin1_swedish_ci' is not valid for CHARACTER SET 'utf8mb4'
Unable to restore database agri_wp883.sql to agri_wp883 : ERROR 1273 (HY000) at line 592: Unknown collation: 'utf8mb4_unicode_520_ci'

Unable to restore database chandi_data1.sql to chandi_data1 : ERROR 1273 (HY000) at line 576: Unknown collation: 'utf8mb4_unicode_520_ci'

Max username length (10) is smaller than cPanel username (13)

Error converting file: /home/admin/converted_user_backup/cpmove-drparneetkaur.tar.gz: Converting drparneetkaur (/home/admin/converted_user_backup/cpmove-drparneetkaur.tar.gz)...


File '/home/admin/converted_user_backup/cpmove-drparneetkaur.tar.gz' has not been restored
WARNING! gearingf_wp908 cannot be owned by gearingfor, renaming database user to gearingfor_wp908
Trying to find files in public_html to rename gearingf_wp908 to gearingfor_wp908. A copy of the file will have '.cpanel_backup_copy_dbname.php' appended at the end.
/home/admin/converted_user_backup/gearingfor/domains/gearingforsuccess.ca/public_html/wp-config.php
WARNING! gearingf_wp908 username cannot be owned by gearingfor, renaming database user to gearingfor_wp908
Trying to find files in public_html to rename gearingf_wp908 to gearingfor_wp908. A copy of the file will have '.cpanel_backup_copy.php' appended at the end.

Error with query:
ALTER DATABASE gearingfor_wp908 CHARACTER SET 'utf8mb4' COLLATE 'latin1_swedish_ci'
COLLATION 'latin1_swedish_ci' is not valid for CHARACTER SET 'utf8mb4'
Unable to restore database gearingfor_wp908.sql to gearingfor_wp908 : ERROR 1273 (HY000) at line 592: Unknown collation: 'utf8mb4_unicode_520_ci'

Error with query:
ALTER DATABASE gkmoga_wp728 CHARACTER SET 'utf8' COLLATE 'latin1_swedish_ci'
COLLATION 'latin1_swedish_ci' is not valid for CHARACTER SET 'utf8'
Unable to restore database gkmoga_wp728.sql to gkmoga_wp728 : ERROR 1273 (HY000) at line 312: Unknown collation: 'utf8mb4_unicode_520_ci'

WARNING! shrivard_newdata cannot be owned by shrivardan, renaming database user to shrivardan_newdata
Trying to find files in public_html to rename shrivard_newdata to shrivardan_newdata. A copy of the file will have '.cpanel_backup_copy_dbname.php' appended at the end.
/home/admin/converted_user_backup/shrivardan/domains/shri-vardan.com/public_html/wp-config.php
WARNING! shrivard_newuser username cannot be owned by shrivardan, renaming database user to shrivardan_newuser
Trying to find files in public_html to rename shrivard_newuser to shrivardan_newuser. A copy of the file will have '.cpanel_backup_copy.php' appended at the end.
/home/admin/converted_user_backup/shrivardan/domains/shri-vardan.com/public_html/wp-config.php


Unable to add user shrivardan_newuser. String 'shrivardan_newuser' is too long for user name (should be no longer than 16)
Can't add user shrivardan_newuser to shrivardan_newdata: Unable to add user shrivardan_newuser. String 'shrivardan_newuser' is too long for user name (should be no longer than 16)

Unable to restore database shrivardan_newdata.sql to shrivardan_newdata : ERROR 1273 (HY000) at line 30: Unknown collation: 'utf8mb4_unicode_520_ci'

These I was just trying to import 10 accounts and I did as per your guide on first page. These websites are basic wordpress websites and tool is not able to restore them. can you plz fix the tool ??

Thanks.
 
First post has both issues mentioned and how to fix them.

1) Increase max username length
2) Update MySQL/MariaDB version
 
I have a trial license to try out DA before I make the switch. As part of testing, I migrated a dummy account which has 1 primary domain and 2 alias in cpanel using the guide in 2nd post of this thread. The backup restored successfully but I'm not able to access the account. Using "Login as user" just logged me out of DA and when I tried to change the password for the restored used, it said its an invalid user?

Also, the restored account has only the primary domain. The other 2 alias domains are not to be found. What am I doing wrong?
 
The migration tool seems to work great overall. Thank you. I've migrated about 50 accounts so far with great success

The one issue I just noticed is that email account quotas are not being restored. All email accounts appear to be restored as "unlimited" regardless of their quota on the cPanel server. Is this a bug or something that just isn't being done? It should be possible to parse the mailbox quota from the first line of the maildirsize file in each account, if a quota is set.

Also I noticed that /usr/local/directadmin/scripts/add_email.sh will never write a quota if the quota file does not already exist. Possibly this is where the issue is happening because on my restored accounts there is no /etc/virtual/<domain>/quota file at all. If the restore is using that script to create the email accounts, there will never be a quota file created. It looks to me that the
if [ -e ${QUOTA} ]; then

needs an
Code: 
else
        echo "${EMAIL}:${QUOTAVAL}" > ${QUOTA}

So the file gets created with the first entry if it doesn't exist.

Thanks,
Ron
 
@capalon it doesn't use add_email.sh for that. Do you have any quota set in /etc/virtual/domain.com/passwd file?
 
@smtalk All lines in the passwd files have bytes=0M unless I've manually modified the email account limits in the web GUI.

Based on what I saw in the add_email.sh script, I've written a script that logs back into the old cPanel server, and reads the maildirsize file for each email account. I run it right after the backup is restored in DA, and it writes the quota entries to the /etc/virtual/domain.com/quota file. I did not know that there was a quota in the passwd file also since that isn't done in the add_email.sh script when it writes to the passwd file.

I just found that if I now run

echo "action=rewrite&value=email_passwd&user=USER" >> /usr/local/directadmin/data/task.queue

after writing the quota file, that it will update the passwd file to have the correct quotas. Once I do that it looks correct.

For what it is worth, I'm completely new to DA and liking it so far, but struggling to find what I can be sure is up-to-date/accurate info in a lot of cases when searching the forums, knowledgebase, and version feature history. There seems to be a lot of information available, but is difficult to know whether it is still valid or not. It seems like the add_email.sh script is a bit outdated.

It appears I have a functional workaround based on what I have written, but it would of course be better if the restore handled it directly so it works for everyone else. I'd like to get this working before I get to my larger accounts.

I'm willing to do whatever testing or debugging you need to help sort this out within the cpanel to DA process, and am comfortable with C/C++, PHP, BASH, and several other languages, if you tell me where to look and everything that needs to be done.

Regards,
Ron
 
You must log in or register to reply here.
Back
Top