Build MariaDB deletes all databases

ZipperZapper

Verified User
Joined
Nov 30, 2015
Messages
132
I don't know if this behaviour is intended or a bug, but for me it was a surprise.
I had some issues with MariaDB crashing the last week so I decided to rebuild it to see if it would fix things, wouldn't hurt either right?

Custombuild > Build Software > Build MariaDB.

I thought this worked fine and went on with my day, until some monitor alerts came up. Turns out all database users and databases itself were deleted in this process.
Luckily I have a good backup strategy and was able to restore the databases within half an hour, but it's not fun to do.

Is this intended behaviour? Maybe some sort of warning would be in place in that case.

Custombuild 2.0.0 (rev: 2895)
Directadmin 1.641
Ubuntu 18.04.6
 

Zhenyapan

Verified User
Joined
Feb 23, 2018
Messages
1,343
Location
UA
Fs you say - mysql was crached, that's why it drops DBs.
You must restore mysql after crash, then make backups, check their consistency, and only after all this - rebuild/reinstall mysql if needed.
 

ZipperZapper

Verified User
Joined
Nov 30, 2015
Messages
132
Crash was maybe a bit much. I had lot of "the service mysql stopped" messages all day, that's why I wanted to try a rebuild.
It was running, working fine and all data was there before I did the rebuild.
 

Zhenyapan

Verified User
Joined
Feb 23, 2018
Messages
1,343
Location
UA
strange, and what was in logs about this "mysql stopped"? maybe openfiles limit or OOM killer?
 

ZipperZapper

Verified User
Joined
Nov 30, 2015
Messages
132
I will open another topic about that.

This one is purely to know if Build MariaDB is intended to delete everything or not.
 

Zhenyapan

Verified User
Joined
Feb 23, 2018
Messages
1,343
Location
UA
No, mariadb in normal cases didn't remove anything. We always update/upgrade Mariadb without data loss. Something went wrong in this specific case.
 

ZipperZapper

Verified User
Joined
Nov 30, 2015
Messages
132
Yes, update/upgrade is never a problem in regular usage.

Mind this is about the "Build Software" tab in Custombuild, not the regular update procedure.
 

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
10,482
Location
LT, EU
I think it just updated the datadir :) Your data is likely in /home/mysql. Am I right? Check:
Code:
ls /var/lib/mysql
ls /home/mysql
 

ZipperZapper

Verified User
Joined
Nov 30, 2015
Messages
132
Hi,

Yes, that seems to be the case. The db's in the var folder are all created during the restore this morning.
The home ones seem te be created when the websites were created.

Which folder is going to be backed up by the Admin Backup function? Should I delete one of the 2?
 

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
10,482
Location
LT, EU
Please check if the following solves your problem:
Code:
service mysqld stop
mv /var/lib/mysql /var/lib/mysql_backup
ln -s /home/mysql /var/lib/mysql
service mysqld start
 

wtptrs

Verified User
Joined
Jul 13, 2015
Messages
275
Had the same issue yesterday on Debian, creating the

Code:
ln -s /home/mysql /var/lib/mysql

symlink indeed fixed it for me. And indeed the databases are not deleted, just the datadir changed after upgrading without the symlink.
 

ZipperZapper

Verified User
Joined
Nov 30, 2015
Messages
132
As I understand it correctly this will put the 'original' data back.

Is it needed? It's been a few hours since the restore and I don't want to cut another few hours of data out if it's not needed.

Edit: I mean: is it needed to make the symlink to make sure everything like backups and stuff keep working, or is it fine like this and will all systems use the new /var location by default.

Doing the symlink would have fixed it when the issue was happening this morning but I couldn't wait for the solution haha.
 

wtptrs

Verified User
Joined
Jul 13, 2015
Messages
275
Better check with smtalk. As I understand it, if everything is working fine now, you can leave it as it is and remove the directory with the data you're not using anymore.

@smtalk I think the issue is that when executing ./build mariadb, Custombuild checks if /var/lib/mysql already exists, and when you're using /home/mysql as datadir (which a lot of our servers, with a default DA install, are using), /var/lib/mysql will not be found and a new MariaDB datadir will be created and configured here. Perhaps the Custombuild script could be tweaked, because I think a lot of users will experience this issue in the future when trying to update MariaDB.
 

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
10,482
Location
LT, EU
Better check with smtalk. As I understand it, if everything is working fine now, you can leave it as it is and remove the directory with the data you're not using anymore.

@smtalk I think the issue is that when executing ./build mariadb, Custombuild checks if /var/lib/mysql already exists, and when you're using /home/mysql as datadir (which a lot of our servers, with a default DA install, are using), /var/lib/mysql will not be found and a new MariaDB datadir will be created and configured here. Perhaps the Custombuild script could be tweaked, because I think a lot of users will experience this issue in the future when trying to update MariaDB.
It's been tweaked already in the latest revision.
 

Richard G

Verified User
Joined
Jul 6, 2008
Messages
8,995
Location
Maastricht
(which a lot of our servers, with a default DA install, are using),
Which distro does that? Because on a default DA install I've never seen /home/mysql being used, it was always /var/lib/mysql.
So do Debian/Ubuntu default DA installs use /home/mysql directory??
 

wtptrs

Verified User
Joined
Jul 13, 2015
Messages
275
Which distro does that? Because on a default DA install I've never seen /home/mysql being used, it was always /var/lib/mysql.
So do Debian/Ubuntu default DA installs use /home/mysql directory??
Not anymore, I think DA changed their setup.sh script for Debian/Ubuntu some time ago, but they used to in the past. So we still have a lot of servers which were installed a few years ago and have been OS upgraded (i.e. Debian 8 -> 11) since, which still use /home/mysql.
 

ZipperZapper

Verified User
Joined
Nov 30, 2015
Messages
132
@smtalk I somehow seems to have created a little mess. Turns out my regular backups now run into this error:

Code:
RoundCube Backup Error:
Fatal error: Uncaught mysqli_sql_exception: Table 'da_roundcube.users' doesn't exist in engine in /usr/local/directadmin/scripts/backup_roundcube.php:150
Stack trace:
#0 /usr/local/directadmin/scripts/backup_roundcube.php(150): mysqli->query('SELECT * FROM  ...')
#1 {main}
  thrown in /usr/local/directadmin/scripts/backup_roundcube.php on line 150

I don't even use Roundcube by the way, but fair enough. I checked /home/mysql and indeed saw "da_roundcube" in there which wasn't in /var/lib/mysql.

So decided to try to set back the original data as you said:

Code:
service mysqld stop
mv /var/lib/mysql /var/lib/mysql_backup
ln -s /home/mysql /var/lib/mysql
service mysqld start

However, now mysqld won't start anymore:

Code:
aug 03 14:16:45 systemd[1]: Starting MySQL database server...
aug 03 14:16:45 systemd[21906]: mysqld.service: Failed to set up mount namespacing: No such file or directory
aug 03 14:16:45 systemd[21906]: mysqld.service: Failed at step NAMESPACE spawning /usr/local/mysql/bin/mysqld: No such file or directory
aug 03 14:16:45 systemd[1]: mysqld.service: Main process exited, code=exited, status=226/NAMESPACE
aug 03 14:16:45 systemd[1]: mysqld.service: Failed with result 'exit-code'.
aug 03 14:16:45 systemd[1]: Failed to start MySQL database server.

I then restored the previous situation and simply copied 'da_roundcube' from /home/mysql to /var/lib/mysql but the backup task will still fail with the same error.

What am I missing here?
 

Zhenyapan

Verified User
Joined
Feb 23, 2018
Messages
1,343
Location
UA
1. maybe wrong symlink owner
2. it's not enough to copy roundcube database dir, this database must be listed in mysql db_list with described user/pass etc.. so if you don't want to install roundcube - you can try to create just database da_roundcube with table da_roundcube.users. but easier for me - install roundcube and remove it.
 
Top