Solved Mysql don't start anymore

[Richard G]

I think I have to wait until tonight when customers are sleeping and then reinstall mariadb and then restore all databases.

 

[Richard G]

Seems I overseen 1 dir. I could not drop the database due to the error.

Now I had some rest, found a da_roundcube directory in the /var/lib/mysql directory. So I stopped mysql, removed the da_roundcube directory, started mysql again and then I was able to build roundcube again.

Issue fixed, but I'm still not happy that the database suddenly start acting strange and we had to create a symlink for things to work again.
So for the time being, goodbye Debian and we are using Almalinux 8 again.

 

[LawsHosting]

I had zero issues with Debian since version 4....... All I can say is that it could be due to the migration of backups from RHEL - a bit of an extreme assumption, but who knows ??‍

 

[Richard G]

I had zero issues with Debian since version 4

Yeah it's odd. I know loads of people use Debian without any problems. But it started with that odd client line which was required causing the /tmp/mysql.sock issue. After that was clear all import went fine I could do everything.
And since yesterday evening suddenly database were working but when a website tried to change anything or create a backup, localhost access was denied again.
This time without any reason, the database was looking for the msyqld.sock in /run/mysqld/ while that directory didn't even exist. Out of the blue.
So I had to create it and make a symlink to make things work again.
To me, creating a symlink for something like that is no solution but a workaround. Never ever had that with Centos before.
Since all was working before I can't imagine that it was caused by the Centos imports.

I'm glad everything works again, but for the big servers we choose Alma again due to the better experience and the fact that it's supported till 2029 while Debian 11 is unsure about 07-2024 (could be longer). And too much is not php 8.1 compatible so Debian 12 is not yet an option (with openssl 3.0). It was a quick decision.

So we decided to try Debian again at a later point. I know it's good stuff, it just made me crazy the last 2 days.
Well crazy... I'm already crazy so I should say "more crazy".

 

[LawsHosting]

Yeah it's odd. I know loads of people use Debian without any problems. But it started with that odd client line which was required causing the /tmp/mysql.sock issue. After that was clear all import went fine I could do everything.
And since yesterday evening suddenly database were working but when a website tried to change anything or create a backup, localhost access was denied again.
This time without any reason, the database was looking for the msyqld.sock in /run/mysqld/ while that directory didn't even exist.

Maybe they were using 127.0.0.1 instead of localhost for the host for SQL....... One actually looks for the socket file, not sure which.... That said, it shouldn't be an OS issue.....

 

[Richard G]

Maybe they were using 127.0.0.1 instead of localhost

No it worked fine on the old server. As you could see I even couldn't login with mysql -u root -p without the same error or using -h 127.0.0.1 in the commandline.
I had issues from the start with this fresh install. Shouldn't be an OS issue, maybe a database issue or combination, I don't know. As long as it works now I'm happy.

 

[gate2vn]

It's not because of Debian. I just saw it's happening with a fresh Almalinux / Cloud Linux 9 too. In my case, restoring is working. mysqldump when running backup is not working. A simple command "mysql -uda_admin -p" is not working either with error:

ERROR 2002 (HY000): Can't connect to local server through socket '/tmp/mysql.sock' (2)

So, in my.cnf, I moved

[mysqld]
socket=/var/lib/mysql/mysql.sock

to

[client]
socket=/var/lib/mysql/mysql.sock

After restarting MariaDB, everything seems working as expected.

 

[Richard G]

I just saw it's happening with a fresh Almalinux / Cloud Linux 9

Oh that's odd. I installed a fresh Almal 8 server yesterday and there it was not an issue, no client line like this needed.
Hope you won't run into the same other issue I encountered.

 

[Richard G]

Just use:

Yes for root this would work, but he problem is Directadmin is getting the same error about /run/mysqld/mysqld.sock too.
Now I tried to create a symlink which solved it.... I thought.

So then I created with mkdir -p /var/run/mysqld directory and that worked... I thought.

But after every reboot, any /run/mysqld or /var/run/mysqld directory is gone again.

And I still don't understand as to why Debian 11 is having this issue and keeps looking at a none existing /run/mysqld directory when Directadmin is trying to restore backups.

Is there also a solution which will keep working after a reboot?

 

[Richard G]

This is a spooky VPS, all kind of odd things are happening.

- DNS was crippled, all kinds of records were setup incorrect after a restore of a reseller account. It was impossible (even as admin) to delete some DNS records due to errors about certain A records not existing (which shouldn't have anything to do with it) or other errors. Tried 2 times with restoring, same thing kept happening again.
2 days later when trying again to show screenshot in a ticket I created, all was suddenly working fine on restore and I could delete the records I wanted without problems, as should be, and no wrong records were present as the 2 times before.

- ipv6 was not added to SPF. Tried at least 5 times with account restores. It was not happening. All other ipv6 records were present.
I went back to 1.651 and the ipv6 was added. So I thought bug in 1.652.
Then a day later I went back to 1.652 tried another restore and........ suddenly working correctly, ipv6 was added to SPF record.

- Mariadb issues as described in this topic.
Re-installed mariadb twice before, every time same error finally with the missing /var/run/mysqld/mysqld.sock issue.

Now I thought I found the cause and wanted to doublecheck.
So today I installed mariadb again. The installation creates the /var/run/myslqd directory as should be done.
But after rebooting the VPS, this directory is dissappeared, so I thought this was causing the issue and now it was confirmed that the installation did create the directory.
On my Centos and Almalinux systems, this directory is also present and keeps being present.
But on this VPS, after re reboot, the directory was completely gone.

Rebooted the VPS a couple of times, and in spite of the /var/run/mysqld directory missing, all keeps working fine now.
I'm prepared to becoming more crazy then I already was before.?

I'm into hosting with da for 15 years now, but never ever have seen this kind of odd issues happening on 1 machine and solving itself without me really changing anything.

Maybe it has to do with the virtual system having some issue since this is a VPS and not a dedi which we normally have, but it's very odd.

 

[sysdev]

Is the /var/run/ mounted as tmpfs? This is done very often als the */run/ is only for running processes and at reboot there are none yet.

 

[Richard G]

Is the /var/run/ mounted as tmpfs?

Yes. It looks like it. Done default by Contabo like this.

Filesystem      Size  Used Avail Use% Mounted on
udev             15G     0   15G   0% /dev
tmpfs           3.0G  520K  3.0G   1% /run
/dev/sda3       785G   40G  706G   6% /
tmpfs            15G     0   15G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda2       2.0G  147M  1.7G   8% /boot
tmpfs           3.0G     0  3.0G   0% /run/user/0

However, even with the directory removed (and not re-created when mariadb is started), it now keeps working while before it kept complaining about the missing mysqld.sock there in spite of all attempts I tried before. Not even speaking about the other oddness happening on this thing.

 

[sysdev]

Wait, the /var/run/ directory is located on the root fs. Even though /run/ should be gone (tmpfs), /var/run/ shouldn't be.

 

[Richard G]

The /var/run directory isn't gone. But if I create /var/run/mysqld then that /mysqld will be gone after reboot anyway.
Logically because /var/run is a symlink to /run.

 

[Richard G]

Which is why I don't understand that the /var/run/mysqld directory will be present (or restored) on Centos and Alma, because there the /run is also mounted on tmpfs.

 

[Ohm J]

did you trying config both section "mysqld" and "client" like this

[mysqld]
socket=/usr/local/mysql/data/mysql.sock

[client]
socket=/usr/local/mysql/data/mysql.sock]

because some logics might not same as in Rhel when compare with ubuntu code base.

 

[sysdev]

Which is why I don't understand that the /var/run/mysqld directory will be present (or restored) on Centos and Alma, because there the /run is also mounted on tmpfs.

The /var/run/mysqld is created with mysqld's initialize fase when it starts. Thats because of the FHS standard that some dictates that /run/stuff should be deleted and recreated. (IIRC)

 

[Richard G]

did you trying config both section "mysqld" and "client" like this

Oh yes, I could change what I wanted in the my.cnf but nothing helped. So I put it back as how it was from th ebeginning.

[client]
socket=/usr/local/mysql/data/mysql.sock

Socket was in /var/lib/mysql/mysl.sock and still is, also the log gave:
Version: '10.5.21-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
and the ps faux gave:

mysql        760  0.0  0.4 2195020 136548 ?      Ssl  18:36   0:03 /usr/local/mysql/bin/mysqld --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock

Still the issue came with /run/mysqld/mysqld.sock missing, there is no .sock, there was no .sock and never will be a .sock. Same my.cnf.

As said, I did exactly the same, it's now the same in the my.cnf as before and now it's working without complaining.
It's a spooky VPS, also the DNS and SPF stuff what happened and suddenly worked without issues.

The /var/run/mysqld is created with mysqld's initialize fase when it starts.

Well... not on Debian or at least not on this Debian 11 VPS. I can't check now on the live Centos servers.
I don't see anything about /run/mysqld in the logs of mysql.

And as said... either on Centos or Alma, the /var/run/mysqld directory is present, but empty.

 

[eva2000]

Which is why I don't understand that the /var/run/mysqld directory will be present (or restored) on Centos and Alma, because there the /run is also mounted on tmpfs.

Looks like Debian isn't properly configured to mout the tmpfs while CentOS and AlmaLinux are. On my non-DirectAdmin CentOS or AlmaLinux/Rocky Linux 8/9, there is no /var/run/mysqld as using MariaDB MySQL from official MariaDB YUM repo.

If Debian 11 doesn't have it setup, to recreate the tmpfs mount on server reboot, you can do that yourself by creating /etc/tmpfiles.d/mysqld.conf using this command

echo 'd      /run/mysqld/         0755 root root' > /etc/tmpfiles.d/mysqld.conf

so that /etc/tmpfiles.d/mysqld.conf has contents of

d      /run/mysqld/         0755 root root

Try server reboot and see if /run/mysqld exists.

 

[Richard G]

there is no /var/mysqld

You mean /var/run/mysqld because I don't have a /var/mysqld either.

to recreate the tmpfs mount on server reboot,

The tmpfs mount is recreated, that is not the problem.
But I had some weird issue that mariadb was asking for a mysqld.sock in the /run/mysqld directory, which did not exist (neither the mysqld directory nore the .sock file). And when creating this, after a reboot only this mysqld directory (with symlink) was gone again.

However, as said, after reinstalling Mariadb for the 3rd time, suddenly everything worked as designed, the /run/mysqld directory was gone after reboot again, but since it was never used anyway, I haven't seen mysqld complaining about it anymore since.
So why did it not work like designed the first 2 times I reinstalled mariadb?

The solution you gave I've found too on some forum, I think it was Linuxquestions.org or something like that. But it was not only the /run/mysqld directory, there was also a symlink required from a non existing run/mysqld/mysqld.sock to /var/lib/mysql/mysql.sock (without the d) for it to start working. I didn't know how to add that automatically after a reboot.

Still... as said, after 3rd install, everything is working fine now without mysql complaining about some none existing mysqld.sock (which still does not exists).

It's just weird. Same as the 2 other things I mentioned about not able to delete DNS records and then an spf record addition not being added and then suddenly problem gone.
I'm in this business for 15 years, if it was only this Mariadb I would have thought "yeah wel....<beep> happens" but combined with the other 2 issues which suddenly dissapeared, it's still spooky to me.
And now we got listed twice on Spamhaus, because we used a new fresh domain on the VPS as main domain and hostname. Only for that reason. That was also a first..... spooky vps.