One of our server encountered the same after updating CentOS from 7.6 to 7.7, and MariaDB from 10.1.37 to 10.1.41
I checked /etc/systemd/system/mysqld.service , it already contains "LimitNOFILE=655350" under [Service] section out of the box.
However, sometimes the MariaDB server crashed (especially high I/O load, e.g. backup period)
We are running database on SSD partition already. The I/O wait shall be small.
Code:
Oct 24 22:36:10 xxxx mysqld: InnoDB: Warning: a long semaphore wait:
Oct 24 22:36:10 xxxx mysqld: --Thread 139658478462720 has waited at dict0dict.cc line 984 for 241.00 seconds the semaphore:
Oct 24 22:36:10 xxxx mysqld: Mutex at 0x7f05b0c112e8 '&dict_sys->mutex', lock var 1
Oct 24 22:36:10 xxxx mysqld: Last time reserved by thread 139658495248128 in file not yet reserved line 0, waiters flag 1
Oct 24 22:36:10 xxxx mysqld: InnoDB: Warning: a long semaphore wait:
Oct 24 22:36:10 xxxx mysqld: --Thread 139662426380032 has waited at dict0dict.cc line 1138 for 241.00 seconds the semaphore:
Oct 24 22:36:10 xxxx mysqld: Mutex at 0x7f05b0c112e8 '&dict_sys->mutex', lock var 1
Oct 24 22:36:10 xxxx mysqld: Last time reserved by thread 139658495248128 in file not yet reserved line 0, waiters flag 1
Then, the MariaDB database crashed and caused a few MyISAM tables corrupted (need mysqlcheck --autorepair)
----
When the server running normally today (not crashed one <-- I cannot get the crashed one, it is auto-restarted)
I tried to use lsof, and found -
Code:
1101473 total opened files, whereas
868028 is httpd
209664 is mysqld
remaining are others - dovecot, ...
---
In the past (2 years ago), we already set the following inside /etc/sysctl.d/xxx.conf , and reboot, to increase semaphore limit.
Code:
kernel.sem=250 256000 32 1024
----
ulimit:
Code:
[root@xxxx ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127697
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 127697
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[root@xxxx ~]#
[root@xxxx ~]#
[root@xxxx ~]# ulimit -Hn
4096
[root@xxxx ~]#
[root@xxxx ~]#
[root@xxxx ~]# ulimit -Sn
1024
[root@xxxx ~]#
[root@xxxx ~]#
[root@xxxx ~]#
----
Also, for unknown reason, after MariaDB update to 10.1.41 , for any mysqld systemctl operation (start/stop/restart/status), it showed -
Code:
Warning: mysqld.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Even I reboot the server, but the same.
These 2 problems occurred on 1 server. They do not occur in other servers (with the same build, same OS, same MariaDB version)
Anything I can do? Please kindly help/advise.
Thank you very much.