MySQL is not shutting down (can't be restarted)

wattie

wattie

Verified User
Joined
May 31, 2008
Messages
1,206
Location
Bulgaria
Check out what's happening:

Code: 
2015-03-16 11:50:06 86503 [Note] /usr/local/mysql/bin/mysqld: Normal shutdown
2015-03-16 11:50:28 86503 [Note] Giving 41 client threads a chance to die gracefully
2015-03-16 11:50:28 86503 [Note] Event Scheduler: Purging the queue. 0 events
2015-03-16 11:50:28 86503 [Note] Shutting down slave threads
150316 11:50:28 mysqld_safe A mysqld process already exists
2015-03-16 11:50:31 86503 [Note] Forcefully disconnecting 0 remaining clients
2015-03-16 11:50:31 86503 [Note] Binlog end
150316 11:50:46 mysqld_safe A mysqld process already exists

What happens is that mysqld process is running; however it no longer accepts new connections to it. I try to restart it and I have to kill the process manually in order to start mysqld again.

It wouldn't be a (big) problem for me if it is not restarting gracefully, but the bigger problem is that it hangs (stops accepting new connections without obvious reason) for some reason like once per two days or something like that. Actually there's no pattern - it happens without obvious reason. The rows prior to the log above are the following:

Code: 
2015-03-14 19:54:13 86503 [ERROR] /usr/local/mysql/bin/mysqld: Table './onlinebul_site/jos_session' is marked as crashed and should be repaired
2015-03-14 19:54:13 86503 [ERROR] /usr/local/mysql/bin/mysqld: Table './onlinebul_site/jos_session' is marked as crashed and should be repaired
2015-03-14 19:54:20 86503 [Note] Found 5848 of 5867 rows when repairing './onlinebul_site/jos_session'

As you can see it's from two days ago so it shouldn't be related to it.

Any thoughts why MySQL won't stop gracefully?
 
Last edited:
By the way (if that's important) MySQL won't restart even if it is not hanging.

/usr/local/etc/rc.d/mysqld restart

always shows "Shutting down: OK" followed by "Starting: Failed"... and I have to kill the process in order to start it up again.
 
Check and repair all your tables:
Code: 
mysqlcheck --defaults-extra-file=/usr/local/directadmin/conf/my.cnf --auto-repair --optimize --all-databases

Make sure your have enough RAM and disk space left.
 
I was doing that a lot of times with:

Code: 
/usr/local/mysql/bin/mysqlcheck -uda_admin -p`grep "^passwd=" /usr/local/directadmin/conf/mysql.conf | cut -d= -f2` --auto-repair=TRUE --optimize --all-databases=TRUE

But it is not resolving the issue. Actually the tables are getting broken because of the non-graceful shutdown (killing the process).
 
during the hang I get that in httpd error log:

Code: 
[Mon Mar 23 16:00:34.906417 2015] [mpm_prefork:error] [pid 98710] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Mon Mar 23 16:01:36.743814 2015] [core:warn] [pid 98710] AH00045: child process 39690 still did not exit, sending a SIGTERM
[Mon Mar 23 16:01:36.862577 2015] [core:warn] [pid 98710] AH00045: child process 42830 still did not exit, sending a SIGTERM
[Mon Mar 23 16:01:36.926138 2015] [core:warn] [pid 98710] AH00045: child process 47546 still did not exit, sending a SIGTERM
[Mon Mar 23 16:01:36.954181 2015] [core:warn] [pid 98710] AH00045: child process 40699 still did not exit, sending a SIGTERM
[Mon Mar 23 16:01:36.970071 2015] [core:warn] [pid 98710] AH00045: child process 40862 still did not exit, sending a SIGTERM
[Mon Mar 23 16:01:37.006040 2015] [core:warn] [pid 98710] AH00045: child process 41623 still did not exit, sending a SIGTERM
[Mon Mar 23 16:01:37.020566 2015] [core:warn] [pid 98710] AH00045: child process 44213 still did not exit, sending a SIGTERM
[Mon Mar 23 16:01:37.237320 2015] [core:warn] [pid 98710] AH00045: child process 43173 still did not exit, sending a SIGTERM
[Mon Mar 23 16:01:37.273010 2015] [core:warn] [pid 98710] AH00045: child process 42856 still did not exit, sending a SIGTERM
[Mon Mar 23 16:01:37.507045 2015] [core:warn] [pid 98710] AH00045: child process 42857 still did not exit, sending a SIGTERM
[Mon Mar 23 16:01:37.558906 2015] [core:warn] [pid 98710] AH00045: child process 44214 still did not exit, sending a SIGTERM
[Mon Mar 23 16:01:37.568166 2015] [core:warn] [pid 98710] AH00045: child process 42884 still did not exit, sending a SIGTERM
[Mon Mar 23 16:01:37.584060 2015] [core:warn] [pid 98710] AH00045: child process 38894 still did not exit, sending a SIGTERM
[Mon Mar 23 16:01:37.612149 2015] [core:warn] [pid 98710] AH00045: child process 44535 still did not exit, sending a SIGTERM
...


Nothing back in the log file in the last minutes. My httpd-mpm.conf has MaxRequestWorkers 650 defined.
 
Try running mysqltuner.pl to see what recommended my.cnf values it suggests:

Code: 
cd ~
wget https://raw.githubusercontent.com/major/MySQLTuner-perl/master/mysqltuner.pl
chmod 700 mysqltuner.pl
./mysqltuner.pl
 
I was using tuning-primer for that... I don't think that's the issue.

I had few slow queries logged (~60 secs). I will get rid of them and see if things will get better.
 
I had the same issue on one of my servers.

That was because there was a website that was continuously under attack.

When i started to cache that website (And change it folder structure of Wordpress)

It didn't happen anymore.

Maybe if you can track down which site it is.

And you can check
/var/log/messages

around the time you get the messages. In that file are alot of lines that will help track you down the issue
 
After optimizing the slow SQL queries the load on the server felt down from ~3.00 to ~2.00. Today it didn't crash. We'll see..

Regarding the attacks - it's constantly under attach. My bruteforce list is full of blocked IP's...
 
Do you know which website is being attacked?

If you look at the IP/server-status page you can almost certainly analyse which site it is.

If it's for example Wordpress start caching it with W3 and change it structure so it isn't wp-content/plugins anymore but plugins/ for example. Same with the rest of the folders. And especially change wp-admin to something completely different (example: thisadminpanelisnotvisibleanymore)

This would reduce alot of WP specific attacks.
 
You must log in or register to reply here.
Back
Top