mysql shutting down 04:30 every day

eds132

eds132

Verified User
Joined
Aug 16, 2005
Messages
26
Location
Rapid City, SD
ok, I have a machine that is driving me batty. The mysql daemon is shut down everyday now around 4:30 am...It started around 2 months ago, every once in a while mysql would die out of the blue and DA would show it running (probably because the PID was still in place.) Then it started to become more frequent, the logs show nothing out of the ordinary except that mysql is shutdown normaly (not crashing). At first thought, I presumed I outgrew the pre-compiled binary that is shipped with DA, so I tested and upgraded to a recent stable version which I compiled and installed in place of the old version. The performance increase was drastic. Two days went by, and around 4:30 AM BOOOM, mysqld is dead again. I don't know how to feel about it because I have an identical setup with more load and never have issues. Any help would be greatly appreciated.

Code: 
-bash-2.05b# uname -na
FreeBSD hub. 5.4-RELEASE FreeBSD 5.4-RELEASE #0:

Code: 
Feb 24 04:30:00 hub /usr/sbin/cron[52118]: (root) CMD (/usr/local/directadmin/dataskq)
Feb 24 04:30:00 hub /usr/sbin/cron[52119]: (root) CMD (/usr/libexec/atrun)
Feb 24 04:31:00 hub /usr/sbin/cron[52124]: (root) CMD (/usr/local/directadmin/dataskq)
Feb 24 04:31:00 hub /usr/sbin/cron[52125]: (root) CMD (adjkerntz -a)
Feb 24 04:32:00 hub /usr/sbin/cron[52129]: (root) CMD (/usr/local/directadmin/dataskq)
Feb 24 04:33:00 hub /usr/sbin/cron[52133]: (root) CMD (/usr/local/directadmin/dataskq)
Feb 24 04:33:00 hub /usr/sbin/cron[52134]: (operator) CMD (/usr/libexec/save-entropy)
Feb 24 04:34:00 hub /usr/sbin/cron[52161]: (root) CMD (/usr/local/directadmin/dataskq)
Feb 24 04:35:00 hub /usr/sbin/cron[52243]: (root) CMD (/usr/local/directadmin/dataskq)
Feb 24 04:35:00 hub /usr/sbin/cron[52244]: (root) CMD (/usr/libexec/atrun)
Code: 
Version: '4.1.22'  socket: '/tmp/mysql.sock'  port: 3306  FreeBSD port: mysql-server-4.1.22
070224  4:34:28 [Note] /usr/local/libexec/mysqld: Normal shutdown

070224  4:34:28  InnoDB: Starting shutdown...
070224  4:34:29  InnoDB: Shutdown completed; log sequence number 0 43684
070224  4:34:29 [Note] /usr/local/libexec/mysqld: Shutdown complete

070224 04:34:29  mysqld ended
 
Last edited:
What's going on at 0434? perhaps it's in the middle of backups? perhaps it's in the middle of log rotating?

Is it restarting? If you can figure out why it's stopping then the question is why is it not restarting.

Look for a malicious cronjob.

Jeff
 
Feb 24 04:34:00 hub /usr/sbin/cron[52161]: (root) CMD (/usr/local/directadmin/dataskq)

So, why is /directadmin/dataskq not restarting mysql after backup?


:mad: I don't think that I should have to create scripts to start mysql every morning. Band aid solutions especially from me are effective but ugly. :eek:
 
Last edited:
I'm guessing this is an older install and you're running the "sysbk" System Backup script roughly at that time? We've since updated the script, but we have a new mysql.bk file for it that doesn't restart mysqld into a proprietary mode:
Code: 
wget -O /usr/local/sysbk/mod/mysql.bk http://www.directadmin.com/mysqk.bk
It's a sort of brute force "lock tables" that it's author implemented, but it's not always successul in undoing this lock, hence we've modded the mysql.bk (in the default package as well) to not do this lock.

John
 
ooooo that would be great! if the file existed there! :p :D :D :D
Code: 
wget -O /usr/local/sysbk/mod/mysql.bk http://www.directadmin.com/mysql.bk

fixed the typo ;) Thanks will try it out.
 
If any one wants to know the diffs....

Code: 
-bash-2.05b# diff mysql.bk mysql.bk.old
6a7,18
> $MYSQL_INIT stop >> /dev/null 2>&1
> if [ -f "$MYSQL_PID" ]; then
>         $MYSQL_INIT stop >> /dev/null 2>&1
>         sleep 4
>         if [ -f "$MYSQL_PID" ]; then
>                 echo_fail
>                 echo "MySQL could not be shutdown, aborting..."
>                 echo "MySQL could not be shutdown, aborting..." >> $QLOG
>                 postbk
>                 exit 1
>         fi
> fi
9c21
< if [ -f "/usr/local/sim/sim" ] && [ ! "$SIM_CRON" == "" ]; then
---
> if [ -f "/usr/local/sim/sim" ] && [ ! "$SIM_CRON" = "" ]; then
26,27c38,39
<
< sleep 1
---
> $MYSQL_INIT start >> /dev/null 2>&1
> sleep 4
42c54
< if [ "$SIM_OFF" == "1" ]; then
---
> if [ "$SIM_OFF" = "1" ]; then
49c61
< if [ -d "$MYSQL_PATH/$i" ] && [ "$DIR_PREFIX" == "" ]; then
---
> if [ -d "$MYSQL_PATH/$i" ] && [ "$DIR_PREFIX" = "" ]; then
52c64
<         $NICE -n $PRI $MYSQL_DUMP -l -u$MYSQL_ROOTUN -p$MYSQL_ROOTPW $i > $BACKUP_PATH/$DATE/$i.sql
---
>       $NICE -n $PRI $MYSQL_DUMP -l -u$MYSQL_ROOTUN -p$MYSQL_ROOTPW $i > $BACKUP_PATH/$DATE/$i.sql
55c67
< elif [ -d "$MYSQL_PATH/$i" ] && [ ! "$DIR_PREFIX" == "" ]; then
---
> elif [ -d "$MYSQL_PATH/$i" ] && [ ! "$DIR_PREFIX" = "" ]; then
 
heh. Would the /usr/local/directadmin/data/admin/services.status file have something to do with mysql not starting up after backups? I just found an old old old(12-10-2004) post about not having to shutdown mysqld before running backups because of "da is now using sysbk". lol I guess I do have a old installation. Any ways, I updated the mysql.bk file and set the mysqld option to ON in the services.status file. I guess Ill kill my Script and give the double barrel shotgun fix a trigger squeeze and see if that makes the evil go away.

Thanks DA support and jlasman....
 
The way that sysbk used to do it (the old way) was to restart mysql in a mode where nobody but sysbk could access it (or completely stopped... I can't quite rememeber.. but anyway..). So mysql was actually running according to DA, because the mysqld binaries were still chugging along.. the only catch was nobody can access them. We figured this isn't the best thing to have happen on a live server, so we simply removed that option from the mysql.bk file. Mysql isn't restarted/touched by sysbk. SQL files are dumped, and database files are copied, but people can still access mysql during that time.

And yes, you've found the typo, you win prize, a new mysql.bk file ;)

So the services.status setting doesn't really matter anymore, because mysql is never restarted or stopped, or anything like that, so it doesn't apply.

John
 
Ok, It is working yay. But what still has me wondering about is, why It would work for the better part of of the week and die (not restart) for one or two days. (seamingly random days at that.)
 
That, I couldn't say for sure. It might be order that the processes show in the ps output when it's being killed.. I couldn't say for sure.
 
You must log in or register to reply here.
Back
Top