DA Showing lots of (fake ??) MySql processes / pids

[Duboux]

When I go to the Show Process List in the control panel, I see this:

mysqld	mysqld (pid 3649 3650 3651 4065 4066 4067 4068 4070 4071 4158 4159 4872 4873 4874 4875 8868 8869 8870 8871 8872 8877 8878 8879 8880 8881 8882 8884 8886 8887 8888 8889 8892 8895 8896 8897 8898 8899 8900 8901 8902 8903 8904 8905 8906 8914 8916 8923 8924 8925 8934 8935 8936 8937 8938 8939 8940 8941 8942 8950 8951 8952 8953 8954 8955 8975 8982 8989 9010 9045 9074 9076 9085 9098 9105 9125 9141 9170 9172 9192 9195 9200 9205 9206 9208 9209 9210 9211 9214 9215 9216 9217 9220 9221 9222 9226 9227 9229 9230 9232 9235 9236 9237 9238 9258 9259 9336 9337 9338 9339 9340 9341 9342 9343 9344 9345 9346 9347 9348 9349 9350 9351 9352 9353 9354 9355 9356 9357 9358 9359 9360 9361 9362 9363 9364 9365 9366 9367 9368 9371 9372 9373 9374 9375 9376 9377 9378 9379 9380 9381 9382 9383 9384 9385 9386 9387 9390 9391 9392 9393 9394 9395 9396 9397 9398 9399 9400 9401 9402 9403 9404 9405 9406 9411 9412 9413 9414 9415 9416 9417 9418 9419 9420 9422 9423 9424 9425 9426 9427 12526 12527 12528 12529 12530 12531 14956 14971 14992 15003 21397 21398 )	Start	Stop

And most of those pids have been standing there for 12hrs now
The amount of http-pids is 9% compared to the amount of MySql pids.

When I ssh #kill 3649
and refresh the process list page, it still exists.
When I MySql

mysql> kill 3649;
ERROR 1094 (HY000): Unknown thread id: 3649

It doesn't exist.

My SHOW QUERIES command shows only like max 5 queries, that quickly disappear again.

And the average serverload is like 0.45.


What's wrong ?

 

[tillo]

What "ps auxww |grep 3649" says, and "ls -l /proc/3649" ?

 

[Duboux]

nothing.

[root@da ~][B]# ps auxww |grep 3649" says, and "ls -l /proc/3649[/B]
[root@da ~]#

and:

[root@da ~][B]# ps auxww |grep 3649[/B]
mysql     3649  0.0 12.3 187456 127452 ?       S    Feb26   0:25 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/[COLOR="DarkOrchid"]<<host>>[/COLOR].pid --skip-external-locking
root     12703  0.0  0.0   3896   696 pts/0    S+   12:26   0:00 grep 3649
[root@da ~][B]# ps auxww |ls -l /proc/3649[/B]
total 0
dr-xr-xr-x 2 mysql mysql 0 Mar 31 12:26 attr
-r-------- 1 root  root  0 Mar 31 12:26 auxv
--w------- 1 root  root  0 Mar 31 12:26 clear_refs
-r--r--r-- 1 root  root  0 Mar 31 12:26 cmdline
-r--r--r-- 1 root  root  0 Mar 31 12:26 cpuset
lrwxrwxrwx 1 root  root  0 Mar 31 12:26 cwd -> /var/lib/mysql
-r-------- 1 root  root  0 Mar 31 12:26 environ
lrwxrwxrwx 1 root  root  0 Mar 31 12:26 exe -> /usr/sbin/mysqld
dr-x------ 2 root  root  0 Mar 31 12:26 fd
dr-x------ 2 root  root  0 Mar 31 12:26 fdinfo
-r--r--r-- 1 root  root  0 Mar 31 12:26 io
-rw-r--r-- 1 root  root  0 Mar 31 12:26 loginuid
-r-------- 1 root  root  0 Mar 31 12:26 maps
-rw------- 1 root  root  0 Mar 31 12:26 mem
-r--r--r-- 1 root  root  0 Mar 31 12:26 mounts
-r-------- 1 root  root  0 Mar 31 12:26 mountstats
-rw-r--r-- 1 root  root  0 Mar 31 12:26 oom_adj
-r--r--r-- 1 root  root  0 Mar 31 12:26 oom_score
lrwxrwxrwx 1 root  root  0 Mar 31 12:26 root -> /
-r--r--r-- 1 root  root  0 Mar 31 12:26 schedstat
-r-------- 1 root  root  0 Mar 31 12:26 smaps
-r--r--r-- 1 root  root  0 Mar 31 12:26 stat
-r--r--r-- 1 root  root  0 Mar 31 12:26 statm
-r--r--r-- 1 root  root  0 Mar 31 10:22 status
dr-xr-xr-x 3 mysql mysql 0 Mar 31 12:26 task
-r--r--r-- 1 root  root  0 Mar 31 12:26 wchan

 

[tillo]

Ok, 3649 is the main mysqld process then. Now try with the others.
"ps auxww |grep <PID>"
"ls -l /proc/<PID>"

 

[Duboux]

[root@da ~][B]# ps auxww |grep 3650[/B]
mysql     3650  0.0 12.3 187448 127544 ?       S    Feb26   0:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/[COLOR="DarkOrchid"]<<host>>[/COLOR].pid --skip-external-locking
root     15727  0.0  0.0   3900   700 pts/0    S+   14:24   0:00 grep 3650
[root@da ~][B]# ls -l /proc/3650[/B]
total 0

[root@da ~][B]# ps auxww |grep 3651[/B]
mysql     3651  0.0 12.3 187448 127496 ?       S    Feb26   1:03 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/[COLOR="DarkOrchid"]<<host>>[/COLOR].pid --skip-external-locking
root     15781  0.0  0.0   3900   696 pts/0    S+   14:26   0:00 grep 3651
[root@da ~][B]# ls -l /proc/3651[/B]
total 0

[root@da ~][B]# ps auxww |grep 4065[/B]
mysql     4065  0.0 12.3 187448 127496 ?       S    Feb26  27:28 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/[COLOR="DarkOrchid"]<<host>>[/COLOR].pid --skip-external-locking
root     15803  0.0  0.0   3896   696 pts/0    S+   14:27   0:00 grep 4065
[root@da ~][B]# ls -l /proc/4065[/B]
total 0

[root@da ~][B]# ps auxww |grep 8868[/B]
mysql     8868  0.0 12.3 187448 127512 ?       S    Feb28  16:13 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/[COLOR="DarkOrchid"]<<host>>[/COLOR].pid --skip-external-locking
root     15830  0.0  0.0   3896   696 pts/0    S+   14:28   0:00 grep 8868
[root@da ~]# ls -l /proc/8868
total 0

[root@da ~][B]# ps auxww |grep 12526[/B]
mysql    12526  0.0 12.3 187448 127492 ?       S    Mar23   4:48 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/[COLOR="DarkOrchid"]<<host>>[/COLOR].pid --skip-external-locking
root     15847  0.0  0.0   3900   700 pts/0    S+   14:28   0:00 grep 12526
[root@da ~][B]# ls -l /proc/12526[/B]
total 0

 

[tillo]

That's very interesting.

This is my setup:

$ pstree -up 21959
mysqld_safe(21959)───mysqld(22009,mysql)─┬─{mysqld}(22012)
                                         ├─{mysqld}(22013)
                                         ├─{mysqld}(22014)
                                         ├─{mysqld}(22015)
                                         ├─{mysqld}(22017)
                                         ├─{mysqld}(22018)
[... and so on]

As you can see, there is a main process (22009) and many threads (22012, 22013...).
I thought that was what you see on your system, but I guess not.

Why? Because:
1) threads can't be seen with "ps auxww"... but you see them
2) threads PID are automatically linked to the parent main process in /proc... but yours have an empty directory

Can I ask you which kernel is your system running on ("uname -a")?
You may run on a very old (or very new) kernel, or it may also be some kind of LKM rootkit (malware) modifying its behaviour.

EDIT: forgot to tell you... try "SHOW PROCESSLIST" instead of "SHOW QUERIES" (which, by the way, I don't have in MySQL 5.0... which version are you using?)

 

[Duboux]

[root@da ~][B]# pstree -up 3649[/B]
mysqld(3649,mysql)âââmysqld(3650)ââ¬âmysqld(3651)
                                  ââmysqld(4065)
                                  ââmysqld(4066)
                                  ââmysqld(4067)
                                  ââmysqld(4068)
                                  ââmysqld(4070)
                                  ââmysqld(4071)
                                  ââmysqld(4158)
                                  ââmysqld(4159)
                                  ââmysqld(4872)
                                  ââmysqld(4873)
                                  ââmysqld(4874)
                                  ââmysqld(4875)
                                  ââmysqld(8868)
                                  ââmysqld(8869)
                                  ââmysqld(8870)
                                  ââmysqld(8871)
                                  ââmysqld(8872)
                                  ââmysqld(8877)
                                  ââmysqld(8878)
                                  ââmysqld(8879)
                                  ââmysqld(8880)
                                  ââmysqld(8881)
                                  ââmysqld(8882)
                                  ââmysqld(8884)
                                  ââmysqld(8886)
                                  ââmysqld(8887)
                                  ââmysqld(8888)
                                  ââmysqld(8889)
                                  ââmysqld(8892)
                                  ââmysqld(8895)
                                  ââmysqld(8896)
                                  ââmysqld(8897)
                                  ââmysqld(8898)
                                  ââmysqld(8899)
                                  ââmysqld(8900)
                                  ââmysqld(8901)
                                  ââmysqld(8902)
                                  ââmysqld(8903)
                                  ââmysqld(8904)
                                  ââmysqld(8905)
                                  ââmysqld(8906)
                                  ââmysqld(8914)
                                  ââmysqld(8916)
                                  ââmysqld(8923)
                                  ââmysqld(8924)
                                  ââmysqld(8925)
                                  ââmysqld(8934)
                                  ââmysqld(8935)
                                  ââmysqld(8936)
                                  ââmysqld(8937)
                                  ââmysqld(8938)
                                  ââmysqld(8939)
                                  ââmysqld(8940)
                                  ââmysqld(8941)
                                  ââmysqld(8942)
                                  ââmysqld(8950)
                                  ââmysqld(8951)
                                  ââmysqld(8952)
                                  ââmysqld(8953)
                                  ââmysqld(8954)
                                  ââmysqld(8955)
                                  ââmysqld(8975)
                                  ââmysqld(8982)
                                  ââmysqld(8989)
                                  ââmysqld(9010)
                                  ââmysqld(9045)
                                  ââmysqld(9074)
                                  ââmysqld(9076)
                                  ââmysqld(9085)
                                  ââmysqld(9098)
                                  ââmysqld(9105)
                                  ââmysqld(9125)
                                  ââmysqld(9141)
                                  ââmysqld(9170)
                                  ââmysqld(9172)
                                  ââmysqld(9192)
                                  ââmysqld(9195)
                                  ââmysqld(9200)
                                  ââmysqld(9205)
                                  ââmysqld(9206)
                                  ââmysqld(9208)
                                  ââmysqld(9209)
                                  ââmysqld(9210)
                                  ââmysqld(9211)
                                  ââmysqld(9214)
                                  ââmysqld(9215)
                                  ââmysqld(9216)
                                  ââmysqld(9217)
                                  ââmysqld(9220)
                                  ââmysqld(9221)
                                  ââmysqld(9222)
                                  ââmysqld(9226)
                                  ââmysqld(9227)
                                  ââmysqld(9229)
                                  ââmysqld(9230)
                                  ââmysqld(9232)
                                  ââmysqld(9235)
                                  ââmysqld(9236)
                                  ââmysqld(9237)
                                  ââmysqld(9238)
                                  ââmysqld(9258)
                                  ââmysqld(9259)
                                  ââmysqld(9336)
                                  ââmysqld(9337)
                                  ââmysqld(9338)
                                  ââmysqld(9339)
                                  ââmysqld(9340)
                                  ââmysqld(9341)
                                  ââmysqld(9342)
                                  ââmysqld(9343)
                                  ââmysqld(9344)
                                  ââmysqld(9345)
                                  ââmysqld(9346)
                                  ââmysqld(9347)
                                  ââmysqld(9348)
                                  ââmysqld(9349)
                                  ââmysqld(9350)
                                  ââmysqld(9351)
                                  ââmysqld(9352)
                                  ââmysqld(9353)
                                  ââmysqld(9354)
                                  ââmysqld(9355)
                                  ââmysqld(9356)
                                  ââmysqld(9357)
                                  ââmysqld(9358)
                                  ââmysqld(9359)
                                  ââmysqld(9360)
                                  ââmysqld(9361)
                                  ââmysqld(9362)
                                  ââmysqld(9363)
                                  ââmysqld(9364)
                                  ââmysqld(9365)
                                  ââmysqld(9366)
                                  ââmysqld(9367)
                                  ââmysqld(9368)
                                  ââmysqld(9371)
                                  ââmysqld(9372)
                                  ââmysqld(9373)
                                  ââmysqld(9374)
                                  ââmysqld(9375)
                                  ââmysqld(9376)
                                  ââmysqld(9377)
                                  ââmysqld(9378)
                                  ââmysqld(9379)
                                  ââmysqld(9380)
                                  ââmysqld(9381)
                                  ââmysqld(9382)
                                  ââmysqld(9383)
                                  ââmysqld(9384)
                                  ââmysqld(9385)
                                  ââmysqld(9386)
                                  ââmysqld(9387)
                                  ââmysqld(9390)
                                  ââmysqld(9391)
                                  ââmysqld(9392)
                                  ââmysqld(9393)
                                  ââmysqld(9394)
                                  ââmysqld(9395)
                                  ââmysqld(9396)
                                  ââmysqld(9397)
                                  ââmysqld(9398)
                                  ââmysqld(9399)
                                  ââmysqld(9400)
                                  ââmysqld(9401)
                                  ââmysqld(9402)
                                  ââmysqld(9403)
                                  ââmysqld(9404)
                                  ââmysqld(9405)
                                  ââmysqld(9406)
                                  ââmysqld(9411)
                                  ââmysqld(9412)
                                  ââmysqld(9413)
                                  ââmysqld(9414)
                                  ââmysqld(9415)
                                  ââmysqld(9416)
                                  ââmysqld(9417)
                                  ââmysqld(9418)
                                  ââmysqld(9419)
                                  ââmysqld(9420)
                                  ââmysqld(9422)
                                  ââmysqld(9423)
                                  ââmysqld(9424)
                                  ââmysqld(9425)
                                  ââmysqld(9426)
                                  ââmysqld(9427)
                                  ââmysqld(12526)
                                  ââmysqld(12527)
                                  ââmysqld(12528)
                                  ââmysqld(12529)
                                  ââmysqld(12530)
                                  ââmysqld(12531)
                                  ââmysqld(14956)
                                  ââmysqld(14971)
                                  ââmysqld(14992)
                                  ââmysqld(15003)
                                  ââmysqld(21397)
                                  ââmysqld(21398)
[root@da ~]#

[root@da ~]# uname -a
Linux <<host>> 2.6.22.2-42.fc6 #1 SMP Wed Aug 15 12:34:26 EDT 2007 i686 i686 i386 GNU/Linux

And Sorry, with:

My SHOW QUERIES command shows only like max 5 queries, that quickly disappear again.

I meant SHOW PROCESSLIST, which I can see in phpMyAdmin as da_admin user.
I still have MySql 4.1.22.
And will upgrade from Apache 1.3.41, PHP 5.2.8 and MySql 4.1 to the latest, once PHP 5.3 is out.

Thanks for the help

 

[tillo]

Well, this is going beyond my knowledge.
My only guesses are that 1) either you FC-patched kernel shows/treats threads differently or 2) MySQL 4.1 doesn't use "normal" 2.6 threading.
Since those weird non-threads have the main mysqld process as parent process, I guess there is nothing to be worried about... it's just annoying to see them in "ps auxww" (and in DA). It's perfectly normal to have many MySQL threads running, even with few concurrent queries.
If you have a very low-load server, try setting thread_cache_size to somehing little (like 4 or 2), it may lower the threads number.

 

[Duboux]

Thanks for your help.

The server has got quite a nice load, also MySql ain't sitting still.

I don't see the restart-link on the show-processes list page in DA.
If I can't fix this before midnight, I'll restart MySql, although a little voice inside me tells me not to do it

Anyone else having an idea ?

 

[DirectAdmin Support]

Hello,

Sometimes mysqld may just need a restart. I'm not sure about the issue either, but I would try:

/etc/init.d/mysqld stop
/etc/init.d/mysqld start

Give it a few seconds to stop before starting it up. Sometimes, if the pid file is out of sync with the true master process, you'd have to run "killall -9 mysqld" just to clear out all mysqld processes, then start it up normally. Note it's best to stop/start mysqld at a non peak hour, as even short periods of 10 seconds, people notice (forums, etc).

Regarding the number of processes, one script call can in theory have many calls to mysqld, so the number of mysql processes can be higher than apache. Also, if any script (usually non-php) doesn't call mysql_close() then the script ends up holding the mysql connection open, which could explain this as well. Doing the above forced restart would help clear that up.

John

 

[Duboux]

Thanks John,

I did it, and the services list page now shows this:

httpd httpd (pid 370 372 373 374 375 376 377 378 379 382 383 384 385 398 399 406 ) Start Stop Restart Reload
mysqld mysqld (pid 560 564 565 566 592 626 738 764 896 976 984 1023 1024 1025 1026 1027 1028 1029 ) Start Stop

I'll have a look in 12hrs to see if the same pids are still there.

How could I find out which query / script relates to the pid ?

 

[DirectAdmin Support]

As tillo mentioned, these sql queries will help you out. Login to /phpMyAdmin as da_admin (password in /usr/local/directadmin/conf/mysql.conf) and run the sql query:

SHOW PROCESSLIST

John

 

[Duboux]

Yeh, I did that before, and it ain't showing those pids..

When I check it now, it only shows me:

stop proces 	28537 	da_admin 	localhost 	mysql 	Query 	0 	--- 	SHOW PROCESSLIST

While the Services page in DA shows me:

mysqld (pid [COLOR="Blue"]560 564 565 566 592 626 738 764 896 976 984 1023 1024 1025 1026 1027 1028 1029[/COLOR] 12658 12659 12660 12661 12662 12663 13141 13144 13150 15305 )

The ones marked in blue are the same ones in my previous post, so this ain't going away

 

[Duboux]

Okay, 3 days later, still the same pids...

mysqld (pid [COLOR="Blue"]560 564 565 566 592 626 738 764 896 976 984 1023 1024 1025 1026 1027 
1028 1029 12658 12659 12660 12661 12662 12663 13141 13144 13150[/COLOR] 30635 )

The ones marked in blue are the same ones in my previous post, so this ain't going away still

help ?

 

[tillo]

You are investigating the wrong thing. Like I said before, it's not because of those pids that your MySQL system is running slow: it's prefectly normal to have many of them.

You should start tuning its settings, using Tuning Primer, mk-audit, mysqlreport and many others. If you need professional help, just contact me.