Apahce is freaking out

[patrik]

Apache is freaking out

The symptom is very much like the thread "apache high load problems" in this forum but I would like to start a new thread.
It happends randomly as it seems, often twice a day. The first thing that happends is that I get an SMS alarm saying the mail server is down or anything like that. So I try to login with SSH and it goes reeeeaaaally slow. Finally I'm in and I'm able to run top (slow as hell).
It shows:
1441 processes:2 running, 1434 sleeping, 5 zombie

Swap: 19G Total, 2104M Used, 17G Free, 10% Inuse, 224K In, 124K Out

1441 processes(!), everytime this occurs the amount of processes is above 1000. Check Swap usage, 2104MB! There's like no Free RAM left and ~200MB inactive. The tool for checking I/O (gstat) shows a 100% load on the disk. Probably it has much to do with the fact that it has to write/read swap all the time.
Okay, I have checked apache error_log and the interesting part is this:

[Fri Dec 15 13:15:53 2006] [error] mod_ssl: SSL handshake failed: HTTP spoken on HTTPS port; trying to send HT
ML error page (OpenSSL library error follows)
[Fri Dec 15 13:15:53 2006] [error] OpenSSL: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request [H
int: speaking HTTP to HTTPS port!?]
^GOut of memory (Needed 8164 bytes)
^GOut of memory (Needed 8164 bytes)
^GOut of memory (Needed 8164 bytes)
httpd in free(): error: page is already free
[Fri Dec 15 14:08:38 2006] [notice] child pid 78790 exit signal Abort trap (6)
httpd in free(): error: page is already free
[Fri Dec 15 14:11:36 2006] [notice] child pid 8307 exit signal Abort trap (6)
httpd in free(): error: page is already free
httpd in free(): error: page is already free
httpd in free(): error: page is already free
httpd in free(): error: page is already free
[Fri Dec 15 14:35:47 2006] [notice] child pid 18018 exit signal Abort trap (6)
[Fri Dec 15 14:35:47 2006] [notice] child pid 15772 exit signal Abort trap (6)
[Fri Dec 15 14:35:47 2006] [notice] child pid 15479 exit signal Abort trap (6)
[Fri Dec 15 14:35:47 2006] [notice] child pid 15433 exit signal Abort trap (6)
[Fri Dec 15 14:50:53 2006] [error] [client 213.246.61.91] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind
[Fri Dec 15 14:50:54 2006] [error] [client 213.246.61.91] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind
[Fri Dec 15 14:50:54 2006] [error] [client 213.246.61.91] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind
[Fri Dec 15 14:50:54 2006] [error] [client 213.246.61.91] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind
[Fri Dec 15 14:52:56 2006] [warn] child process 75976 still did not exit, sending a SIGTERM
[Fri Dec 15 14:52:56 2006] [warn] child process 75978 still did not exit, sending a SIGTERM
[Fri Dec 15 14:52:56 2006] [warn] child process 75979 still did not exit, sending a SIGTERM
[Fri Dec 15 14:52:56 2006] [warn] child process 76653 still did not exit, sending a SIGTERM
[Fri Dec 15 14:52:56 2006] [warn] child process 76655 still did not exit, sending a SIGTERM
.... and it goes on like this and then there's a row like this:
httpd in free(): error: recursive call
... and then it continues with SIGTERM rows..

The CPU load is high but not terribly high. The only module that is not added by default to Apache (by DA) that we have loaded is FastCGI but I have tried to disable this module but no luck.

I have googled a lot and read about DoS attacks and such, could it be the problem?
Or is it MySQL as the neighbour forum thread guesses?

Hmm, actually, when I was writing this the damn thing occured again, server-status on apache showed very many 'W' (where apache is sending replies) so I tried a new thing, restart MySQL and after a while I reloaded server-status and almost every 'W' was gone.
I have upgraded apache to 1.3.37 and MySQL is running 4.1.14 and it's PHP 4.4.2.

 

[patrik]

Number of processes are growing and apache server status says:

WWKWKW_WWWWK_WWW__WWWWWWWWWWWW_WWWW_W_WWWWWWW_WWWWWWWWWWWK__KWWW
_WWWKWWWWWWWWWWWKWWWKW_WW_WWKWKKW__WKWWWWWWWWWW_................

After a MySQLd restart:

W_WK___K______W___WK____K_K__W____W__KK___KK__K_K_______K__KK___
__WK__K___WW_____KW___KW___K_______W___________KW_K_K_KW__K___R_
_._.__._____K.._K...............................................

Strange?? Something's wrong with MySQL as it seems.
my.cnf:

[mysqld]
port = 3306
socket = /tmp/mysql.sock
max_connections = 600
max_connect_errors = 100
skip-locking
key_buffer = 16M
max_allowed_packet = 1M
table_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M

log-bin
server-id = 1

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

 

[xemaps]

lol 5 zombies, are you hacked ?

Don't trust your user, i hope you have not lot of security holes and a firewall. Hope you didn't give ssh.

I suggest you to contact a specialist to look further.

 

[patrik]

This is a web hosting environment and yes we have ssh enabled for our customers. 5 zombies is not normal, it happend to be the the time I did the copy. Anyway, we have now moved some customers in order to decrease load on the machine. We have also upgraded MySQL and moved the MySQL data to another harddrive. It seems to run a lot smoother now actually. We will see next week if the issues are gone.