Server give 300+ load once a day

[Cybex]

Hi,

I am running several DirectAdmin webservers and there's one server that is causing problems. The problem is that once a day (sometimes twice a day) the load of the server goes sky high (300+ and sometimes to 1000(!)).

I've noticed that during the high load there are about 500-600 httpd processes. Normally (under normal load) there are 10 httpd processes.

Needless to say, during the high load it's almost impossible to see anything in ssh or /server-status. They are both very inresponsive.

When I manage to "service httpd stop" (need to run this command several times before httpd actually shuts down), and start it again immediately after shutdown, the load goes down to 0.30 (average) until the next time the load goes sky high again.

The load is NOT jumping to 300+ on fixed times. So that rules out a cronjob.

Here's an example graph of high load (as you can see twice a day):

I have no clue where to look anymore, could you point me into a direction to fix this problem?

 

[Cybex]

This might also be usefull:

 

[nobaloney]

You can check your logs (perhaps all your httpd/domains logs, but the larger ones probably are the best place to start looking). Or you can use a brute force method, and run a script as often as every five minutes, to check the server load and restart httpd if it gets above a certain number. On one of our servers we restart httpd every time the server load gets over five.

Jeff

 

[Cybex]

Hello Jeff,

Thank you for replying. I'll check the logs a bit more intense, cause I had a look at it but it's quite hard to find something usefull in all these entries

About the server you mentioned. Do you have a script that auto restarts httpd when the loads jumps above five? I would like to have it if possible. Thanks!

John

 

[nobaloney]

The script was not written by me, and I'm trying to understand it. If you send me an email request for /root/scripts/load.sh I'll clean it up as best I can and post it on my site but I cannot guarantee it will work for you without error.

Jeff

 

[massive]

you can use sysfence also. i am using it and it's working good

 

[Cybex]

Thanks Jeff, I'll try Massive's advice (sysfence).

@Massive: Do you cronjob sysfence to start every 5 mins orso? It's not starting as a daemon is it?

 

[massive]

acts as a daemon and you can define rules. eg

rule "APACHE" {
swapused > 450M
}
run once 'service httpd restart'

 

[Cybex]

I can't find how to let it act as a daemon, could you please tell me? The docs doesn't speak about it

 

[massive]

you can just run it like this :
sysfence httpd.conf

where httpd.conf if the file that you will put the rules you want to.

In top you can find it as sfwatch.

 

[Cybex]

Ah thank you very much. Looks like I have it running

 

[massive]

you can even get an email in case where a service restarted eg :

rule "APACHE" {
swapused > 450M
}
run once 'service httpd restart | mutt -s "Warning: Apache server on `hostname` restarted due to memory problems" [email protected]'

ps
mutt is a text based email client.