Warning: The system load average is...

I find these mystifying. Got two of them on Nov 13. The only thing of any possible interest in the logs is an unknown SIGTERM in httpd log:
Sat Nov 13 00:13:01 2010] [notice] caught SIGTERM, shutting down

The httpd processes in the 'top' look odd - 22% CPU? Usually it would be around 4%.

These messages don't happen often, and the reported high load does not appear to affect the operation of the forum that is on the server. Any guesses about what causes it?

Code: 
Subject: Warning: The system load average is 20.76	Today at 12:52

This is an automated message notifying you that the 1 minute load average on your system is 20.76.
This has exceeded the 10 threshold.

One Minute - 20.76
Five Minutes - 5.89
Fifteen Minutes - 2.06

top - 12:52:04 up 23:12, 3 users, load average: 20.76, 5.89, 2.06
Tasks: 173 total, 2 running, 170 sleeping, 0 stopped, 1 zombie
Cpu(s): 7.5%us, 1.1%sy, 0.0%ni, 90.5%id, 0.8%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 4051448k total, 3857708k used, 193740k free, 36316k buffers
Swap: 2031608k total, 112k used, 2031496k free, 2885984k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
852 apache 15 0 167m 17m 4460 S 22.1 0.5 0:02.49 httpd 
882 apache 16 0 172m 22m 4436 S 22.1 0.6 0:01.28 httpd 
868 apache 15 0 169m 20m 4464 S 20.3 0.5 0:01.90 httpd 
2350 mysql 15 0 272m 99m 4000 S 16.6 2.5 21:28.55 mysqld 
906 apache 16 0 169m 20m 4400 R 12.9 0.5 0:00.29 httpd 
876 apache 15 0 168m 19m 4428 S 9.2 0.5 0:01.15 httpd 
1 root 15 0 10352 700 588 S 0.0 0.0 0:00.82 init 
2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0 
3 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0 
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 
5 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/1 
6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 
8 root 10 -5 0 0 0 S 0.0 0.0 0:00.01 events/0 
9 root 10 -5 0 0 0 S 0.0 0.0 0:00.01 events/1 
10 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 khelper 
47 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kthread 
52 root 10 -5 0 0 0 S 0.0 0.0 0:00.19 kblockd/0 
53 root 10 -5 0 0 0 S 0.0 0.0 0:00.02 kblockd/1 
54 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid 
126 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/0 
127 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/1 
130 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khubd 


================================
Automated Message Generated by DirectAdmin
 
How do you upgrade with directadmin ? Is there a script I can run? I am having the same issue as all of you with freebsd just started today.
 
Log in to DirectAdmin as admin. At the bottom of Admin Level, under Extra Features, you will find 'Licensing / Updates'. Click that.

Then at the very bottom of the License page you will find 'Update DirectAdmin'. Click that.
 
Last edited:
cnm said:
These messages don't happen often, and the reported high load does not appear to affect the operation of the forum that is on the server. Any guesses about what causes it?
Click to expand...

Examine httpd server status, when you've got such a message.
You will see what sites and pages are requested most.
 
Yes of course - but I don't know how to get server-status for the time (long past) that the large load occurred.

Maybe I could write my own script to detect large load and run server-status, but I don't know how to do that either. Where does DirectAdmin get the 'one minute load average' anyway?

Edit: Is there an httpd log I should look at? /var/log/httpd/error_log didn't have anything interesting for the time in question and access_log was empty..
 
Last edited:
There's a script, the idea of which is to monitor httpd-status in order to find hosts, that use CPU too much. You can find this script and re-write it for your purposes and tasks.

/var/log/httpd/error_log won't show much. You need examine httpd logs for each virtual host (active and archived).
 
I'm not sure, where to find this script now, it was developed by 3rd part company.
Some years ago I've been using it... I believe, it's not difficult to write a PHP/PERL/BASH script to parse server status once per 5 minutes or use some advanced monitoring system that can work with the httpd server status output.

Sorry, no names, no links can I give
 
Well, I have found out that 'uptime' gives load averages without being as heavy as 'top'. So I can write a script to run once a minute and if uptime shows large load, wget the Apache server-status URL to a html file. This will give me a little more info.

I did find a third party plugin http://scoutapp.com/plugin_urls/4-server-load but I don't want to mess with our server by installing it.
 
Warning: The system load average is ...

A couple of updates ago, DA was altered to include a server load checker

Unfortunately, it was set to be so anal, the moment your server spikes above a load average of 10.00, it sends you an email.

For 95% of DA users, there is no need for it to be set so anal - and is going to result in a large number of such false alarm warnings.

I have more than 40 clients using DA, and you can only imagine how this (PITA) new load checker lit up our blackberrys like a Christmas tree!

The solution for me was to add this to the beginning of each directadmin.conf file:

Internal default:
check_load=10.0 (this value can even be increased to 15 or 20 if you have well appointed hardware)
check_load_minute=15 (can be 1,5, or 15 minutes - default is set to 1)

I use mostly Intel i7-920 and Xeon Quad Core X3360 (or newer) processors, with decent allotments of RAM, so these machines can handle several minutes of above ideal max load (which is 8.00 on these rigs)

By cutting the load checker back to the 15 minute setting, I only get notified if the load spikes above 10.00 for more than 15 minutes ... thus eliminating 99% of the nuisance alerts, while keeping the feature useful.

http://www.directadmin.com/features.php?id=1088

.
.
.
.
 
Whatever works for you :).

Note that the line Internal default: should not be included.

Jeff
 
mr.applesauce said:
You can change the settings it uses.

Check help.directadmin.com or directadmin.com/versions.php
Click to expand...

I too, have been receiving these e-mails on a fairly regular basis. I try to check the process monitor, but I'm not noticing anything out of the ordinary, however it does seem like some sites are running slower than usual
 
In my opinion there should be an option for this in the admin settings.... a load of 10.0 is nothing for a 16 core machine these days.

I'm running into this on a new server I just setup... guess I'll go dig for the setting and up it a little...
 
Check this http://www.directadmin.com/features.php?id=1088


Every minute, this feature will check the system load to see if the 1 minute load average is greater than the value set in this option. The default is 10.0.
If the load is greater than 10 when the dataskq runs, an email will be sent to all Admins, at most once per day.

Internal default:
check_load=10.0
check_load_minute=1
Click to expand...
 
hurleyint1386 said:
When I do top, it seems like apache is a huge resource hog. I'll try doing some more investigating
Click to expand...

A while back ago I wrote a script that restarts apache after server load gets to about ten. That solved a lot of problems on that server (which is no longer in service). The real answer of course, is to figure out why. In the case of that particular service, it was underspecified and oversold (my first DirectAdmin server).
 
Due to what looks like an attack on my server I'm getting a lot of these messages. Load average sometimes over 300 in short bursts. This is despite using cloudflare for all my domains. My issue is the ips listed are showing cloudflare ips. I'm also seeing the same in webalizer stats. How do I get the real IP shown in these logs? I see there is /etc/nginx/nginx-cloudflare.conf but I'm not sure if it's being used as I would have expected to see real ips. It looks like mod_remoteip is loaded. On Apache 2.4.46 using CentOS6. Any suggestions?
 
Its best to start a new thread rather that use one that is 10 years old.
 
floyd said:
Its best to start a new thread rather that use one that is 10 years old.
Click to expand...
Why? It's still relevant. Why duplicate the same topic? If it's just because of age then older threads should be closed.
 
You must log in or register to reply here.
Back
Top