Warning: The system load average is...

Invader Zim

Invader Zim

Verified User
Joined
Sep 4, 2004
Messages
188
The past few days I've been getting messages from one of our servers saying "Warning: The system load average is..." When it so happens that I am getting this message while at the computer I immediately jump on the server to check what is causing this load, but it only lasts for a few seconds apparently, because the load is back to normal, and even the past 5 minutes load is normal.

It'd be helpful if the message stated which process was hogging the resources since I've not been able to check for myself. It just doens't last long enough.
 
You can change the settings it uses.

Check help.directadmin.com or directadmin.com/versions.php
 
I too, have been receiving these e-mails on a fairly regular basis. I try to check the process monitor, but I'm not noticing anything out of the ordinary, however it does seem like some sites are running slower than usual. :confused:
 
The email is probably coming from grepping the uptime command, which is a simple one-line output.

You can write your own scripts to delve further into your system innards when this sort of thing happens.

Jeff
 
A new message or response with subject:

Warning: The system load average is 61.31

has arrived for you to view.
Follow this link to view it:
Click to expand...

When I do top, it seems like apache is a huge resource hog. I'll try doing some more investigating
 
A while back ago I wrote a script that restarts apache after server load gets to about ten. That solved a lot of problems on that server (which is no longer in service). The real answer of course, is to figure out why. In the case of that particular service, it was underspecified and oversold (my first DirectAdmin server).

Jeff
 
One of our servers keeps spiking to loads well over 20 (but it's FreeBSD (7.2) so you can still log in and do your thing) but the culprit seems to be proftpd. Many processes have been spawned than normal, and like I said, the load is very high. And when the machine has a high load, exim refuses to process mail.

The load has been very high since last evening, and there were over 1000 emails in the queue, which are getting processed now. But again, the culprit was proftpd.

Anyone have any similar experiences? I haven't had this issue on the other servers, they had high loads but it dropped off eventually. This one server however does not.
 
I do not know anything about FreeBSD but have you checked the disk I/O
to see if there is a lot of activity? That will cause the load to rise and stay
high as long as the I/O is high.
 
I didn't check i/o, since I noticed proftpd in top's list, with each thread consuming 20-25% cpu time.

Code: 
killall -9 proftpd ; /usr/local/etc/rc.d/proftpd stop ; sleep 2 ; /usr/local/etc/rc.d/proftpd start

brings the load back down to normal levels. So I've cronned it to run every hour on the hour, for now, until I find a solution.
 
Invader Zim said:
I didn't check i/o, since I noticed proftpd in top's list, with each thread consuming 20-25% cpu time.
Click to expand...

If I remember right proftpd allows users to write files.

If you have multiple threads then it could be that a user has opened multiple threads and writing multiple files at the same time which will result in high I/O.
 
Another issue that may cause high load is usage of more than minimal cache memory.

Jeff
 
For some reason multiple threads are opened. But ftpwho -v reveals no users to be online.

Also, just now I had another message from another server:

"Warning: The system load average is 20.1699"

proftpd only runs once. But then again, by the time this mail is sent, the load has dropped back to normal levels and the culprit cannot be identified, because exim doesn't process mail while the load is high. So this message comes after the fact. It's nice to see an iteration of top being included. However, I see nothing indicating a high i/o, but then again, top doesn't show Free's i/o load.

Here's top:

Code: 
last pid:  8757;  load averages: 20.17,  4.35,  1.53  up 150+02:01:54    17:48:06
456 processes: 54 running, 396 sleeping, 6 lock

Mem: 981M Active, 1820M Inact, 238M Wired, 128M Cache, 112M Buf, 75M Free
Swap: 2048M Total, 2328K Used, 2046M Free


 PID USERNAME   THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
8679 vind01       1  46    0 20504K 10720K RUN    0   0:00  0.20% php-cgi
 613 root         1  44    0   149M   145M select 1 747:53  0.10% snmpd
8717 vind01       1  -4    0 22780K 13168K ufs    0   0:00  0.10% php-cgi
8601 vind01       1  -4    0 22780K 13276K ufs    1   0:00  0.10% php-cgi
8673 vind01       1  -4    0 22780K 13168K ufs    1   0:00  0.10% php-cgi
8707 vind01       1  -4    0 22780K 13168K ufs    0   0:00  0.10% php-cgi
8641 vind01       1  -4    0 20504K 10720K ufs    1   0:00  0.10% php-cgi
8708 vind01       1  46    0 20504K 10692K RUN    0   0:00  0.10% php-cgi
8550 vind01       1  -4    0 19392K  9740K ufs    0   0:00  0.10% php-cgi
8719 vind01       1  -4    0 19380K  9728K RUN    0   0:00  0.10% php-cgi
 884 root         1   4    0  6348K  4472K kqread 0  40:57  0.00% dovecot-auth
13116 root         1  44    0 35140K 31076K select 1  19:17  0.00% perl
 837 root         1   4    0  3148K  1364K kqread 1  18:55  0.00% dovecot
 546 root         1  44    0  3184K  1024K select 1   4:23  0.00% syslogd
 640 clamav       1  20    0 23756K 10348K pause  1   4:02  0.00% freshclam
 695 clamav       1  20    0 24780K 11240K pause  1   3:02  0.00% freshclam
21306 root         1   8    0  8032K  5592K nanslp 0   2:40  0.00% da-popb4smtp
 607 root         1  44    0  6652K  3260K select 1   2:18  0.00% snmptrapd
 
The same has occurred with us on FreeBSD tonight. ProFTPd is limited to 30 threads.
30 simultaneous connections were established from one IP (DC in Germany). None activity, no tries to authenticate, nothing transfered. Empty logs on empty server waiting for a hardware upgrade. And LA ~30-33. No unusual disk activity, nothing on munin graphs, except too high LA.

We killed all threads with "kill -9" and upgraded to current proftpd.
 
imho related to the Telnet IAC stack overflow vulnerability which has been fixed in 1.3.3c.

Same problem here (FreeBSD). Time to upgrade :)
 
I wouldn't be surprised if this was my problem the other day (here) seeing as zEitEr said an IP was German, so were ours - although I could be wrong (for once lol!).
 
That IP doesn't ring bells.

There were only 2 IPs that were hitting our ProFTP (both visible in that thread I linked), one from Germany, and other from France (the same IP block as our server).
 
You must log in or register to reply here.
Back
Top