Understanding Warning: The system load average is...

H

hendrik250874

Verified User
Joined
Nov 27, 2006
Messages
19
Hello,
for some time I have on regular basis the reports that the system load is high.
At first I didn't think too much of it, and ordered more ram, but this is not really a constructive approach for a problem that I don't understand.
I'm running a VPS with average specs (4 vCPU / 12Gb ram), with -from what I think- a normal amount of sites (30-ish).

I could use some support to solve this issue, and understand the reports better, so I can take appropriate action.
Here's the last one from today;

---
This is an automated message notifying you that the 5 minute load average on your system is 16.14.
This has exceeded the 10 threshold.

One Minute - 19.28
Five Minutes - 16.14
Fifteen Minutes - 11.75

top - 19:16:09 up 74 days, 23:16, 0 users, load average: 19.28, 16.14, 11.75
Tasks: 315 total, 8 running, 305 sleeping, 0 stopped, 2 zombie
%Cpu(s): 59.3 us, 40.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 12137348 total, 4053652 free, 6538028 used, 1545668 buff/cache
KiB Swap: 8388604 total, 7318436 free, 1070168 used. 5130656 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23879 opencoa+ 20 0 355652 142816 7224 R 100.0 1.2 0:01.92 /usr/sbin/httpd -DFOREGROUND
23851 opencoa+ 20 0 331788 118476 6964 R 94.4 1.0 0:03.58 /usr/sbin/httpd -DFOREGROUND
23881 opencoa+ 20 0 358308 145764 7220 R 66.7 1.2 0:02.31 /usr/sbin/httpd -DFOREGROUND
23882 opencoa+ 20 0 341044 128344 7156 R 66.7 1.1 0:01.31 /usr/sbin/httpd -DFOREGROUND
23392 opencoa+ 20 0 360088 146192 7620 R 38.9 1.2 0:08.28 /usr/sbin/httpd -DFOREGROUND
23886 opencoa+ 20 0 368612 155388 7260 S 38.9 1.3 0:02.44 /usr/sbin/httpd -DFOREGROUND
23877 opencoa+ 20 0 295468 82880 7360 R 33.3 0.7 0:02.13 /usr/sbin/httpd -DFOREGROUND
23941 root 20 0 58688 2292 1484 R 11.1 0.0 0:00.02 /usr/bin/top -c -b -n 1
5271 mysql 20 0 2674100 663284 4500 S 5.6 5.5 594:48.26 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
12641 root 20 0 177788 20920 1028 S 5.6 0.2 1:27.79 lfd - sleeping
23829 apache 20 0 234456 18236 3668 S 5.6 0.2 0:00.01 /usr/sbin/httpd -DFOREGROUND
1 root 20 0 191440 2756 1408 S 0.0 0.0 29:34.88 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
2 root 20 0 0 0 0 S 0.0 0.0 0:02.59 [kthreadd]
4 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kworker/0:0H]
6 root 20 0 0 0 0 S 0.0 0.0 53:47.15 [ksoftirqd/0]
7 root rt 0 0 0 0 S 0.0 0.0 0:15.51 [migration/0]
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [rcu_bh]
9 root 20 0 0 0 0 R 0.0 0.0 180:25.09 [rcu_sched]
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [lru-add-drain]
11 root rt 0 0 0 0 S 0.0 0.0 0:26.52 [watchdog/0]
12 root rt 0 0 0 0 S 0.0 0.0 0:23.11 [watchdog/1]
13 root rt 0 0 0 0 S 0.0 0.0 0:15.37 [migration/1]
14 root 20 0 0 0 0 S 0.0 0.0 54:09.35 [ksoftirqd/1]

Connection counts:
1 114.119.150.210
1 161.38.204.226
1 185.31.160.147
1 192.168.0.10
1 194.49.52.104
1 23.90.160.122
1 34.77.13.3
1 40.101.45.197
1 40.77.167.220
1 46.148.40.139
1 46.148.40.44
1 52.97.202.245
1 52.98.144.133
2 109.129.92.173
2 127.0.0.1
2 178.51.75.191
2 195.135.195.135
2 213.119.186.29
2 52.167.144.68
3 188.188.12.159
4 46.148.40.179
4 81.83.154.42
4 84.196.114.67
5 46.148.40.131
7 146.70.120.44
9 84.197.166.46
12 2a01
66 2a02

IP '2a02' currently has '66' connections

Connection info for '2a02':
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:1810:2e82:85:59268 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:1810:c03:720:53101 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:993 2a02:a03f:82fb:f1:60760 ESTABLISHED
tcp6 0 32 2a01:7c8:d003:19a:5:993 2a02:1810:3f2c:29:57500 LAST_ACK
tcp6 0 17 2a01:7c8:d003:19a:5:143 2a02:1810:407:a50:52095 ESTABLISHED
tcp6 0 0 2a01:7c8:d003:19a:5:993 2a02:1810:3f2c:29:51626 ESTABLISHED
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:1810:2e82:85:59279 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:c03:720:53106 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:1810:c03:720:53096 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:2e2e:71:53135 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:a03f:e09a:2a:45670 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:2e82:85:51597 FIN_WAIT2
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:a03f:e09a:2a:45536 FIN_WAIT2
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:2e82:85:51593 FIN_WAIT2
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:a03f:e09a:2a:45688 TIME_WAIT
tcp6 0 784 2a01:7c8:d003:19a:50:80 2a02:1810:290a:36:50942 LAST_ACK
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:a443:6d9:1:f:57670 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:2e39:7e:62816 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:1810:2e82:85:59281 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:c03:720:53099 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:a452:a9b2:1::53903 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:a03f:e09a:2a:51926 ESTABLISHED
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:c03:720:53107 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:1810:2e82:85:59280 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:1810:2e39:7e:62815 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:a03f:e09a:2a:45672 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:993 2a02:1810:3f2c:29:51621 ESTABLISHED
tcp6 0 0 2a01:7c8:d003:19a:5:993 2a02:1810:3f2c:29:52193 ESTABLISHED
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:2e82:85:51602 FIN_WAIT2
tcp6 0 785 2a01:7c8:d003:19a:50:80 2a02:1810:290a:36:50948 LAST_ACK
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:1810:c03:720:53104 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:a03f:e09a:2a:45712 TIME_WAIT
tcp6 0 917 2a01:7c8:d003:19a:5:443 2a02:1810:2e39:7e:62830 ESTABLISHED
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:a452:a9b2:1::53909 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:a03f:e09a:2a:45702 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:993 2a02:a03f:812e:72:64790 ESTABLISHED
tcp6 0 23584 2a01:7c8:d003:19a:5:443 2a02:1810:290a:36:60653 LAST_ACK
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:2e82:85:51590 FIN_WAIT2
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:1810:c03:720:53100 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:2e39:7e:62820 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:2e82:85:51604 FIN_WAIT2
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:2e39:7e:62824 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:1810:2e82:85:59269 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:2e39:7e:62813 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:993 2a02:1810:3f2c:29:51657 ESTABLISHED
tcp6 0 368 2a01:7c8:d003:19a:5:993 2a02:a03f:82fb:f1:60763 ESTABLISHED
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:1810:2e2e:71:53134 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:2e82:85:51601 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:a03f:e09a:2a:51924 ESTABLISHED
tcp6 0 0 2a01:7c8:d003:19a:5:993 2a02:a03f:812e:72:64791 ESTABLISHED
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:a03f:e09a:2a:51916 ESTABLISHED
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:1810:2e39:7e:62828 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:c03:720:53097 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:2e82:85:51585 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:993 2a02:1810:3f2c:29:52290 ESTABLISHED
tcp6 0 32 2a01:7c8:d003:19a:5:993 2a02:1810:3f2c:29:57494 LAST_ACK
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:a452:a9b2:1::53912 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:a452:a9b2:1::53906 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:a03f:e09a:2a:45676 TIME_WAIT
tcp6 0 517 2a01:7c8:d003:19a:5:443 2a02:1810:290a:36:60622 LAST_ACK
tcp6 0 0 2a01:7c8:d003:19a:50:80 2a02:a03f:e09a:2a:45690 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:2e39:7e:62823 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:c03:720:53103 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:c03:720:53105 TIME_WAIT
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:a452:a9b2:1::53913 ESTABLISHED
tcp6 0 0 2a01:7c8:d003:19a:5:443 2a02:1810:c03:720:53098 TIME_WAIT

Top Memory Usage:
root 4940 0.8 11.9 2033180 1456348 ? Ssl Feb02 236:25 /usr/local/sbin/clamd --foreground=yes
mysql 5271 2.0 5.4 2674100 663284 ? Sl Feb02 594:48 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
apache 23139 14.3 1.3 443888 158708 ? Sl 19:13 0:25 /usr/sbin/httpd -DFOREGROUND
apache 22188 11.4 1.3 443480 158132 ? Sl 19:09 0:47 /usr/sbin/httpd -DFOREGROUND
opencoa+ 23886 18.7 1.2 368612 155388 ? Sl 19:15 0:02 /usr/sbin/httpd -DFOREGROUND
opencoa+ 23881 19.4 1.2 368612 155316 ? Sl 19:15 0:02 /usr/sbin/httpd -DFOREGROUND
opencoa+ 23879 18.4 1.2 366500 154172 ? Rl 19:15 0:02 /usr/sbin/httpd -DFOREGROUND
opencoa+ 23872 19.3 1.2 368612 155324 ? Sl 19:15 0:02 /usr/sbin/httpd -DFOREGROUND
opencoa+ 23845 17.7 1.2 368612 155372 ? Sl 19:15 0:02 /usr/sbin/httpd -DFOREGROUND
opencoa+ 23392 8.5 1.2 360088 146604 ? Rl 19:14 0:08 /usr/sbin/httpd -DFOREGROUND

Virtual Memory Info:
procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu-------- -----timestamp-----
r b swpd free buff cache si so bi bo in cs us sy id wa st CET
7 0 1070168 4102908 88636 1457496 0 0 131 142 1 2 17 13 69 1 0 2023-02-22 19:16:10
3 0 1070168 4098812 88740 1457516 4 0 4 512 5270 4148 46 23 29 2 0 2023-02-22 19:16:11
14 0 1070168 3989408 88764 1457620 0 0 0 320 4383 1645 70 28 2 0 0 2023-02-22 19:16:12

Current MySQL Queries
1005031 opencoach_admin localhost opencoach_begrafenissen Sleep 4 NULL
1005036 opencoach_admin localhost opencoach_begrafenissen Sleep 3 NULL
1005039 opencoach_admin localhost opencoach_begrafenissen Sleep 2 NULL
1005040 opencoach_admin localhost opencoach_begrafenissen Sleep 2 NULL
1005041 opencoach_admin localhost opencoach_begrafenissen Sleep 3 NULL
1005042 opencoach_admin localhost opencoach_begrafenissen Sleep 1 NULL
1005046 opencoach_admin localhost opencoach_begrafenissen Sleep 0 NULL
1005047 opencoach_admin localhost opencoach_begrafenissen Sleep 0 NULL
1005048 opencoach_admin localhost opencoach_begrafenissen Sleep 0 NULL
1005049 opencoach_admin localhost opencoach_begrafenissen Sleep 0 NULL
1005050 opencoach_admin localhost opencoach_begrafenissen Sleep 0 NULL
1005051 opencoach_admin localhost opencoach_begrafenissen Sleep 0 NULL
1005052 opencoach_admin localhost opencoach_begrafenissen Sleep 0 NULL
1005053 opencoach_admin localhost opencoach_begrafenissen Sleep 0 NULL
1005054 da_admin localhost NULL Query 0 starting SHOW FULL PROCESSLIST


================================
Automated Message Generated by DirectAdmin 1.646

Help is greatly appreciated.
Hendrik
 
Install opcache, give it more memory, at least 256mb.
Also check mysqltuner recommendations.
 
Hello Hendrik,

Since you have many httpd processes on the top, and there are no PHP processes, I guess you have Apache+mod_ruid2+mod_php. This is an outdated and not very efficient setup. You might consider switching to Apache+PHP-FPM or even Nginx+Apache+PHP-FPM.

Then you might enable a Brute-Force Monitor in Directadmin and allow it to scan apache's logs for possible attacks. This way you can stop possible brute-force attacks, or reduce an impact of them.

You might consider enabling caching on an application level.

Keep investigating further and deeper if the mentioned steps do not help.
 
You must log in or register to reply here.
Back
Top