Warning: The system load average is 103.21

mr-secret

Verified User
Joined
Jan 20, 2015
Messages
31
Hello

on user on my hosting run many proccess and directadmin send this message Warning: The system load average is 103.21
i check user files but cant find any unnormally files
proccess picture atached.
 

Attachments

  • loadaverage.jpg
    loadaverage.jpg
    140.3 KB · Views: 112
I get this too, but if you have a multi-core system, it's ok. It only lasts seconds.

It's ever since I changed to FPM it happens....
 
this get hang server after 20-30 mintes
when syspend user everything is ok
how can resolve this problem?
 
I'm not too sure, I never had this issue when running php in mod_php & mod_ruid2, so it has to be something with FPM.....

Try asking in the PHP/MySQL sub-forum.
 
Hello,

For any issues with Apache+PHP you should start from analyzing of Apache server-status page. https://help.directadmin.com/item.php?id=91

Then check Apache server logs for a domain under question and search duplicated and repeated POST requests, your site might be under a brute-force or a spam-bots attack.

Check MySQL status with mysqladmin for long running queries and tables locks.

Use maldet and/or clamav for searching possible malware under public_html of users.
 
i found problem but i cant resolve this
i check file
domain.com.log
many reqauest from more than100000 ips

Code:
183.191.237.195 - - [18/May/2018:17:28:13 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432&direction=rtl&direction=ltr HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
66.249.73.143 - - [18/May/2018:17:28:17 +0430] "GET /wp-content/uploads/2018/04/12122-1-300x300.jpg HTTP/1.0" 200 247 "-" "Googlebot-Image/1.0"
58.19.1.87 - - [18/May/2018:17:28:13 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432&direction=rtl&direction=rtl HTTP/1.0" 404 6051 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
171.109.46.42 - - [18/May/2018:17:28:13 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249&direction=rtl&direction=ltr&direction=ltr HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
222.243.211.113 - - [18/May/2018:17:28:14 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1269 HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
39.71.131.216 - - [18/May/2018:17:28:15 +0430] "GET /product/??????-???-???-????-????/?add-to-cart=1249 HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
110.230.197.122 - - [18/May/2018:17:28:15 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249&direction=rtl HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
180.143.244.42 - - [18/May/2018:17:28:15 +0430] "GET /product/??????-???-???-????-????/ HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
58.19.1.87 - - [18/May/2018:17:28:15 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1269 HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
117.158.179.177 - - [18/May/2018:17:28:16 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249 HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
113.237.244.18 - - [18/May/2018:17:28:16 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1389 HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
113.108.126.6 - - [18/May/2018:17:28:17 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249&direction=rtl&direction=ltr&direction=ltr&direction=rtl HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
218.58.137.186 - - [18/May/2018:17:28:17 +0430] "GET /product/??????-???-???-????-????/ HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
116.21.62.202 - - [18/May/2018:17:28:17 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432&direction=rtl HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
114.243.243.64 - - [18/May/2018:17:28:18 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1269 HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
219.149.193.198 - - [18/May/2018:17:28:18 +0430] "GET /product/??????-???-???-????-????/?add-to-cart=1284 HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
27.204.50.38 - - [18/May/2018:17:28:18 +0430] "GET /product/??????-???-???-????-????/?add-to-cart=1284 HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
171.109.46.42 - - [18/May/2018:17:28:19 +0430] "GET /product/??????-???-???-????-????/ HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
111.172.232.102 - - [18/May/2018:17:28:19 +0430] "GET /product/??????-???-???-????-????/?add-to-cart=1249 HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
118.239.9.175 - - [18/May/2018:17:28:19 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432 HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
119.144.124.21 - - [18/May/2018:17:28:20 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249 HTTP/1.0" 404 6056 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
59.27.232.32 - - [18/May/2018:17:28:20 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432 HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
36.40.14.38 - - [18/May/2018:17:28:21 +0430] "GET /product/??????-???-???-????-????/?add-to-cart=1284 HTTP/1.0" 404 6056 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
42.63.236.214 - - [18/May/2018:17:28:21 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249&direction=rtl&direction=ltr HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
182.33.162.209 - - [18/May/2018:17:28:21 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1269 HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
110.230.197.122 - - [18/May/2018:17:28:23 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1389&direction=rtl HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
112.102.137.228 - - [18/May/2018:17:28:24 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432&direction=rtl&direction=ltr HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
117.13.144.85 - - [18/May/2018:17:28:24 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432&direction=rtl HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
119.144.124.21 - - [18/May/2018:17:28:25 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249&direction=rtl&direction=ltr HTTP/1.0" 404 6052 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
42.63.236.214 - - [18/May/2018:17:28:25 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1389&direction=rtl HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
42.63.236.214 - - [18/May/2018:17:28:26 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432&direction=rtl&direction=ltr HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
123.163.152.12 - - [18/May/2018:17:28:28 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249 HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
123.152.246.111 - - [18/May/2018:17:28:28 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249&direction=rtl&direction=ltr&direction=ltr&direction=rtl HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
117.158.179.177 - - [18/May/2018:17:28:28 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249&direction=rtl&direction=ltr HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
171.127.249.53 - - [18/May/2018:17:28:29 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432&direction=rtl HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windo
 
i found problem but i cant resolve this
i check file
domain.com.log
many reqauest from more than100000 ips

Code:
183.191.237.195 - - [18/May/2018:17:28:13 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432&direction=rtl&direction=ltr HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
66.249.73.143 - - [18/May/2018:17:28:17 +0430] "GET /wp-content/uploads/2018/04/12122-1-300x300.jpg HTTP/1.0" 200 247 "-" "Googlebot-Image/1.0"
58.19.1.87 - - [18/May/2018:17:28:13 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432&direction=rtl&direction=rtl HTTP/1.0" 404 6051 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
171.109.46.42 - - [18/May/2018:17:28:13 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249&direction=rtl&direction=ltr&direction=ltr HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
222.243.211.113 - - [18/May/2018:17:28:14 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1269 HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
39.71.131.216 - - [18/May/2018:17:28:15 +0430] "GET /product/??????-???-???-????-????/?add-to-cart=1249 HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
110.230.197.122 - - [18/May/2018:17:28:15 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249&direction=rtl HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
180.143.244.42 - - [18/May/2018:17:28:15 +0430] "GET /product/??????-???-???-????-????/ HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
58.19.1.87 - - [18/May/2018:17:28:15 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1269 HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
117.158.179.177 - - [18/May/2018:17:28:16 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249 HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
113.237.244.18 - - [18/May/2018:17:28:16 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1389 HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
113.108.126.6 - - [18/May/2018:17:28:17 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249&direction=rtl&direction=ltr&direction=ltr&direction=rtl HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
218.58.137.186 - - [18/May/2018:17:28:17 +0430] "GET /product/??????-???-???-????-????/ HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
116.21.62.202 - - [18/May/2018:17:28:17 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432&direction=rtl HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
114.243.243.64 - - [18/May/2018:17:28:18 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1269 HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
219.149.193.198 - - [18/May/2018:17:28:18 +0430] "GET /product/??????-???-???-????-????/?add-to-cart=1284 HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
27.204.50.38 - - [18/May/2018:17:28:18 +0430] "GET /product/??????-???-???-????-????/?add-to-cart=1284 HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
171.109.46.42 - - [18/May/2018:17:28:19 +0430] "GET /product/??????-???-???-????-????/ HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
111.172.232.102 - - [18/May/2018:17:28:19 +0430] "GET /product/??????-???-???-????-????/?add-to-cart=1249 HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
118.239.9.175 - - [18/May/2018:17:28:19 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432 HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
119.144.124.21 - - [18/May/2018:17:28:20 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249 HTTP/1.0" 404 6056 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
59.27.232.32 - - [18/May/2018:17:28:20 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432 HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
36.40.14.38 - - [18/May/2018:17:28:21 +0430] "GET /product/??????-???-???-????-????/?add-to-cart=1284 HTTP/1.0" 404 6056 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
42.63.236.214 - - [18/May/2018:17:28:21 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249&direction=rtl&direction=ltr HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
182.33.162.209 - - [18/May/2018:17:28:21 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1269 HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
110.230.197.122 - - [18/May/2018:17:28:23 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1389&direction=rtl HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
112.102.137.228 - - [18/May/2018:17:28:24 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432&direction=rtl&direction=ltr HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
117.13.144.85 - - [18/May/2018:17:28:24 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432&direction=rtl HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
119.144.124.21 - - [18/May/2018:17:28:25 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249&direction=rtl&direction=ltr HTTP/1.0" 404 6052 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
42.63.236.214 - - [18/May/2018:17:28:25 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1389&direction=rtl HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
42.63.236.214 - - [18/May/2018:17:28:26 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432&direction=rtl&direction=ltr HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
123.163.152.12 - - [18/May/2018:17:28:28 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249 HTTP/1.0" 404 6055 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
123.152.246.111 - - [18/May/2018:17:28:28 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249&direction=rtl&direction=ltr&direction=ltr&direction=rtl HTTP/1.0" 404 6053 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
117.158.179.177 - - [18/May/2018:17:28:28 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1249&direction=rtl&direction=ltr HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
171.127.249.53 - - [18/May/2018:17:28:29 +0430] "GET /product/??????-???-???-????-????/?add_to_wishlist=1432&direction=rtl HTTP/1.0" 404 6054 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windo


Greetings , from this little bit of info you gave us , it maybe a dos attack on that domain, which web server are using, if its apache , look into mod_evasive , this might limit it enough for you
as and extra layer , look into fail2ban , this will help with individual ips or ranges.

Let us know how you go.
 
A referer is empty, you probably could block all connections with add_to_wishlist= and empty referer.
 
0 (!). Consider adding more ram to the server. You have 4G and what we see on the last attachment is that under normal load you use 3/4 of it. Probably when you are under DDoS attack you get out of ram and you swap. I am pretty sure that that's the explanation for the high load - when your server swaps, it's making everything via disk operations and your hard drives are not fast enough to keep up the requests (happens even on SSD, and happens always on HDD)..

1. Tune up your firewall. Don't allow too many connections from one IP.

2. Lower all your timeouts. This includes the apache general timeout, keep alive time in apache, mysql idle timeout (just in case there are badly written scripts which are attacked)

3. Tune up your network card settings via sysctl.conf.

4. Think of implementing caching strategies for larger websites which use lots of resources

You can't stop DDoS attacks but you can at least make the server more durable.
 
Last edited:
hello , i have same problem and all my websites are down now when i look on TOP i see there is no load and ram is enough

top - 10:59:46 up 1:22, 1 user, load average: 0.10, 0.12, 0.09


Tasks: 333 total, 1 running, 332 sleeping, 0 stopped, 0 zombie


%Cpu(s): 0.1 us, 0.1 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st


MiB Mem : 5774.5 total, 3649.8 free, 937.1 used, 1187.6 buff/cache


MiB Swap: 2112.0 total, 2112.0 free, 0.0 used. 4560.6 avail Mem
 
Back
Top