Abnormal system load

[sifro]

Hello,

i suddendly started receiving emails from directadmin like this one:

This is an automated message notifying you that the 5 minute load average on your system is 73.62.
This has exceeded the 10 threshold.

One Minute - 74.21
Five Minutes - 73.62
Fifteen Minutes - 71.39

top - 03:52:04 up 3 days, 8:52, 0 users, load average: 74.35, 73.66, 71.41
Tasks: 304 total, 75 running, 229 sleeping, 0 stopped, 0 zombie
Cpu(s): 72.1%us, 2.0%sy, 0.1%ni, 23.9%id, 1.2%wa, 0.4%hi, 0.4%si, 0.0%st
Mem: 3115464k total, 2840484k used, 274980k free, 197472k buffers
Swap: 5177336k total, 76k used, 5177260k free, 1702148k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19392 apache 25 0 34096 15m 3548 R 19.6 0.5 0:10.49 /usr/sbin/httpd -k start -DSSL
19480 apache 24 0 34172 15m 3788 R 19.6 0.5 0:04.79 /usr/sbin/httpd -k start -DSSL
19481 apache 25 0 34048 14m 3436 R 19.6 0.5 0:10.49 /usr/sbin/httpd -k start -DSSL
19482 apache 25 0 34180 14m 3500 R 19.6 0.5 0:06.63 /usr/sbin/httpd -k start -DSSL
19881 apache 25 0 34048 14m 3436 R 19.6 0.5 0:10.49 /usr/sbin/httpd -k start -DSSL
20262 root 20 0 2424 1032 696 R 2.0 0.0 0:00.01 /usr/bin/top -c -b -n 1
1 root 15 0 2160 596 516 S 0.0 0.0 0:00.62 init [3]
2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [migration/0]
3 root 34 19 0 0 0 R 0.0 0.0 0:00.00 [ksoftirqd/0]
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/0]
5 root 10 -5 0 0 0 S 0.0 0.0 0:00.04 [events/0]
6 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [khelper]
7 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [kthread]
10 root 10 -5 0 0 0 S 0.0 0.0 0:00.37 [kblockd/0]
11 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 [kacpid]
109 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 [cqueue/0]
112 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [khubd]
114 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [kseriod]
179 root 15 0 0 0 0 S 0.0 0.0 0:00.00 [khungtaskd]
180 root 25 0 0 0 0 S 0.0 0.0 0:00.00 [pdflush]
181 root 15 0 0 0 0 S 0.0 0.0 0:04.77 [pdflush]
182 root 10 -5 0 0 0 S 0.0 0.0 0:01.07 [kswapd0]
183 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 [aio/0]

If i go to "service monitor" i can see lots of apache PIDs, and the websites are very slow.

If i restart apache, sites get back to normal, but after around 30 minutes, there are once again a lot of spawned apache processes and an unusual high load.

Any ideas on how to solve this?

I haven't had any abnormal increase in the amount of traffic, so why did it start freaking out?


Here's the result of my "system info" page:

Processor Name QEMU Virtual CPU version (cpu64-rhel6)
Vendor ID GenuineIntel
Processor Speed (MHz) 2394.306
Total Memory 3115464 kB
Free Memory 864084 kB
Total Swap Memory 5177336 kB
Free Swap Memory 5177260 kB
System Uptime 3 Days, 14 Hours and 14 Minutes
Apache 2.2.22 Running
DirectAdmin 1.39.3 Running
Exim 4.76 Running
MySQL 5.0.67 Running
Named 9.3.6 Running
ProFTPd 1.3.3c Running
sshd Running
dovecot 2.1.1 Running
Php 5.3.10 Installed

 

[zEitEr]

Hello,

You must be under DDoS attack. You might need to enable server-status and see what's going on there, and what page/domain is mostly requested. Then either block the account with the domain, or tune your server (configure apache and install additional software/modules).

P.S. Feel free to contact me with PM, if you need my help as a commerce service.

 

[sifro]

Thanks, with server-status i could identify the IP causing the problems. I banned him, and now everything is fine.

 

[conconnl]

Hi Sifro,

Can you write down how you found the IP Address and how you banned it?
Sometimes i have the same problem.

The strange thing is, that i just recieved a new server.
There are no users on it and i already get brute force messages.

Thx for your how to.

 

[zEitEr]

Enable server-status in Apache config and open it in your browser, you'll all active connections and IPs.

 

[sifro]

Hi, you really should enable server-status... there are tons of annoying IPs out there, mine can't be the same of yours.
For more info, have a look at this: http://httpd.apache.org/docs/2.0/mod/mod_status.html
You just have to edit your httpd-info.cfg file in /etc/httpd/conf/extra (or something similar).

Anyway, this problem is happening again... and this time i guess it's caused by... Googlebot! It opens lots of connections and then apache doesn't close them!
Here are the outputs of some useful commands:

netstat -plan|grep :80 |awk '{print $5}' | awk -F : '{print $(NF-1)}' | sort | uniq -c | sort -n

1
1 151.37.103.129
1 151.51.45.144
1 151.66.59.146
1 151.95.20.105
1 178.198.31.93
1 188.11.48.206
1 2.196.63.135
1 31.190.74.248
1 87.21.160.98
3 159.20.194.86
4 31.26.20.157
5 66.249.66.44
5 79.31.143.176
6 151.73.52.57
6 93.46.112.207
8 2.227.18.196
8 2.38.35.29
8 31.159.152.45
8 82.91.108.210
8 95.244.88.65
9 95.250.33.93
12 88.86.172.81
15 66.249.72.88
27 95.231.5.100

netstat -a | grep 66.249 shows 20 connections from googlebot ip, 1 in ESTABILISHEd status, the others in TIME WAIT.

Apache server-status shows this:

Total Accesses: 292068
Total kBytes: 5540956
CPULoad: .0884054
Uptime: 45823
ReqPerSec: 6.37383
BytesPerSec: 123823
BytesPerReq: 19426.8
BusyWorkers: 99
IdleWorkers: 10
Scoreboard: WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWCWWWWWWWWWKWWWWWWWWWWKWWW_W_CWWW__WS__K_WWK__K_S........
........................................................
...................................................

... and the non machine-friendly version adds hundreds of lines like this one:

22-0 8559 0/58/149 _ 1.57 3 293 0.0 2.36 3.80 66.249.66.44 www.mydomain.com GET /modules.php?op=modload&name=Kalender&file=index&Date=01/31

66.249.66.44 is googlebot's IP.


So, basically, googlebot opens connections, and my webserver doesn't closes them.... any ideas to solve this?

 

[scsi]

Block googlebot?

 

[sifro]

Well, i can't block googlebot!
It is supposed to index my websites, how are my visitors going to find me then?

 

[zEitEr]

If it's one project server, then create an account in Google Webmaster and set limits for requests of Google-Bots, something of this king is possible, as far as I know....... or tune your software (for this part I'm available to do the job for a price as well as some other members of the forums here).