High number of processes of apache and high server load

[kaon]

Hello All,

I am facing a problem recently.
There are sudden spike in the server load.
We did not upload any new software and only our software run on the server. So it is not a third party hosting.

In recent days we see this a lot in the top command. Check the load average also

top - 13:24:17 up 5 days, 21:56,  1 user,  load average: 48.34, 11.06, 7.01
Tasks: 474 total, 210 running, 264 sleeping,   0 stopped,   0 zombie
Cpu(s): 90.1%us,  9.1%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.8%si,  0.0%st
Mem:   7979076k total,  7522608k used,   456468k free,   230484k buffers
Swap:  8388592k total,    10800k used,  8377792k free,  3553860k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
26178 mysql     20   0 1455m 132m 5964 S 17.9  1.7   0:05.93 mysqld
26463 apache    20   0  429m  24m 4204 R  1.1  0.3   0:00.25 httpd
26480 apache    20   0  429m  24m 4212 R  1.1  0.3   0:00.21 httpd
26597 apache    20   0  429m  24m 4352 R  1.1  0.3   0:00.22 httpd
26602 apache    20   0  430m  25m 4372 R  1.1  0.3   0:00.22 httpd
27208 apache    20   0  429m  24m 4236 R  1.1  0.3   0:00.08 httpd
27215 apache    20   0  429m  24m 4188 R  1.1  0.3   0:00.08 httpd
27299 apache    20   0  429m  24m 4204 R  1.1  0.3   0:00.06 httpd
27306 apache    20   0  429m  24m 4352 R  1.1  0.3   0:00.06 httpd
27322 apache    20   0  429m  24m 4176 R  1.1  0.3   0:00.07 httpd
25877 apache    20   0  429m  25m 5016 R  0.8  0.3   0:00.72 httpd
25904 apache    20   0  431m  26m 4496 R  0.8  0.3   0:00.64 httpd
26284 apache    20   0  429m  24m 4376 R  0.8  0.3   0:00.39 httpd
26307 apache    20   0  429m  24m 4352 R  0.8  0.3   0:00.34 httpd
26325 apache    20   0  429m  24m 4224 R  0.8  0.3   0:00.30 httpd
26326 apache    20   0  429m  24m 4236 R  0.8  0.3   0:00.29 httpd
26330 apache    20   0  429m  24m 4188 R  0.8  0.3   0:00.29 httpd
26333 apache    20   0  429m  24m 4192 R  0.8  0.3   0:00.29 httpd
26372 apache    20   0  429m  24m 4224 R  0.8  0.3   0:00.25 httpd
26377 apache    20   0  429m  24m 4216 R  0.8  0.3   0:00.26 httpd
26380 apache    20   0  429m  24m 4372 R  0.8  0.3   0:00.26 httpd
26385 apache    20   0  430m  24m 4212 R  0.8  0.3   0:00.25 httpd
26462 apache    20   0  429m  24m 4384 R  0.8  0.3   0:00.21 httpd
26465 apache    20   0  429m  24m 4200 S  0.8  0.3   0:00.21 httpd
26470 apache    20   0  429m  24m 4260 R  0.8  0.3   0:00.20 httpd
26471 apache    20   0  429m  24m 4256 R  0.8  0.3   0:00.21 httpd
26473 apache    20   0  429m  24m 4216 R  0.8  0.3   0:00.20 httpd
26479 apache    20   0  429m  24m 4216 R  0.8  0.3   0:00.21 httpd
26484 apache    20   0  429m  24m 4392 R  0.8  0.3   0:00.21 httpd
26486 apache    20   0  429m  24m 4204 R  0.8  0.3   0:00.21 httpd
26489 apache    20   0  429m  24m 4208 S  0.8  0.3   0:00.22 httpd
26491 apache    20   0  429m  24m 4260 R  0.8  0.3   0:00.21 httpd
26582 apache    20   0  429m  24m 4328 R  0.8  0.3   0:00.22 httpd
26583 apache    20   0  430m  25m 4344 R  0.8  0.3   0:00.21 httpd
26584 apache    20   0  430m  25m 4412 R  0.8  0.3   0:00.21 httpd
26588 apache    20   0  430m  25m 4356 R  0.8  0.3   0:00.21 httpd
26589 apache    20   0  430m  25m 4332 R  0.8  0.3   0:00.21 httpd
26591 apache    20   0  430m  25m 4364 R  0.8  0.3   0:00.22 httpd
26594 apache    20   0  430m  25m 4368 R  0.8  0.3   0:00.22 httpd
26595 apache    20   0  430m  25m 4332 S  0.8  0.3   0:00.21 httpd
26603 apache    20   0  430m  25m 4336 R  0.8  0.3   0:00.21 httpd
26604 apache    20   0  430m  25m 4344 R  0.8  0.3   0:00.21 httpd
26626 apache    20   0  429m  24m 4220 R  0.8  0.3   0:00.14 httpd
26627 apache    20   0  429m  24m 4196 S  0.8  0.3   0:00.13 httpd
26909 apache    20   0  429m  24m 4352 R  0.8  0.3   0:00.13 httpd

Is there any way of knowing what is going on?
Access logs seem fine, error logs seem fine.

Please help.

Cent OS
Apache 2.0
Php 5.3

 

[scsi]

Why do you think its a security problem? Its pretty normal.

http://help.directadmin.com/item.php?id=91

 

[kaon]

Why do you think its a security problem? Its pretty normal.

It goes to 80 load average and my server has crashed a few times recently.
Further I have never ever seen this much load on my server before. Again my server is idle.

http://help.directadmin.com/item.php?id=91

I get these strange requests in the server-status besides the normal calls.

Srv	PID	Acc	M	CPU	SS	Req	Conn	Child	Slot	Client	VHost	Request
0-0	6868	0/43/56	_	1.29	6	341	0.0	0.05	0.05	69.127.233.115	localhost	NULL
1-0	7232	0/25/80	_	0.73	6	332	0.0	0.00	0.05	69.127.233.117	localhost	NULL
2-0	7626	0/10/96	_	0.22	6	317	0.0	0.00	0.02	69.127.233.112	localhost	NULL
4-0	7646	0/11/48	_	0.18	4	0	0.0	0.00	0.00	86.99.118.30	localhost	NULL
5-0	7669	0/9/61	_	0.13	4	0	0.0	0.00	0.01	86.99.118.30	localhost	NULL
6-0	7670	0/8/82	_	0.23	5	85	0.0	0.00	0.11	66.167.158.114	localhost	NULL
8-0	7671	0/9/71	_	0.14	4	0	0.0	0.00	0.02	86.99.118.30	localhost	NULL
10-0	7715	0/6/58	_	0.13	5	477	0.0	0.00	0.00	66.167.158.117	localhost	NULL
12-0	7716	0/6/62	_	0.14	5	440	0.0	0.00	0.07	66.167.158.112	localhost	NULL
13-0	7717	0/6/26	_	0.14	5	487	0.0	0.00	0.00	66.167.158.116	localhost	NULL
14-0	7718	0/6/26	_	0.13	5	438	0.0	0.00	0.00	66.167.158.114	localhost	NULL
15-0	7109	0/34/34	_	0.94	6	338	0.0	0.00	0.00	69.127.233.117	localhost	NULL
18-0	7720	0/8/12	_	0.24	4	54	0.0	0.00	0.00	66.167.158.119	localhost	NULL
19-0	7377	0/22/22	_	0.58	4	431	0.0	0.00	0.00	86.99.118.30	localhost	NULL
20-0	7721	0/7/12	_	0.13	4	365	0.0	0.00	0.00	86.99.118.30	localhost	NULL
22-0	7772	0/4/8	_	0.09	5	431	0.0	0.00	0.00	66.167.158.115	localhost	NULL
23-0	7410	0/19/19	_	0.46	4	0	0.0	0.00	0.00	86.99.118.30	localhost	NULL
24-0	7773	0/4/4	_	0.09	5	430	0.0	0.00	0.00	66.167.158.112	localhost	NULL
25-0	7774	0/4/6	_	0.08	5	474	0.0	0.00	0.00	66.167.158.112	localhost	NULL
26-0	-	0/0/17	.	0.44	4	296	0.0	0.00	0.00	69.127.230.114	localhost	NULL
27-0	7775	0/4/6	_	0.09	5	502	0.0	0.00	0.00	66.167.158.118	localhost	NULL
28-0	7415	0/18/18	_	0.57	6	327	0.0	0.01	0.01	69.127.233.112	localhost	NULL
29-0	7776	0/4/6	_	0.09	5	497	0.0	0.00	0.00	66.167.158.114	localhost	NULL
30-0	7777	0/4/6	_	0.09	5	554	0.0	0.00	0.00	66.167.158.116	localhost	NULL
31-0	7778	0/4/4	_	0.09	5	496	0.0	0.00	0.00	66.167.158.115	localhost	NULL
32-0	7779	0/4/4	_	0.09	5	444	0.0	0.00	0.00	66.167.158.119	localhost	NULL
33-0	7780	0/4/4	_	0.09	5	433	0.0	0.00	0.00	66.167.158.117	localhost	NULL
34-0	7781	0/4/4	_	0.08	5	495	0.0	0.00	0.00	66.167.158.114	localhost	NULL
35-0	7782	0/4/4	_	0.09	5	384	0.0	0.00	0.00	66.167.158.113	localhost	NULL
36-0	7783	0/4/4	_	0.09	5	444	0.0	0.00	0.00	66.167.158.113	localhost	NULL
37-0	7784	0/4/4	_	0.08	5	404	0.0	0.00	0.00	66.167.158.118	localhost	NULL
38-0	7785	0/4/4	_	0.09	5	408	0.0	0.00	0.00	66.167.158.114	localhost	NULL
39-0	7786	0/4/4	_	0.09	5	513	0.0	0.00	0.00	66.167.158.117	localhost	NULL
40-0	7787	0/4/4	_	0.09	5	490	0.0	0.00	0.00	66.167.158.114	localhost	NULL
41-0	-	0/0/2	.	0.05	3	399	0.0	0.00	0.00	69.127.230.115	localhost	NULL
42-0	7864	0/2/2	_	0.05	6	428	0.0	0.00	0.00	69.127.233.119	localhost	NULL
44-0	-	0/0/2	.	0.05	2	385	0.0	0.00	0.00	69.127.230.117	localhost	NULL
46-0	7868	0/4/4	_	0.09	3	47	0.0	0.00	0.00	173.252.103.0	localhost	NULL
47-0	-	0/0/2	.	0.05	1	407	0.0	0.00	0.00	69.127.230.116	localhost	NULL
49-0	7871	0/2/2	C	0.05	0	411	0.0	0.00	0.00	69.127.230.118	localhost	NULL
52-0	-	0/0/2	.	0.05	0	402	0.0	0.00	0.00	69.127.230.113	localhost	NULL
57-0	7879	0/2/2	_	0.04	7	56	0.0	0.00	0.00	69.127.233.113	localhost	NULL
58-0	7880	0/2/2	_	0.05	6	493	0.0	0.00	0.00	69.127.233.118	localhost	NULL
59-0	7881	0/2/2	_	0.04	6	482	0.0	0.00	0.00	69.127.233.113	localhost	NULL
60-0	7882	0/2/2	_	0.05	6	471	0.0	0.00	0.00	69.127.233.113	localhost	NULL
61-0	7883	0/2/2	_	0.05	6	407	0.0	0.00	0.00	69.127.233.116	localhost	NULL
62-0	7884	0/2/2	_	0.04	6	391	0.0	0.00	0.00	69.127.233.118	localhost	NULL
63-0	7885	0/2/2	_	0.04	6	407	0.0	0.00	0.00	69.127.233.113	localhost	NULL
64-0	7886	0/2/2	_	0.05	6	448	0.0	0.00	0.00	69.127.233.118	localhost	NULL
65-0	7887	0/2/2	_	0.04	6	462	0.0	0.00	0.00	69.127.233.113	localhost	NULL
66-0	7888	0/2/2	_	0.04	6	394	0.0	0.00	0.00	69.127.233.116	localhost	NULL
68-0	7890	0/2/2	_	0.05	6	450	0.0	0.00	0.00	69.127.233.114	localhost	NULL
69-0	7891	0/2/2	_	0.04	6	411	0.0	0.00	0.00	69.127.233.114	localhost	NULL
70-0	7892	0/2/2	_	0.04	6	413	0.0	0.00	0.00	69.127.233.112	localhost	NULL
71-0	7893	0/2/2	_	0.04	6	404	0.0	0.00	0.00	69.127.233.115	localhost	NULL
72-0	7894	0/3/3	_	0.07	5	0	0.0	0.01	0.01	86.99.118.30	localhost	NULL

 

[scsi]

Grep your access logs to see what page those ips are hitting. You might have some sort of vulnerability somewhere.

 

[kaon]

Grep your access logs to see what page those ips are hitting. You might have some sort of vulnerability somewhere.

Can you please give me the command?
I am really really sorry but i am novice in server administration.

I am on directadmin so each domain also has its own access_log

 

[kaon]

Grep your access logs to see what page those ips are hitting. You might have some sort of vulnerability somewhere.

can you please help.
please

 

[kaon]

ok i used this command after searching.

This is the issue. Server is making connection to httpd due to some facebook

Why does Facebook appear in my server logs?

Facebook allows its users to send links to interesting web content to other Facebook users. Part of how this works on the Facebook system involves the temporary display of certain images or details related to the web content, such as the title of the webpage or the embed tag of a video. Our system retrieves this information only after a user provides us with a link. You may have found this page because a Facebook user sent a link from your website to other Facebook users. If you have any questions or concerns about any links or content sent by one of our users, please contact us at [email protected].

 

[scsi]

grep '69.127.233' /var/log/httpd/access_log

or

grep '69.127.233' /var/log/httpd/domains/domain.com.log

 

[interfasys]

There seems to be a lot of mini-DDOS attacks created by brute-force Wordpress scanners at the moment. You may want to implement fail2ban.

 

[kaon]

There seems to be a lot of mini-DDOS attacks created by brute-force Wordpress scanners at the moment. You may want to implement fail2ban.

I am using DDOS deflate
http://www.inetbase.com/scripts/ddos/install.ddos

Is it not enough. I read some tutorial on how to install it and use it so I installed it.
Is it not same?

Please let me know.

 

[interfasys]

I am using DDOS deflate
http://www.inetbase.com/scripts/ddos/install.ddos

Is it not enough. I read some tutorial on how to install it and use it so I installed it.
Is it not same?

Please let me know.

Seems to be the same. mod_security also has a ban system based on RBLs, but it crashes under load :S

 

[kaon]

Seems to be the same. mod_security also has a ban system based on RBLs, but it crashes under load :S

I already have mod_security also installed.

DDOS deflate and mod_security both are installed.

 

[interfasys]

I already have mod_security also installed.

DDOS deflate and mod_security both are installed.

More than 162,000 WordPress sites used in DDoS attack
http://www.computerweekly.com/news/2240215998/More-the-162000-WordPress-sites-used-in-DDoS-attack